|
PHP - Processing Large Csv Files
I need to process a large CSV file (40 MB - 300,000 rows). While I have been working with smaller files my existing code is not able to work with large files.
All I need to do is - read a particular column from file and then count total number of rows and add all the values from column. My exisitng piece of code imports whole CSV file into an array (a class is used) and then using 'ForEach' loop, reads the required column and values into another array. Once the data is in this array i can simply sum or count it. While this served me well for smaller files, i am not able to use this approach to read a larger php file. I have already increased the memory allocated to php and max_execution_time but the script just keeps on running I am no php expert but usualy get around with trial and error.......your thoughts and help will be greatly appreciated Exisiting code: Once data has been processed by initial class (Class used is freely available and known as 'parsecsv', available at http://code.google.com/p/parsecsv-for-php/) After calling the class and processing the csv file: ?php ini_set('max_execution_time', 3000); $init = array(0); //Initialize a dummy array, which hold value '0' foreach ($csv->data as $key => $col): //Get value from array, 'Data' is an array processed by class and holds csv data $ColValue = $col[SALARY']; //retrieves the column you want { $SAL= $col['SALARY']; //Column that you want to process from csv array_push ($init, $SAL); // Push value into dummy array created above echo "<pre>"; } endforeach; $total_rows = (Count($init) -1); //Count total number of value, '-1' to remove the first initilaized value in array echo "Total # of rows: ". $total_rows . "\n"; echo "Total Sum: ". array_sum($init) . "\n"; ?> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Similar TutorialsHello, Im trying to find a way to check around 500-600 links to check if they are alive. It works fine for 5-6 links but once i add more links it just times out. Is there a way i could process this so it does 1 link at a time or somthing ? <?php include("config.php"); $query = "SELECT * FROM `games` WHERE `r_fileserve` <> \"\" LIMIT 500"; $result = mysql_query($query); while($row=mysql_fetch_assoc($result)) { $link_str = file_get_contents("$row[r_fileserve]"); $pattern = '<input type="hidden" name="download" value="normal"/>'; preg_match($pattern,$link_str,$match); if ($match[0] != null) { echo "Working <br />"; } else { echo "File Down <br />"; } } ?> I have 2 scripts that take a while to process... The first takes a few minutes, the second takes over an hour... The first script completed successfully, but the second script seems like it just stops most of the way through the script... I have this set on both files: set_time_limit(0); ini_set('memory_limit', -1); I NEED to disable ALL time limits and all of that, does ANYONE have any clue as to why my second script is failing? It is NOT the programming, I know that for sure... But the script just stops executing, and I think it is because it times out... But after setting those 2 options, I don't see why it times out.. Also for reference, I am NOT using any database... I am simply downloading files from an FTP server to my server... Any help would be appreciated, I have been searching for the last few days on an answer. Rob Hey Guys, I need a solution for uploading very large files. As I found PHP has some memory limits. Is it even possible to upload files with a size of 4GB? I need to parse large XML files ranging in size from ~ 500 to ~ 1700 Mb. I use XMLReader Code: [Select] set_time_limit(0); $start_time = microtime(true); include_once 'inc/Misc.php'; include_once 'inc/Database.php'; $files = array('xml/large_file.xml'); foreach($files as $file) { echo "\n"; echo 'Filename: '.basename($file)."\n"; echo 'Filesize: '.convert(filesize($file))."\n"; echo 'Start parsing...'."\n"; echo "\n"; $reader = new XMLReader(); $reader->open($file); while ($reader->read()) { switch ($reader->nodeType) { case (XMLREADER::ELEMENT): if ($reader->localName == "element-name") { $dom = new DomDocument(); $n = $dom->importNode($reader->expand(),true); $dom->appendChild($n); $sxe = simplexml_import_dom($n); $tess->file_big->insert($sxe); echo "Insert done! "; benchmark(); } } } } Everything is fine in the beginning ... Parsed file and slowly inserted my desired data, but is gradually growing memory consumption and has run out of resources. That is, I took the file to 400 Mb and as long as it is spent parsing of 2000 Mb of RAM and all the resources ran out and the script is stopped. How to deal with large files? ~ 500 to ~ 1700 Mb. Will there XML Parser? Yes, and how to apply it to my problem? Another option could have? I have a flash application that talks to upload.php Say I upload a 500mb file; it will obviously take a little while to upload. Will the max_execution_time settings cause this to fail? Its set at 60 right now and the upload is obviously taking longer than 1 minute. I have an xml file thats about 20MB, I want to parse it all into an array. SimpleXML seems to have trouble processing it, I just get "Fatal error: Balloc() failed to allocate memory". I've googled this topic and can't seem to find a straight forward solution. There must be someone here who has had to do this before. Is there an easy way to parse this?
<!doctype html>
<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="content-type">
<title>Untitled Document</title>
</head>
<body>
<form action="upload_image.php" method="post" enctype="multipart/form-data">
<label>select page<input type="file" name="image">
<input type="submit" name="upload">
</label>
</form>
<?php
if(isset($_POST['upload']))
{
$image_name=$_FILES['image']['name']; //return the name ot image
$image_type=$_FILES['image']['type']; //return the value of image
$image_size=$_FILES['image']['size']; //return the size of image
$image_tmp_name=$_FILES['image']['tmp_name'];//return the value
if($image_name=='')
{
echo '<script type="text/javascript">alert("please select image")</script>';
exit();
}
else
{
$ex=move_uploaded_file($image_tmp_name,"image/".$image_name);
if($ex)
{
echo 'image upload done "<br>"';
echo $image_name.'<br>';
echo $image_size.'<br>';
echo $image_type.'<br>';
echo $image_tmp_name.'<br>';
}
else
{
echo 'error';
}
}
}
?>
</body>
</html>
I make this simple script for upload files like photo , It's work correctly ,but not upload the large files ex 8MB images 12MB images
Edited by Ch0cu3r, 04 October 2014 - 09:50 AM. Modified topic title - changed download to upload Hi I am running debian lenny, apache2 php5. My fail upload fails for large file sizes. A 6.2 MB file uploads file, but a 10.2Mb file fails. I have set the Max file size to 50MB and max_execution to 600 etc in php.ini, but still have the same problems. I have noticed many others having similar problems. Is there a solution? Hi I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File. What the function must do is this: Originally I have the CSV like: Code: [Select] Copyright (c) 2007 MaxMind LLC. All Rights Reserved. locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,, When I pass the CSV file to this function I got: Code: [Select] locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,, It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory. The function is: public function deleteLine($line_no, $csvFileName) { // this function strips a specific line from a file // if a line is stripped, functions returns True else false // // e.g. // deleteLine(-1, xyz.csv); // strip last line // deleteLine(1, xyz.csv); // strip first line // Assigna o nome do ficheiro $filename = $csvFileName; $strip_return=FALSE; $data=file($filename); $pipe=fopen($filename,'w'); $size=count($data); if($line_no==-1) $skip=$size-1; else $skip=$line_no-1; for($line=0;$line<$size;$line++) if($line!=$skip) fputs($pipe,$data[$line]); else $strip_return=TRUE; return $strip_return; } It is possible to refactor this function to not blow up with the 256MB PHP Memory? Give me some clues. Best Regards, Hello everyone, I have a datatbase containing fields id, question, answer, and type. The questions are in true/false format, multiple choice and fill in the blank(text). At the moment, I am only trying to get the true and false questions to work. My code is as follows Code: [Select] <?php $connection = mysql_connect("localhost","root","root") or die("couldn't connect"); $select = mysql_select_db("login") or die ("cannot select database!"); $query = mysql_query("SELECT * FROM questions"); $numrows = mysql_num_rows($query); if ($numrows!=0) { //code to login while($row = mysql_fetch_assoc($query)) { $dbquestion = $row['question']; $dbanswer = $row['answer']; $dbtype = $row['type']; $dbid = $row['id']; $correctanswer = explode('|',"$dbanswer"); switch($dbtype){ case "mchoice": echo"<br/>"; echo $dbquestion; echo"<br/>"; break; case "boolean": echo"<br/>"; echo $dbquestion; echo"<br/>"; echo $correctanswer[0]; //display radio buttons ?> <form name="boolean" method="POST" action="<?php echo $_SERVER['PHP_SELF'] ?>"> True <input name="question" type="radio" value= "True" /> False <input name="question" type="radio" value= "False" /> <input name="id" input type="hidden" value="<?php $dbid ?>" /> <input type="submit" name="submit" id="submit" value="Submit" /> <?php echo "The answer submitted was "; echo $_POST['question']; ?> <?php break; case "text": echo"<br/>"; echo $dbquestion; echo"<br/>"; //display text box break; } } } ?> The problem I'm having is that this creates a submit button for each question. When choosing true or false for the answer and pressing submit, the answer is selected for every question. I need the code to look at what the user has entered, check the answer against correctanswer[0]. I'm really in a bit of a hole here and if anyone can help me it would be greatly appreciated. Many thanks in advance James Can you take a look at this page? It uses php code that I have been creating. I need it to accept the form even if checkboxes are left empty. Currently it comes up with an error. http://www.pilotrock.com/color_contact/ Here is the PHP code: <?php if(isset($_POST['Email_Address'])) { include 'lite_settings.php'; function died($error) { echo "Sorry, but there were error(s) found with the form you submitted. "; echo "These errors appear below.<br /><br />"; echo $error."<br /><br />"; echo "Please go back and fix these errors.<br /><br />"; die(); } if(!isset($_POST['organization']) || !isset($_POST['Full_Name']) || !isset($_POST['title']) || !isset($_POST['company']) || !isset($_POST['address']) || !isset($_POST['city_state']) || !isset($_POST['zip']) || !isset($_POST['Telephone_Number']) || !isset($_POST['cell_phone']) || !isset($_POST['fax']) || !isset($_POST['Email_Address']) || !isset($_POST['confirm_email']) || !isset($_POST['powder_coated_steel_blue']) || !isset($_POST['powder_coated_steel_black']) || !isset($_POST['powder_coated_steel_green']) || !isset($_POST['powder_coated_steel_brown']) || !isset($_POST['powder_coated_steel_yellow']) || !isset($_POST['powder_coated_steel_red']) || !isset($_POST['powder_coated_steel_gray']) || !isset($_POST['powder_coated_steel_burgendy']) || !isset($_POST['thermo_plastic_coated_type_expanded']) || !isset($_POST['thermo_plastic_coated_type_perforated']) || !isset($_POST['thermo_plastic_coated_black']) || !isset($_POST['thermo_plastic_coated_blue']) || !isset($_POST['thermo_plastic_coated_brown']) || !isset($_POST['thermo_plastic_coated_gray']) || !isset($_POST['thermo_plastic_coated_red']) || !isset($_POST['thermo_plastic_coated_green']) || !isset($_POST['recycle_plastic_red']) || !isset($_POST['recycle_plastic_yellow']) || !isset($_POST['recycle_plastic_black']) || !isset($_POST['recycle_plastic_blue']) || !isset($_POST['recycle_plastic_gold']) || !isset($_POST['recycle_plastic_redwood']) || !isset($_POST['recycle_plastic_cedar']) || !isset($_POST['recycle_plastic_green']) || !isset($_POST['recycle_plastic_gray']) || !isset($_POST['recycle_plastic_brown'])) { died('We are sorry, but there appears to be a problem with the form you submitted.'); } $organization = $_POST['organization']; // required $full_name = $_POST['Full_Name']; // required $title = $_POST['title']; // not required $company = $_POST['company']; // not required $address = $_POST['address']; // required $city_state = $_POST['city_state']; // required $zip = $_POST['zip']; // required $telephone = $_POST['Telephone_Number']; // required $cell_phone = $_POST['cell_phone']; // not required $fax = $_POST['fax']; // not required $email_from = $_POST['Email_Address']; // required $confirm_email = $_POST['confirm_email']; // required $powder_coated_steel_blue = $_POST['powder_coated_steel_blue']; // required $powder_coated_steel_black = $_POST['powder_coated_steel_black']; // required $powder_coated_steel_green = $_POST['powder_coated_steel_green']; // required $powder_coated_steel_brown = $_POST['powder_coated_steel_brown']; // required $powder_coated_steel_yellow = $_POST['powder_coated_steel_yellow']; // required $powder_coated_steel_red = $_POST['powder_coated_steel_red']; // required $powder_coated_steel_gray = $_POST['powder_coated_steel_gray']; // required $powder_coated_steel_burgendy = $_POST['powder_coated_steel_burgendy']; // required $thermo_plastic_coated_type_expanded = $_POST['thermo_plastic_coated_type_expanded']; // required $thermo_plastic_coated_type_perforated = $_POST['thermo_plastic_coated_type_perforated']; // required $thermo_plastic_coated_black = $_POST['thermo_plastic_coated_black']; // required $thermo_plastic_coated_blue = $_POST['thermo_plastic_coated_blue']; // required $thermo_plastic_coated_brown = $_POST['thermo_plastic_coated_brown']; // required $thermo_plastic_coated_gray = $_POST['thermo_plastic_coated_gray']; // required $thermo_plastic_coated_red = $_POST['thermo_plastic_coated_red']; // required $thermo_plastic_coated_green = $_POST['thermo_plastic_coated_green']; // required $recycle_plastic_red = $_POST['recycle_plastic_red']; // required $recycle_plastic_yellow = $_POST['recycle_plastic_yellow']; // required $recycle_plastic_black = $_POST['recycle_plastic_black']; // required $recycle_plastic_blue = $_POST['recycle_plastic_blue']; // required $recycle_plastic_gold = $_POST['recycle_plastic_gold']; // required $recycle_plastic_redwood = $_POST['recycle_plastic_redwood']; // required $recycle_plastic_cedar = $_POST['recycle_plastic_cedar']; // required $recycle_plastic_green = $_POST['recycle_plastic_green']; // required $recycle_plastic_gray = $_POST['recycle_plastic_gray']; // required $recycle_plastic_brown = $_POST['recycle_plastic_brown']; // required $error_message = ""; $email_exp = "^[A-Z0-9._%-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$"; if(!eregi($email_exp,$email_from)) { $error_message .= 'The Email Address you entered does not appear to be valid.<br />'; } if(strlen($full_name) < 2) { $error_message .= 'Your Name does not appear to be valid.<br />'; } if(strlen($address) < 2) { $error_message .= 'Your Address does not appear to be valid.<br />'; } if(strlen($city_state) < 2) { $error_message .= 'Your City/State does not appear to be valid.<br />'; } if(strlen($zip) < 2) { $error_message .= 'Your Zip/Postal Code does not appear to be valid.<br />'; } if(strlen($telephone) < 2) { $error_message .= 'Your Telephone Number does not appear to be valid.<br />'; } if(strlen($error_message) > 0) { died($error_message); } $email_message = "Form details below.\r\n"; function clean_string($string) { $bad = array("content-type","bcc:","to:","cc:","href"); return str_replace($bad,"",$string); } $email_message .= "Organization: ".clean_string($organization)."\r\n"; $email_message .= "Full Name: ".clean_string($full_name)."\r\n"; $email_message .= "Title: ".clean_string($title)."\r\n"; $email_message .= "Company Name: ".clean_string($company)."\r\n"; $email_message .= "Street Address: ".clean_string($address)."\r\n"; $email_message .= "City/State: ".clean_string($city_state)."\r\n"; $email_message .= "ZIP/Postal Code: ".clean_string($zip)."\r\n"; $email_message .= "Telephone: ".clean_string($telephone)."\r\n"; $email_message .= "Cell Phone: ".clean_string($cell_phone)."\r\n"; $email_message .= "Fax: ".clean_string($fax)."\r\n"; $email_message .= "Email: ".clean_string($email_from)."\r\n"; $email_message .= "Confirm Email: ".clean_string($confirm_email)."\r\n"; $email_message .= "Powder Coated Steel Blue: ".clean_string($powder_coated_steel_blue)."\r\n"; $email_message .= "Powder Coated Steel Black: ".clean_string($powder_coated_steel_black)."\r\n"; $email_message .= "Powder Coated Steel Green: ".clean_string($powder_coated_steel_green)."\r\n"; $email_message .= "Powder Coated Steel Brown: ".clean_string($powder_coated_steel_brown)."\r\n"; $email_message .= "Powder Coated Steel Yellow: ".clean_string($powder_coated_steel_yellow)."\r\n"; $email_message .= "Powder Coated Steel Red: ".clean_string($powder_coated_steel_red)."\r\n"; $email_message .= "Powder Coated Steel Gray: ".clean_string($powder_coated_steel_gray)."\r\n"; $email_message .= "Powder Coated Steel Burgendy: ".clean_string($powder_coated_steel_burgendy)."\r\n"; $email_message .= "Thermo Plastic Coated Type Expanded: ".clean_string($thermo_plastic_coated_type_expanded)."\r\n"; $email_message .= "Thermo Plastic Coated Type Perforated: ".clean_string($thermo_plastic_coated_type_perforated)."\r\n"; $email_message .= "Thermo Plastic Coated Black: ".clean_string($thermo_plastic_coated_black)."\r\n"; $email_message .= "Thermo Plastic Coated Blue: ".clean_string($thermo_plastic_coated_blue)."\r\n"; $email_message .= "Thermo Plastic Coated Brown: ".clean_string($thermo_plastic_coated_brown)."\r\n"; $email_message .= "Thermo Plastic Coated Gray: ".clean_string($thermo_plastic_coated_gray)."\r\n"; $email_message .= "Thermo Plastic Coated Red: ".clean_string($thermo_plastic_coated_red)."\r\n"; $email_message .= "Thermo Plastic Coated Green: ".clean_string($thermo_plastic_coated_green)."\r\n"; $email_message .= "Recycle Plastic Red: ".clean_string($recycle_plastic_red)."\r\n"; $email_message .= "Recycle Plastic Yellow: ".clean_string($recycle_plastic_yellow)."\r\n"; $email_message .= "Recycle Plastic Black: ".clean_string($recycle_plastic_black)."\r\n"; $email_message .= "Recycle Plastic Blue: ".clean_string($recycle_plastic_blue)."\r\n"; $email_message .= "Recycle Plastic Gold: ".clean_string($recycle_plastic_gold)."\r\n"; $email_message .= "Recycle Plastic Redwood: ".clean_string($recycle_plastic_redwood)."\r\n"; $email_message .= "Recycle Plastic Cedar: ".clean_string($recycle_plastic_cedar)."\r\n"; $email_message .= "Recycle Plastic Green: ".clean_string($recycle_plastic_green)."\r\n"; $email_message .= "Recycle Plastic Gray: ".clean_string($recycle_plastic_gray)."\r\n"; $email_message .= "Recycle Plastic Brown: ".clean_string($recycle_plastic_brown)."\r\n"; $headers = 'From: '.$email_from."\r\n". 'Reply-To: '.$email_from."\r\n" . 'X-Mailer: PHP/' . phpversion(); @mail($email_to, $email_subject, $email_message, $headers); header("Location: $thankyou"); ?> <script>location.replace('<?php echo $thankyou;?>')</script> <? } ?> I have an SQL table such as this: Pairing1 Pairing2 Week 6 1 1 5 2 1 4 3 1 1 3 2 5 4 2 3 6 2 and another with teams such as: teamid teamname 1 Teamname 1 2 Teamname 2 3 Teamname 3 The numbers in the pairing are numbers of teams, which is in another SQL table. What would be the most efficient way... of displaying the information in the form of pairngs such as Week 1: Team 6 vs Team 1 Team 5 vs Team 2 etc .... What is the best way for me to sort my results? I have page1.php which has a post form and page2.php which display data/results. I have a seperate php script file which queries a database. Having completed the query and assuming a result, I would end up with something like Code: [Select] while($result = mysql_fetch_assoc($query)) how do I get the data onto the display page. Looking at example approaches I found this using a go-between file Code: [Select] <?PHP include_once(process.php; process($_POST); display_results; ?> would display_results be a form of redirect? If not? Any help appreciated. Okay after correcting the issues with the jQuery side of things I'm having a small error with the php processing side. I'm getting a response back in firebug that there is an undefined index "name" and undefined index "value". Now I'm not sure on a fix but I know I'll have to do some sort of foreach I think because on the jquery dataString is a each so there's going to be more than name and value coming in at one time. Code: [Select] <?php error_reporting(E_ALL); // Include the database page include ('../inc/dbconfig.php'); $styleID = $_GET['id']; $query = "SELECT fields.ID, fields.fullName, fields.enabled FROM fields INNER JOIN styles ON styles.ID = fields.styleID WHERE styles.ID = '" . $styleID . "'"; $result = mysqli_query ( $dbc, $query ); // Run The Query ?> <script> $(document).ready(function() { $('div.message-error').hide(); $('div.message-success').hide(); $("input.submit").click(function() { $('div.message-error').hide(); var dataString = '&submitBioFields=True'; $('#bioConfigForm .field').each(function() { dataString += '&'+$(this).find('input:first').attr('name')+'='; dataString += ($(this).find('input[value|=0]').is(':checked')) ? '0' : '1'; }); alert(dataString); $.ajax({ type: "POST", url: "processes/bioconfig.php", data: dataString, success: function() { $('div.message-error').hide(); $("div.message-success").html("<h6>Operation successful</h6><p>Bio fields saved successfully.</p>"); $("div.message-success").show().delay(10000).hide("slow", function() { $('#content').load('mods/bioconfiguration.php'); }); } }); return false; }); }); </script> <!-- Title --> <div id="title" class="b2"> <h2>Bio Configuration</h2> <!-- TitleActions --> <div id="titleActions"> <!-- ListSearch --> <div class="listSearch actionBlock"> <div class="search"> <label for="search">Recherche</label> <input type="text" name="search" id="search" class="text" /> </div> <div class="submit"> <button type="submit" id="search-button" class="button"><strong><img src="img/icons/search_48.png" alt="comments" class="icon "/></strong></button> </div> </div> <!-- /ListSearch --> </div> <!-- /TitleActions --> </div> <!-- Title --> <!-- Inner Content --> <div id="innerContent"> <!-- Form --> <form action="#" id="bioConfigForm" > <fieldset> <legend>Bio Config</legend> <?php while ( $row = mysqli_fetch_array ( $result, MYSQL_ASSOC ) ) { ?> <div class="field"> <label for="<?php '' . $row['ID'] . '' ?>"><?php echo '' . $row['fullName'] . ''?></label> <input type="radio" value="0" name="<?php echo $row['ID']; ?>" class="status" <?php if($row['enabled'] == 0) echo ' checked="checked"'; ?> />Enabled <input type="radio" value="1" name="<?php echo $row['ID']; ?>" class="status" <?php if($row['enabled'] == 1) echo ' checked="checked"'; ?> />Disabled </div> <?php } ?> <input type="submit" class="submit" name="submitBioFields" id="SubmitBioFields" title="Submit Bio Fields" value="Submit Bio Fields"/> </fieldset> </form> <!-- /Form --> <!-- Messages --> <div class="message message-error"> <h6>Required field missing</h6> <p>Please fill in all required fields. </p> </div> <div class="message message-success"> <h6>Operation succesful</h6> <p>Bio configuraton was eddited to the database.</p> </div> <!-- /Messages --> <?php error_reporting(E_ALL); // Include the database page require ('../inc/dbconfig.php'); if (isset($_POST['submitBioFields'])) { $fieldID = (int)$_POST['name']; $value = (int)$_POST['value']; $query = "UPDATE `fields` SET `enabled` = '".$value."' WHERE `ID` = '".$fieldID."'"; mysqli_query($dbc,$query); $result = "good"; } //Output the result echo $result; ?> I have a form that I'm posting the contents to a 3rd party data management server via cURL. They return XML to let you know if the data was posted successfully or not. I'm trying to figure out how I can take that return XML and parse the data out of there. Quote <?php extract($_POST); $url = 'https://www.*domain*.com/DataReceiver'; $fields = array( blah blah blah ); foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; } rtrim($fields_string,'&'); $ch = curl_init(); curl_setopt($ch,CURLOPT_URL,$url); curl_setopt($ch,CURLOPT_POST,count($fields)); curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string); $result = curl_exec($ch); Now my question is, when the data posts and the XML is returned, how do I capture that? When I don't use cURL and post directly using HTTP POST, I get the XML returned at the exact same address I post to. So to process the XML returned, is this code correct? Quote $xml_feed_url = 'https://www.*domain*.com/DataReceiver'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $xml_feed_url); curl_setopt($ch, CURLOPT_HEADER, false); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $xml = curl_exec($ch); curl_close($ch); I'm kind of at a loss on what to do from this point... Thanks to anyone who takes the time to read and can help! Hi, I am just looking for some input on how I could make this form processing script more secure. In fact extremely secure. Like the most secure server side filtering can get! Code: [Select] <?php // Mail header removal function remove_headers($string) { $headers = array( "/to\:/i", "/from\:/i", "/bcc\:/i", "/cc\:/i", "/Content\-Transfer\-Encoding\:/i", "/Content\-Type\:/i", "/Mime\-Version\:/i" ); if (preg_replace($headers, '', $string) == $string) { return $string; } else { die('Spam much?'); } } // Build the email $to = 'info@example.com'; $subject = "Secure contact form message from: $subject"; $message = "$name said: $message"; $headers = "From: $email"; // field validation if ($subject=="" || $message=="" || $name=="") { print ("All form fields are required. Please go back and try again."); } else { // email validation if(!eregi('^([._a-z0-9-]+[._a-z0-9-]*)@(([a-z0-9-]+\.)*([a-z0-9-]+)(\.[a-z]{2,3})?)$', $email)) { print ("Your email address does not appear to be valid. Please go back and try again."); exit; } // Send the mail mail($to, $subject, $message, $headers); // Redirect header('Location: ../submitted.php'); } ?> I've read the php manual and a good few posts with similar but not identical problems and I am struggling to understand how to do this. Ive tailored my test site to help describe the problems that I have. www.peterod.co.uk the form is the first link in the left hand bar. Here is the code that creates the form. <?php // make connection require_once "../classes/sqlconselfbuild.php"; $query = mysql_query("SELECT * FROM `materials` ORDER BY `id` ASC"); ?> <form action="../engines/testengine1.php" method="post"> <?php // results while ($row = mysql_fetch_array($query)): $id = $row['id']; ?> <p> id number: <input type="text" name="materials[<?php echo $id; ?>][name]" value="<?php echo $row['id']; ?>" /> Material: <input type="text" name="materials[<?php echo $id; ?>][material]" value="<?php echo $row['material']; ?>" /> Stage: <input type="text" name="materials[<?php echo $id; ?>] [stage]" value="<?php echo $row['stage']; ?>" /> Qantity: <input type="text" name="materials[<?php echo $id; ?>][quantity]" value="<?php echo $row['quantity']; ?>" /> Unit: <input type="text" name="materials[<?php echo $id; ?>][unit]" value="<?php echo $row['unit']; ?>" /> <br /> </p> <?php endwhile; ?> <input type="submit" name="submit" value="Submit" /> </form> I am unsure how to get this information to insert into the database. How do you use the UPDATE query with a loop generated form ? [php] I have a form that is not doing anything after I click "submit". The code is below and a demo with the code is he http://communitycouch.com/index.php?action=register Code: [Select] <?php /* Things to do: Create error to show error when special characters are inputted in First and Last Name Fields Verify Dates to be true upon submit (for bithdays) Create birthday variable to be submitted into database, also add is_numberic() change activate.php to index?action=activate */ if($action == 'register') { if (isset($_POST['submit'])) { $errors = array(); require_once('connection.php'); //////////Checks Username //////////Makes sure username is 4-20 characters and contains only letters and numbers if(ereg("[[:alnum:]]{4,20}",stripslashes(trim($_POST['username'])))) { $user = mysql_real_escape_string($_POST['username']); $query = "SELECT username FROM reg_vars WHERE username = '$username'"; $result = @mysql_query($query); $num = @mysql_num_rows($result); if ($num> 0) { $errors[] = '<font color="red">The username you have chosen has already been taken, please try again.</font>'; } else { $username = mysql_real_escape_string($_POST['username']); } } else { $errors[] = '<font color="red">Please provide a valid username between 4 and 30 characters.</font>'; } //////////Checks E-mail if (!eregi('^[a-zA-Z]+[a-zA-Z0-9_-]*@([a-zA-Z0-9]+){1}(\.[a-zA-Z0-9]+){1,2}', stripslashes(trim($_POST['email'])) ) || empty($_POST['email'])) { $errors[] = '<font color="red">Please provide a valid email address.</font>'; } else { $email = mysql_real_escape_string($_POST['email']); } ///////// Check Names if (empty($f_name)) { $error["f_name"] = "First Name is blank."; } if (empty($l_name)) { $error["l_name"] = "Last Name is blank."; } ////////// Check PASSWORDS if (!empty($_POST['password'])) { if ($_POST['password'] != $_POST['cpassword']) { $errors[] = '<font color="red">The passwords you have entered do not match.</font>'; } else { $password = $_POST['password']; } } else { $errors[] = '<font color="red">Please provide a password.</font>'; } /////////Send Activation E-mail //////// Add Birthday to the registrarion Variables if (empty($errors)) { $a = md5(uniqid(rand(), true)); $query = "INSERT INTO reg_vars (username, f_name, l_name, email, password, active) VALUES ('$username', '$email', SHA1('$password'), '$a')"; $result = @mysql_query($query); if (mysql_affected_rows() == 1) { ////////// Send the Activation email $body = "Thank you for registering at the User Registration site. To activate your account, please click on this link:\n\n"; $body .= "http://www.communitycouch.com/activate.php?x=" . mysql_insert_id() . "&y=$a"; mail($_POST['email'], 'Registration Confirmation', $body, 'From: noreply@communitycouch.com'); ////////// Show thank you message echo '<h3>Thank You!</h3> You have been registered, you have been sent an e-mail to the address you specified before. Please check your e-mails to activate your account.'; } else { echo '<font color="red">You could not be registered, please contact us about the problem and we will fix it as soon as we can.</font>'; } } else { echo '<h3>Error!</h3> The following error(s) occured:<br />'; foreach ($errors as $msg) { echo " <div id=\"success_or_fail\"><font color=\"red\">$msg</font><br />\n</div>"; } } } echo " <div id=\"register_container\" class=\"Container\"> <div id=\"register_Center\" class=\"BoxCenter\"> "; echo" <div id=\"register_BoxContainer\" class=\"Container\"> <div class=\"HeaderLeft\"></div><div id=\"register_Header\" class=\"HeaderCenter\">Register for Community Couch</div><div class=\"HeaderRight\"></div><br /> <div id=\"register_Content\" class=\"BoxContent\"> <form name=\"reg_form\" action=\"";$_SERVER['PHP_SELF']; echo"\" method=\"post\" style=\"margin: 0;\"> <div class=\"Container FormBoxLeft\"> <b>Username:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"username\" size=\"30\" type=\"text\" /> </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>First Name:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"f_name\" size=\"30\" type=\"text\" /> </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>Last Name:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"l_name\" size=\"30\" type=\"text\" /> </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>Birthdate:</b> </div> <div class=\"Container FormBoxRight\"> <select name=\"month\"><option value=\"\">-Month-</option> <option value=\"01\">January</option> <option value=\"02\">February</option> <option value=\"03\">March</option> <option value=\"04\">April</option> <option value=\"05\">May</option> <option value=\"06\">June</option> <option value=\"07\">July</option> <option value=\"08\">August</option> <option value=\"09\">September</option> <option value=\"10\">October</option> <option value=\"11\">November</option> <option value=\"12\">December</option> </select> <select name=\"day\"> <option value=\"\"> -Day-</option>'; //Print 31 Days "; for ($x=1; $x<=31; $x++) { echo "<option value='".$x."'"; if(isset($_POST['submit']) && $day == $x) { echo " selected"; } echo ">".$x."</option>\n"; } echo "</select> <input name=\"year\" size=\"2\" maxlength=\"4\" type=\"text\"> </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>E-Mail:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"email\" size=\"30\" type=\"text\" /> </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>Display E-Mail to the Public:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"disp_email\" value=\"yes\" type=\"radio\" /> Yes <input name=\"disp_email\" value=\"no\" type=\"radio\" checked /> No </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>Password:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"password\" size=\"30\" type=\"password\" /> </div> <br class=\"EndColumn\" /> <div class=\"Container FormBoxLeft\"> <b>Confirm Password:</b> </div> <div class=\"Container FormBoxRight\"> <input name=\"cpassword\" size=\"30\" type=\"password\" /> </div> <br class=\"EndColumn\" /> <div id=\"register_submit\" class=\"Container\"> <input name=\"tos\" type=\"checkbox\" /> I have read and agree to the Terms of Use and Privacy Policy<br /> <input type=\"submit\" value=\"Submit\"><input type=\"reset\" value=\"Reset\"> </div> <br class=\"EndColumn\" /> </form> </div> <div class=\"FooterLeft\"></div><div id=\"register_Footer\" class=\"FooterCenter\"></div><div class=\"FooterRight\"></div> </div> </div> </div> <br class=\"EndColumn\" /> <br />"; }?> [\code] I seem to be having a problem with the below script, on the 1st run it doesnt print the $variables, but on refresh these a being written to file once? Code: [Select] if(isset($_GET['sprop']) && ($_GET['sprop'] == "update")) { //my connection stuff // // $playername = "userid"; $nether = $_POST["nether"]; $lname = $_POST["lname"]; $query = $_POST["query"]; $flight = $_POST["flight"]; $portdd = $_POST["port"]; $rcon = $_POST["rcon"]; $seed = $_POST["seed"]; $sip = $_POST["sip"]; $wlist = $_POST["wlist"]; $san = $_POST["san"]; $omode = $_POST["omode"]; $pvp = $_POST["pvp"]; $dif = $_POST["dif"]; $sname = $_POST["sname"]; $gmode = $_POST["mode"]; $pmax = $_POST["max"]; $sm = $_POST["sm"]; $view = $_POST["view"]; $motd = $_POST["motd"]; sleep(3); $sftp = new Net_SFTP($ftp_s); if (!$sftp->login($ftp_u, $ftp_p)) { exit('Login Failed'); } echo $sftp->pwd() . "\r\n"; $sftp->put ("/root/mc/server.properties", "allow-nether=$nether level-name=$lname enable-query=$query allow-flight=$flight server-port=$portdd enable-rcon=$rcon level-seed=$seed server-ip=$sip white-list=$wlist spawn-animals=$san online-mode=$omode pvp=$pvp difficulty=$dif server-name=$sname gamemode=$gmode max-players=$pmax spawn-monsters=$sm view-distance=$view motd=$motd"); } As you can see, I have try adding a sleep to give it time to catch up but something tells me the variables are not being set 1st time for some reason, can anyone shine any light on this? |
|||||||