PHP - Uploading Large Files
I have a flash application that talks to upload.php
Say I upload a 500mb file; it will obviously take a little while to upload. Will the max_execution_time settings cause this to fail? Its set at 60 right now and the upload is obviously taking longer than 1 minute. Similar TutorialsHey Guys, I need a solution for uploading very large files. As I found PHP has some memory limits. Is it even possible to upload files with a size of 4GB? I need to parse large XML files ranging in size from ~ 500 to ~ 1700 Mb. I use XMLReader Code: [Select] set_time_limit(0); $start_time = microtime(true); include_once 'inc/Misc.php'; include_once 'inc/Database.php'; $files = array('xml/large_file.xml'); foreach($files as $file) { echo "\n"; echo 'Filename: '.basename($file)."\n"; echo 'Filesize: '.convert(filesize($file))."\n"; echo 'Start parsing...'."\n"; echo "\n"; $reader = new XMLReader(); $reader->open($file); while ($reader->read()) { switch ($reader->nodeType) { case (XMLREADER::ELEMENT): if ($reader->localName == "element-name") { $dom = new DomDocument(); $n = $dom->importNode($reader->expand(),true); $dom->appendChild($n); $sxe = simplexml_import_dom($n); $tess->file_big->insert($sxe); echo "Insert done! "; benchmark(); } } } } Everything is fine in the beginning ... Parsed file and slowly inserted my desired data, but is gradually growing memory consumption and has run out of resources. That is, I took the file to 400 Mb and as long as it is spent parsing of 2000 Mb of RAM and all the resources ran out and the script is stopped. How to deal with large files? ~ 500 to ~ 1700 Mb. Will there XML Parser? Yes, and how to apply it to my problem? Another option could have? I need to process a large CSV file (40 MB - 300,000 rows). While I have been working with smaller files my existing code is not able to work with large files. All I need to do is - read a particular column from file and then count total number of rows and add all the values from column. My exisitng piece of code imports whole CSV file into an array (a class is used) and then using 'ForEach' loop, reads the required column and values into another array. Once the data is in this array i can simply sum or count it. While this served me well for smaller files, i am not able to use this approach to read a larger php file. I have already increased the memory allocated to php and max_execution_time but the script just keeps on running I am no php expert but usualy get around with trial and error.......your thoughts and help will be greatly appreciated Exisiting code: Once data has been processed by initial class (Class used is freely available and known as 'parsecsv', available at http://code.google.com/p/parsecsv-for-php/) After calling the class and processing the csv file: ?php ini_set('max_execution_time', 3000); $init = array(0); //Initialize a dummy array, which hold value '0' foreach ($csv->data as $key => $col): //Get value from array, 'Data' is an array processed by class and holds csv data $ColValue = $col[SALARY']; //retrieves the column you want { $SAL= $col['SALARY']; //Column that you want to process from csv array_push ($init, $SAL); // Push value into dummy array created above echo "<pre>"; } endforeach; $total_rows = (Count($init) -1); //Count total number of value, '-1' to remove the first initilaized value in array echo "Total # of rows: ". $total_rows . "\n"; echo "Total Sum: ". array_sum($init) . "\n"; ?> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- I have an xml file thats about 20MB, I want to parse it all into an array. SimpleXML seems to have trouble processing it, I just get "Fatal error: Balloc() failed to allocate memory". I've googled this topic and can't seem to find a straight forward solution. There must be someone here who has had to do this before. Is there an easy way to parse this? <!doctype html> <html> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"> <title>Untitled Document</title> </head> <body> <form action="upload_image.php" method="post" enctype="multipart/form-data"> <label>select page<input type="file" name="image"> <input type="submit" name="upload"> </label> </form> <?php if(isset($_POST['upload'])) { $image_name=$_FILES['image']['name']; //return the name ot image $image_type=$_FILES['image']['type']; //return the value of image $image_size=$_FILES['image']['size']; //return the size of image $image_tmp_name=$_FILES['image']['tmp_name'];//return the value if($image_name=='') { echo '<script type="text/javascript">alert("please select image")</script>'; exit(); } else { $ex=move_uploaded_file($image_tmp_name,"image/".$image_name); if($ex) { echo 'image upload done "<br>"'; echo $image_name.'<br>'; echo $image_size.'<br>'; echo $image_type.'<br>'; echo $image_tmp_name.'<br>'; } else { echo 'error'; } } } ?> </body> </html>I make this simple script for upload files like photo , It's work correctly ,but not upload the large files ex 8MB images 12MB images Edited by Ch0cu3r, 04 October 2014 - 09:50 AM. Modified topic title - changed download to upload Hi I am running debian lenny, apache2 php5. My fail upload fails for large file sizes. A 6.2 MB file uploads file, but a 10.2Mb file fails. I have set the Max file size to 50MB and max_execution to 600 etc in php.ini, but still have the same problems. I have noticed many others having similar problems. Is there a solution? Hi All, I am having trouble with this script that was from a example script online, basically I am getting the last error message everytime I try an up a zip file? Unknown Error: No file uploaded <?php include('dbconnect.php'); session_start(); $user_id = $_SESSION['user_id']; $query = " SELECT * FROM roms WHERE user_id = '$user_id' ORDER BY rom_date DESC LIMIT 1"; $result = mysql_query($query); if (!mysql_query($query)) { die('Error: ' . mysql_error()); } while($row = mysql_fetch_array($result)) { $_SESSION['rom_version'] = $row['rom_version']; } if ((!empty($_FILES["rom"])) && ($_FILES['rom']['error'] == 0)) { $filename = $user_id . $_SESSION['rom_version'] . '.zip'; $ext = substr($filename, strrpos($filename, '.') + 1); if (($ext == "zip") && ($_FILES["rom"]["type"] == "application/zip") && ($_FILES["rom"]["size"] < 2500000)) { $newname = dirname(__FILE__).'roms/'.$filename; if (!file_exists($newname)) { if ((move_uploaded_file($_FILES['uploaded_file'][ 'tmp_name'],$newname))) { echo "It's done! The file has been saved as: ".$newname; } else { echo "Error: A problem occurred during file upload!"; } } else { echo "Error: File ".$_FILES["rom"]["name"]." already exists"; } } else { echo "Error: Only .zip images under 250mb are accepted for upload"; } } else { echo "Unknown Error: No file uploaded"; } ?> I am using xammp with FileZilla running on it. I've changed my php.ini settings to "C:/xxx/tmp" directory but am unable to upload a file. Th cod is running but it dos not seem to upload the temporary fil. Plase somebody help me. ALso my form in upload.html has <form action="upload_file.php" method="post" enctype="multipart/form-data"> hello all, i moved my inhouse website to a hosting company. now i have a few files that i used to upload data to the website, but every time i try uploading it gives me this error. Warning: move_uploaded_file(..\pdfs\htdocs\pdfs\citysect/03152011CCRM.pdf) [function.move-uploaded-file]: failed to open stream: No such file or directory in /data/26/2/13/16/2502179/user/2740540/htdocs/loginscripts/uploadpdf.php on line 34 Warning: move_uploaded_file() [function.move-uploaded-file]: Unable to move '/data/20/tmp_new/php2GMXs3' to '..\pdfs\htdocs\pdfs\citysect/03152011CCRM.pdf' in /data/26/2/13/16/2502179/user/2740540/htdocs/loginscripts/uploadpdf.php on line 34 could not moved can anyone help me? Hello everyone, I'm writing a script that allows a user to upload multiple attachments to an email from PHP. So far, everything seems to be working except the multiple attachments part. If I upload one file, it works. If I upload 2, it's messing up the file names so the files don't exist. So if I upload a.jpg, it works. If I upload a.jpg and b.jpg, b.jpg gets named a b.jpg. Can someone help me understand why this is happening and how to fix it? I'm almost certain its happening on my while loop around lune 72, but the more I look at the syntax the more it looks okay to me. I had it appending the path to the file names (so a.jpg become uploads/a.jpg) so I commented that out in hopes that was causing the error but it didn't fix it. Below is the full script. <?php /******************************************* /* contact_us_process.php /* Author: Brandon Pence (brandonpence@gmail.com) /* /* Desc: This page process and checks the submitted form. /* /* variable meaning /* -------- ------- /* $FieldCheck Instance of the FieldCheck class. /* /******************************************/ //start session session_start(); //include files require ("./inc/include.inc.php"); //include Field Check Class include("classes/FieldCheck.php"); //instantiate fieldcheck class $FieldCheck = new FieldCheck; extract($_POST); /*****DEBUG*********/ echo '<h1>POST ARRAY:</h1>'; print_r($_POST); echo '<hr/>'; echo '<h1>DATAFILE:</h1>'; print_r($datafile); echo 'foreach: <hr/>'; echo '<h1>FILES ARRAY:</h1>'; print_r($_FILES); echo '<hr/>'; echo 'File name test: '.$_FILES['uploadedfile']['name']['0'].'<br/>'; echo 'File name test: '.$_FILES['uploadedfile']['name']['1'].'<br/>'; echo 'File name test: '.$_FILES['uploadedfile']['name']['2'].'<br/>'; /********************/ //foreach($datafile as $key=>$value){ //echo 'Datafile: '.$datafile.' || Key: '.$key.' || Value: '.$value.'<br/>'; //$value = $id.'/'.$value; //echo 'Datafile: '.$datafile.' || Key: '.$key.' || Value: '.$value.'<br/>'; //} if(file_exists("uploads/$id")){ echo 'Folder Exists!'; }else{ echo 'Folder does not exist. Create folder.'; //create directory mkdir("uploads/$id",0777,true); } //path we want the files stored in $target_path = "uploads/$id/"; //begin counter $i = 0; $files = array(); //begin loop while($i < 3){ //make sure each array key has valid data if(!is_null($_FILES['uploadedfile']['name'][$i])){ //append path to filenames in array //$target_path = $target_path . basename($_FILES['uploadedfile']['name'][$i]); //upload files if(move_uploaded_file($_FILES['uploadedfile']['name'][$i], $target_path)) { echo "The file ". basename($_FILES['uploadedfile']['name'][$i]). " has been uploaded"; $files[$i] = 'uploads/'.$id.'/'.$_FILES['uploadedfile']['name'][$i]; } //if file upload failed }else{ echo "There was an error uploading the file, please try again!"; } $i++; } //validate input if (!$FieldCheck->checkName($your_name)){ $errors['your_name'] = "Your name is not valid.";} if (!$FieldCheck->isEmpty($your_date)){ $errors['your_date'] = "Your date is not valid.";} if (!$FieldCheck->isEmpty($your_message)){ $errors['your_message'] = "Your message is not valid.";} foreach($_POST as $key=>$value){ $fields[$key] = $value; } //print_r($_POST); //check for error messages if (isset($errors)) { $_SESSION['errors'] = $errors; $_SESSION['fields'] = $fields; $_SESSION['error_msg'] = "Errors found! Please review your entries."; $url = "./testimonials.php?action=errors"; header ("Location: $url"); exit; }else{ echo '$files ARRAY:'; print_r($files); // email fields: to, from, subject, and so on $to = "a@c.om"; $from = "a@b.com"; $subject ="Testing Script"; $message = "My message"; $headers = "From: $from"; // boundary $semi_rand = md5(time()); $mime_boundary = "==Multipart_Boundary_x{$semi_rand}x"; // headers for attachment $headers .= "\nMIME-Version: 1.0\n" . "Content-Type: multipart/mixed;\n" . " boundary=\"{$mime_boundary}\""; // multipart boundary $message = "This is a multi-part message in MIME format.\n\n" . "--{$mime_boundary}\n" . "Content-Type: text/plain; charset=\"iso-8859-1\"\n" . "Content-Transfer-Encoding: 7bit\n\n" . $message . "\n\n"; $message .= "--{$mime_boundary}\n"; // preparing attachments for($x=0;$x<count($files);$x++){ $file = fopen($files[$x],"rb"); $data = fread($file,filesize($files[$x])); fclose($file); $data = chunk_split(base64_encode($data)); $message .= "Content-Type: {\"application/octet-stream\"};\n" . " name=\"$files[$x]\"\n" . "Content-Disposition: attachment;\n" . " filename=\"$files[$x]\"\n" . "Content-Transfer-Encoding: base64\n\n" . $data . "\n\n"; $message .= "--{$mime_boundary}\n"; } // send $ok = @mail($to, $subject, $message, $headers); if ($ok) { echo "<p>mail sent to $to!</p>"; } else { echo "<p>mail could not be sent!</p>"; } echo 'EMAIL SENT!'; } ?> Hi, I have a script where employees of a company can upload invoices to their clients. This is done via a simple upload form. My question is, What would be most secure way to do this? How should i upload files, should i make different directories for each customer, or should i have one huge, where file names will be random or, what would you do? Hi, I have a script that currently works for uploading images and I have tried to modify it to upload PDF's and DOC's too, but I can't get it to work - it works fine with JPG or GIF. Any ideas? Thanks in advance. Code: [Select] <?php include "scripts/connect.php"; $idir = "../documents/"; // Path To Images Directory if (isset ($_FILES['fupload'])){ $randomd=rand(0000,9999); //upload the image to tmp directory $url = $_FILES['fupload']['name']; // Set $url To Equal The Filename For Later Use if ($_FILES['fupload']['type'] == "image/jpg" || $_FILES['fupload']['type'] == "image/jpeg" || $_FILES['fupload']['type'] == "image/pjpeg" || $_FILES['fupload']['type'] == "image/gif" || $_FILES['fupload']['type'] == "image/pdf" || $_FILES['fupload']['type'] == "image/doc") { $file_ext = strrchr($_FILES['fupload']['name'], '.'); // Get The File Extention In The Format Of , For Instance, .jpg, .gif or .php $copy = copy($_FILES['fupload']['tmp_name'], "$idir" . "$randomd" . $_FILES['fupload']['name']); // Move Image From Temporary Location To Permanent Location } } error_reporting (E_ALL ^ E_NOTICE); if ($_POST['submit']) { $document = mysql_real_escape_string("$idir" . "$randomd" . $_FILES['fupload']['name']); $name = mysql_real_escape_string($_POST['name']); $description = mysql_real_escape_string($_POST['description']); $SQL = " INSERT INTO documents"; $SQL .= " (document, name, description) VALUES "; $SQL .= " ('$document', '$name', '$description') "; $result = mysql_db_query($db,$SQL,$cid); $last=mysql_insert_id(); if (!$result) { echo("ERROR: " . mysql_error() . "\n$SQL\n"); } header("location:document-added.php?ref=$last"); exit(); } ?>[code] I'm trying out a script that lets users upload files into a directory, the file path then should be saved on the user information in the database. This script keeps throwing "Undefined index: file" errors, even though I;m sure it should be defined. Could someone take a look please? Here's the form I'm using: Code: [Select] <form id ='change0' action ='pic_up.php' method ='post' accept-charset='UTF-8'> <fieldset > <legend>Confirm Details</legend> <input type ='hidden' name ='file' id ='file' value ='800000'/> <label for ='file' >Upload Profile Pictu </label> <input type ='file' name ='file' id ='file' /> <input type ='submit' name ='Submit' value ='Submit' /> </fieldset> </form> Here's the PHP script: Code: [Select] <?php include 'connect.php'; session_start(); $_SESSION['username']; $username = $_SESSION['username']; if(!(isset($_SESSION['login']) && $_SESSION['login']!= " ")){ header("Location: login.php"); } $tablename = 'usr_test'; $targ = "localhost/img/"; $targ = $targ . basename($_FILES['file']['name']); $file = ($_FILES['file']['name']); mysql_query("INSERT INTO $tablename (pic) VALUES ($file) WHERE usr = '$username'"); if(move_uploaded_file($_FILES['file']['tmp_name'], $targ)) { echo "File ". basename( $_FILES['uploadedfile']['name']). " has been uploaded"; } else { echo "Sorry, not happening"; } ?> Hey everyone, I'm having an intermittent problem with my upload script. It works all the time I've tested it both locally and in its live environment, and it works nearly 100% of the time in its live environment during normal usage. I've coded it so it logs any errors as and when they happen, and the log tells me that move_uploaded_file fails so returns false and that $_FILES['Filedata']['tmp_name'] is empty. I think the empty tmp_name is causing move_uploaded_file to fail, but I can't work out the cause of that problem. This is the code in question: public function upload() { if (!empty($_FILES)) { $uploadDir = 'uploads'; $subDir = uniqid(); $targetPath = $uploadDir . '/' . $subDir; $mkDir = mkdir($targetPath); $targetFile = $targetPath . '/' . $_FILES['Filedata']['name']; $tempFile = $_FILES['Filedata']['tmp_name']; $mvFile = move_uploaded_file($tempFile,$targetFile); if(!$mkDir || !$mvFile) { // Something went wrong $debug['FILES'] = $_FILES; $debug['mkdir'] = var_export($mkDir, true); $debug['mvFile'] = var_export($mvFile, true); Application_Models_Log::append($debug); echo '1'; }else { echo $targetFile; } } } The server's error_log doesn't contain anything that happened at the time of the error and as I've mentioned, I've been unable to recreate the problem. Does anybody know why the tmp_name would be empty on rare occasions? Or am I going about this in the wrong way? What else could be the problem? I need to be able to upload multiple files with the use of one form. Right now I have support for one file and it works great. I am stuck on what route I should take for times sake and reliability and functionality. Can I run each file on its own through the PHP script to upload the file; I would have to create a loop to run through the script as many times as there are files. OR Do I create new functionality and add the files through the use of an array? This is where I am getting the ARRAY idea: http://www.phpeasystep.com/phptu/2.html This is the PHP code that is submitting the image and uploading to file system. This is what I would use to loop through multiple files if I take the loop method. <? header("location: /classifieds/index.php"); echo '<html><center>'; //first lets upload any files that were selected// $date = date("m/d/y",time()); //check that we have a file if((!empty($_FILES["uploaded_file"])) && ($_FILES['uploaded_file']['error'] == 0)) { //Check if the file is JPEG image and it's size is less than 350Kb $filename = basename($_FILES['uploaded_file']['name']); $ext = substr($filename, strrpos($filename, '.') + 1); if (($ext == "jpg") && ($_FILES["uploaded_file"]["type"] == "image/jpeg") && ($_FILES["uploaded_file"]["size"] < 2500000)) { //Determine the path to which we want to save this file $newname = dirname(__FILE__).'/uploads/'.$filename; //Check if the file with the same name is already exists on the server if (!file_exists($newname)) { //Attempt to move the uploaded file to it's new place if ((move_uploaded_file($_FILES['uploaded_file']['tmp_name'],$newname))) { //strip off file path for $newname variable so path is not accessible via html// $tempnewname = explode('/', $newname); echo $tempnewname; $newname=$tempnewname[9].'/'.$tempnewname[10]; echo "It's done! The file has been saved as: ".$filename; } else { echo "Error: A problem occurred during file upload!"; } } else { //echo "Error: File ".$_FILES["uploaded_file"]["name"]." already exists";// $timestampname = str_replace('.jpg', date('j-n-Y_g:i:s').'.jpg', (basename($_FILES['uploaded_file']['name']))); $path = dirname(__FILE__).'/uploads/'; $fullname = $path.$timestampname; //strip off file path for $newname variable so path is not accessible via html// $tempnewname = explode('/', $fullname); $newname=$tempnewname[7].'/'.$tempnewname[8]; $picname=$tempnewname[8]; ((move_uploaded_file($_FILES['uploaded_file']['tmp_name'],$newname))); } } else { echo "Error: Only .jpg images under 2.5MB are accepted for upload"; } } else { echo "Error: No file uploaded"; } ?> Thanks for hte help -Beemer Hi, I am trying upload files to a remote server using CURL however It is not sending all the values in the array. $auth_local='testing123'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $auth_remote_url[$auth_no] ); curl_setopt($ch, CURLOPT_POST, TRUE); $post_array = array( "auth"=>$auth_local, "auth_e"=>$auth_e, "my_file"=>"@".$myfile, "md5"=>$md5, "check"=>"0", "upload"=>"Upload"); curl_setopt($ch, CURLOPT_USERAGENT, "Test Upload"); curl_setopt($ch, CURLOPT_POSTFIELDS, $post_array); curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); $response = curl_exec($ch ); curl_close($ch ); When I check the value of $_POST['auth'] it is blank. When I remove "my_file"=>"@".$myfile, It does read the correct value Why is this not returning a value? Thanks, mme Hi, I have a strange problem with uploading files. My script was working fine on the test server but since moving to the live server it isn't working. Here is a printout of the FILES variable: Code: [Select] Array ( [file] => Array ( [name] => filename.ppt [type] => [tmp_name] => [error] => 1 [size] => 0 ) ) Does anyone know what the problem might be? The error message that comes back is 'Invalid file'. Code: [Select] function upload_ppt_files() { echo 'print_r($_FILES) = '.print_r($_FILES).'<br />'; global $CFG; if (($_FILES["file"]["type"] == "application/vnd.ms-powerpoint") && ($_FILES["file"]["size"] > 100)) { if ($_FILES["file"]["error"] > 0) { $CFG->message = "Error code:". $_FILES["file"]["error"]; manage_ppt_files(); } else { if (file_exists("../../uploads/" . $_FILES["file"]["name"])) { $CFG->message = $_FILES["file"]["name"] . " already exists."; manage_ppt_files(); } else { move_uploaded_file($_FILES["file"]["tmp_name"], "../../uploads/" . $_FILES["file"]["name"]); $CFG->message = "File uploaded successfully."; manage_ppt_files(); } } } else { $CFG->message = "Invalid File."; manage_ppt_files(); } } Hey all, I done some research and can only seem to find solutions that involve a java applet in the browser to break up the files into parts, does anyone else know of any other pure php solutions to uploading large files and using chunks? cheers Hi I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File. What the function must do is this: Originally I have the CSV like: Code: [Select] Copyright (c) 2007 MaxMind LLC. All Rights Reserved. locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,, When I pass the CSV file to this function I got: Code: [Select] locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,, It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory. The function is: public function deleteLine($line_no, $csvFileName) { // this function strips a specific line from a file // if a line is stripped, functions returns True else false // // e.g. // deleteLine(-1, xyz.csv); // strip last line // deleteLine(1, xyz.csv); // strip first line // Assigna o nome do ficheiro $filename = $csvFileName; $strip_return=FALSE; $data=file($filename); $pipe=fopen($filename,'w'); $size=count($data); if($line_no==-1) $skip=$size-1; else $skip=$line_no-1; for($line=0;$line<$size;$line++) if($line!=$skip) fputs($pipe,$data[$line]); else $strip_return=TRUE; return $strip_return; } It is possible to refactor this function to not blow up with the 256MB PHP Memory? Give me some clues. Best Regards, I have created a page for users to upload files, but for some reason end users can't access the files. we are getting an access denied page and every time files get uploaded i have to ftp to the site and change the permissions. is there something i am missing? Thank you in advance. |