PHP - Script Can't Upload The Large Files

I need to process a large CSV file (40 MB - 300,000 rows). While I have been working with smaller files my existing code is not able to work with large files.
All I need to do is - read a particular column from file and then count total number of rows and add all the values from column.
My exisitng piece of code imports whole CSV file into an array (a class is used) and then using 'ForEach' loop, reads the required column and values into another array. Once the data is in this array i can simply sum or count it. While this served me well for smaller files, i am not able to use this approach to read a larger php file. I have already increased the memory allocated to php and max_execution_time but the script just keeps on running

I am no php expert but usualy get around with trial and error.......your thoughts and help will be greatly appreciated

Exisiting code:
Once data has been processed by initial class  (Class used is freely available and known as 'parsecsv',  available at http://code.google.com/p/parsecsv-for-php/)

After calling the class and processing the csv file:

?php
   
ini_set('max_execution_time', 3000);
      $init = array(0);                        //Initialize a dummy array, which hold value '0'
      foreach ($csv->data as $key => $col):     //Get value from array, 'Data' is an array processed by class and holds csv data
         
         $ColValue = $col[SALARY'];       //retrieves the column you want

         {     
          $SAL= $col['SALARY'];   //Column that you want to process from csv
            array_push ($init, $SAL);    // Push value into dummy array created above
            echo "<pre>"; 
         }
         endforeach;

            $total_rows = (Count($init) -1);   //Count total number of value, '-1' to remove the first initilaized value in array           
            echo "Total # of rows: ". $total_rows . "\n";
            echo "Total Sum: ". array_sum($init) . "\n";
?>
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Hey Guys,

I need a solution for uploading very large files. As I found PHP has some memory limits.
Is it even possible to upload files with a size of 4GB?

I have a flash application that talks to upload.php

Say I upload a 500mb file; it will obviously take a little while to upload. Will the max_execution_time settings cause this to fail? Its set at 60 right now and the upload is obviously taking longer than 1 minute.

I need to parse large XML files ranging in size from ~ 500 to ~ 1700 Mb.

I use XMLReader

Code: [Select]

        set_time_limit(0);

$start_time = microtime(true);


include_once 'inc/Misc.php';
include_once 'inc/Database.php';

$files = array('xml/large_file.xml');



foreach($files as $file) {

echo "\n";
echo 'Filename: '.basename($file)."\n";
echo 'Filesize: '.convert(filesize($file))."\n";
echo 'Start parsing...'."\n";
echo "\n";

$reader = new XMLReader();

$reader->open($file);


while ($reader->read()) {
    switch ($reader->nodeType) {
        case (XMLREADER::ELEMENT):
        if ($reader->localName == "element-name") {                
                $dom = new DomDocument();
                $n = $dom->importNode($reader->expand(),true);
                $dom->appendChild($n);
                $sxe = simplexml_import_dom($n);   
                $tess->file_big->insert($sxe);         
                echo "Insert done! "; benchmark();                
        }        
    }    
}

}

Everything is fine in the beginning ...
Parsed file and slowly inserted my desired data, but is gradually growing memory consumption and has run out of resources.

That is, I took the file to 400 Mb and as long as it is spent parsing of 2000 Mb of RAM and all the resources ran out and the script is stopped.

How to deal with large files? ~ 500 to ~ 1700 Mb.

Will there XML Parser? Yes, and how to apply it to my problem?

Another option could have?

In the past, whenever I write an image upload script in php that needs to generate a thumbnail or resized version, I have had to make sure the image is a reasonable size before uploading otherwise you get the old 'allowed memory bytes exceeded' thing.

What are my options if I want people to be able to upload a full size image from their camera i.e. a 15-20mb 4000x3000px image and then have a thumbnail and something like 500px wide version for displaying on the site? The large unaltered original needs to be stored as well as it will be used for prints.

Is this just not possible with PHP? Or is it down to needing a dedicated server?

I have an xml file thats about 20MB, I want to parse it all into an array.  SimpleXML seems to have trouble processing it, I just get "Fatal error: Balloc() failed to allocate memory".

I've googled this topic and can't seem to find a straight forward solution.  There must be someone here who has had to do this before.

Is there an easy way to parse this?

Going to try and explain this the best I can but I don't really have the best idea on what's happening here.

I have a submission form for users to fill out their information and upload an image. I've set the file limit size at 500000 which I assumed would be safe for images at 400k or below. When testing locally, any image that is below that file size gets uploaded successfully. However, when testing on my online host/server.. the submission form and data is successfully entered but the image isn't saved at all. It obviously isn't over the size of the file limit I set because it dooesn't return an error.. it successfuly submits but doesn't save or resize my image.

I really have no clue what the problem could be. I went over the variables I set for folder locations to move the image to and everything works fine locally, but once on the host and online, it doesn't happen.

So far I have managed to create an upload process which uploads a picture, updates the database on file location and then tries to upload the db a 2nd time to update the Thumbnails file location (i tried updating the thumbnails location in one go and for some reason this causes failure)

But the main problem is that it doesn't upload some files

Here is my upload.php

<?php
include 'dbconnect.php';

$statusMsg = '';

$Title = $conn -> real_escape_string($_POST['Title']) ; 
$BodyText = $conn -> real_escape_string($_POST['ThreadBody']) ; 
	 

// File upload path
$targetDir = "upload/";
$fileName = basename($_FILES["file"]["name"]);
$targetFilePath = $targetDir . $fileName;
$fileType = pathinfo($targetFilePath,PATHINFO_EXTENSION);
$Thumbnail = "upload/Thumbnails/'$fileName'";

if(isset($_POST["submit"]) && !empty($_FILES["file"]["name"])){
    // Allow certain file formats
    $allowTypes = array('jpg','png','jpeg','gif','pdf', "webm", "mp4");
    if(in_array($fileType, $allowTypes)){

        // Upload file to server
        if(move_uploaded_file($_FILES["file"]["tmp_name"], $targetFilePath)){

            // Insert image file name into database
            $insert = $conn->query("INSERT into Threads (Title, ThreadBody, filename) VALUES ('$Title', '$BodyText', '$fileName')");
            if($insert){
               $statusMsg = "The file ".$fileName. " has been uploaded successfully."; 

		$targetFilePathArg = escapeshellarg($targetFilePath);
		$output=null;
		$retval=null;
		 //exec("convert $targetFilePathArg -resize 300x200 ./upload/Thumbnails/'$fileName'", $output, $retval);
		 exec("convert $targetFilePathArg -resize 200x200 $Thumbnail", $output, $retval);
		echo "REturned with status $retval and output:\n" ; 
		if ($retval == null) { 
		echo "Retval is null\n" ;
		echo "Thumbnail equals $Thumbnail\n" ; 
		}

            }else{
                $statusMsg = "File upload failed, please try again.";
            } 
        }else{
            $statusMsg = "Sorry, there was an error uploading your file.";
        }
    }else{
        $statusMsg = 'Sorry, only JPG, JPEG, PNG, GIF, mp4, webm & PDF files are allowed to upload.';
    }
}else{
    $statusMsg = 'Please select a file to upload.';
}

//Update SQL db by setting the thumbnail column to equal $Thumbnail
$update = $conn->query("update Threads set thumbnail = '$Thumbnail' where filename = '$fileName'");
            if($update){
               $statusMsg = "Updated the thumbnail to sql correctly.";
		echo $statusMsg ;  } 
else 
{ echo "\n Failed to update Thumbnail. Thumbnail equals $Thumbnail" ; } 


// Display status message
echo $statusMsg;
?>

And this does work on most files however it is not working on a 9.9mb png file  which is named "test.png" I tested on another 3.3 mb gif file and that failed too?

For some reason it returns the following

Updated the thumbnail to sql correctly.Updated the thumbnail to sql correctly.

Whereas on the files it works on it returns

REturned with status 0 and output: Retval is null Thumbnail equals upload/Thumbnails/'rainbow-trh-stache.gif' Failed to update Thumbnail. Thumbnail equals upload/Thumbnails/'rainbow-trh-stache.gif'The file rainbow-trh-stache.gif has been uploaded successfully.

Any idea on why this is?

Hello All,

I have a simple upload form which I am using to upload files to Box.net using PHP Curl. It works fine for small files, but times out for larger files. Anyone have any suggestions for this?

Thanks,
Pete

Here is the code:

Code: [Select]
<html>
<body bgcolor="black">
<div align="center">
                <img src="Homepage_02.jpg" border="0" />
               
                <br>
                <br>
                <font color="#f1ca63"; font face="Arial"; font size="5">Upload</font>
                <br>
                <br>
               
                <?php
   
                                if (isset($_POST['upload'])) 
                                {
                                                if (!empty($_FILES['new_file_1']['name'])) 
                                                {
                $allowedExtensions = array("txt","csv","xml","css","doc", "docx","xls","xlsx","rtf","ppt","pdf","swf","flv","avi","wmv","mov","jpg","jpeg","gif","png");
                
                foreach ($_FILES as $file) 
                {
                    if ($file['tmp_name'] > '') 
                    {
                        if (!in_array(end(explode(".", strtolower($file['name']))),$allowedExtensions)) 
                        {
                            echo $file['name'].' is an invalid file type!<br/>';
                        }
                        else
                        {
                            $temp_name = $_FILES['new_file_1']['name'];
                            $localfile = $_FILES['new_file_1']['tmp_name'];
                            $file = fopen($localfile,'r');
                            
                            $request_url = 'https://upload.box.net/api/1.0/upload/[Token Here]/[Folder ID]';
                            $post_params['check_name_conflict_folder_option'] = urlencode('1');
                            $post_params['new_file_1'] = "@$localfile";
                            $post_params['description'] = urlencode($_POST['description']);
                            $post_params['uploader_email'] = urlencode($_POST['uploader_email']);
                            $post_params['upload'] = urlencode('upload');

                            $ch = curl_init();
                            
                            curl_setopt($ch, CURLOPT_URL, $request_url);
                            curl_setopt($ch, CURLOPT_VERBOSE, 1); 
                            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
                            curl_setopt($ch, CURLOPT_POST, true);
                            curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
                            curl_setopt($ch, CURLOPT_POSTFIELDS, $post_params);
                            curl_setopt($ch, CURLOPT_TIMEOUT, 300);
                    
                            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                            $result = curl_exec($ch);
                            curl_close($ch);
                            
                            $resultArray = explode(' ',$result);
                            
                            if($resultArray[5] != '')
                            {
                                $fileID = substr($resultArray[5],4,-1);
                                $shareName = $temp_name;
                                $link = 'http://www.box.net/shared/'.$shareName;
                            }
                            
                            $renameurl = addslashes("https://www.box.net/api/1.0/rest?action=rename&api_key=[API KEY]&auth_token=[TOKEN Here]&target=file&target_id=".$fileID."&new_name=".$shareName);
                            
                            $renameResult = file_get_contents($renameurl);
                                            
                            echo '<font color="white">Upload Successful</font>';                        
                                        
                        }
                    }
                }
                                                } 
                                                else 
                                                {
                                                                echo '<font color="white">Please select a file</font>';
                                                }
                                }
                ?>
               
                <hr width=600 color=grey>
                <br>

                <div align="center">
                                <form action="box_upload_curl.php" enctype="multipart/form-data" method="post">
                                                <input type="hidden" name="check_name_conflict_folder_option" value="1"/>
                                                                <table>
                                                                <tr>
                                                                                <td class="field" style="color: #f1ca63; font-family: Arial; font-size: 14px" width="50%">Choose File to Upload:&nbsp;&nbsp;&nbsp;</td>
                                                                                <td class="input"><input type="file" name="new_file_1" /></td>
                                                                </tr>
                                                                <tr>
                                                                                <td class="field field_top" style="color: #f1ca63; font-family: Arial; font-size: 14px" ><br/> Description (optional):</td>
                                                                                <td class="input"><br/><textarea name="description"></textarea></td>
                                                                </tr>
                                                                <tr>
                                                                                <td class="field field_top" style="color:#f1ca63; font-family: Arial; font-size: 14px" >
                                                                                                <br/>
                                                                                                Your e-mail <font color="red">*</font>:
                                                                                </td>
                                                                                <td class="input field_top" style="color: #f1ca63; font-family: Arial; font-size: 14px" >
                                                                                                <br/>
                                                                                                <input type="text" name="uploader_email" id="email_input"></input>
                                                                                </td>
                                                                </tr>
                                                                <tr>
                                                                                <td colspan="2" class="submit" align="center">
                                                                                                <br />
                                                                                                <input type="submit" name="upload" value="Upload" />
                                                                                </td>
                                                                </tr>
                                                                </table>
                                </form>
                <hr width=600 color="grey">
</div>

Hi

I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File.

What the function must do is this:

Originally I have the CSV like:
Code: [Select]
Copyright (c) 2007 MaxMind LLC.  All Rights Reserved.
locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
1,"O1","","","",0.0000,0.0000,,
2,"AP","","","",35.0000,105.0000,,
3,"EU","","","",47.0000,8.0000,,
4,"AD","","","",42.5000,1.5000,,
5,"AE","","","",24.0000,54.0000,,
6,"AF","","","",33.0000,65.0000,,
7,"AG","","","",17.0500,-61.8000,,
8,"AI","","","",18.2500,-63.1667,,
9,"AL","","","",41.0000,20.0000,,

When I pass the CSV file to this function I got:
Code: [Select]
locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
1,"O1","","","",0.0000,0.0000,,
2,"AP","","","",35.0000,105.0000,,
3,"EU","","","",47.0000,8.0000,,
4,"AD","","","",42.5000,1.5000,,
5,"AE","","","",24.0000,54.0000,,
6,"AF","","","",33.0000,65.0000,,
7,"AG","","","",17.0500,-61.8000,,
8,"AI","","","",18.2500,-63.1667,,
9,"AL","","","",41.0000,20.0000,,

It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory.

The function is:

public function deleteLine($line_no, $csvFileName) {

// this function strips a specific line from a file
// if a line is stripped, functions returns True else false
//
// e.g.
// deleteLine(-1, xyz.csv); // strip last line
// deleteLine(1, xyz.csv); // strip first line

// Assigna o nome do ficheiro
$filename = $csvFileName;

$strip_return=FALSE;

$data=file($filename);
$pipe=fopen($filename,'w');
$size=count($data);

if($line_no==-1) $skip=$size-1;
else $skip=$line_no-1;

for($line=0;$line<$size;$line++)
if($line!=$skip)
fputs($pipe,$data[$line]);
else
$strip_return=TRUE;

return $strip_return;
}


It is possible to refactor this function to not blow up with the 256MB PHP Memory?

Give me some clues.

Best Regards,

Hi everyone,

I have a page that i use to upload images to my website, i got a bit fed up of uploading one at a time so i decided to add multiple file fields to the form to upload multiple images at the same time.

Im having a few problems, iv read up he http://www.php.net/manual/en/features.file-upload.multiple.php and it seems all i have to do is add [] to the form names to turn them into arrays. However when i come to upload the images, i keep getting the "$error[] = "Incorrect format!...." error from the code below. I cant seem to figure out what the problem is.

Could anybody please point me in the right direction?


<?php 
session_start();

$id = $_SESSION['id'];

$connect = mysql_connect("localhost","leemp5_admin","p7031521");
mysql_select_db("leemp5_database");

$query = mysql_query("SELECT * FROM users WHERE id='$id'");
$row = mysql_fetch_assoc($query);

$username = $row['username'];

$submit = $_POST['submit'];

$type = $_FILES['image']['type'];
$size = $_FILES['image']['size'];
$max_size = "1000"; 
$width = "100"; 
$height = "100";
 
$error = array();

function make_thumb($image_name,$filename,$new_width,$new_height) 
{

$ext=getExtension($image_name); 

if(!strcmp("jpg",$ext) || !strcmp("jpeg",$ext)) 

$source_image=imagecreatefromjpeg($image_name);   

if(!strcmp("png",$ext)) 

$source_image=imagecreatefrompng($image_name);

if(!strcmp("gif",$ext)) 

$source_image=imagecreatefromgif($image_name); 

$old_x=imageSX($source_image);

$old_y=imageSY($source_image);   

$ratio1=$old_x/$new_width; 

$ratio2=$old_y/$new_height; 

if($ratio1>$ratio2)

{ 

$thumb_width=$new_width; 

$thumb_height=$old_y/$ratio1; 

} 

else 

{ 

$thumb_height=$new_height;

$thumb_width=$old_x/$ratio2; 

} 
  

$destination_image=ImageCreateTrueColor($thumb_width,$thumb_height);   

imagecopyresampled($destination_image,$source_image,0,0,0,0,$thumb_width,$thumb_height,$old_x,$old_y);   

if(!strcmp("jpg",$ext) || !strcmp("jpeg",$ext))

{

imagejpeg($destination_image,$filename); 

}

if(!strcmp("png",$ext)) 

{

imagepng($destination_image,$filename); 

}

if(!strcmp("gif",$ext)) 

{

imagegif($destination_image,$filename); 

}

imagedestroy($destination_image); 

imagedestroy($source_image); 
}   


function getExtension($str)
{ 

$i = strrpos($str,"."); 

if (!$i) { return ""; } 

$l = strlen($str) - $i; 

$ext = substr($str,$i+1,$l); 

return $ext; 
}   

if($submit) 
{ 

$image=$_FILES['image']['name']; 

if ($image) 

{ 

$filename = stripslashes($_FILES['image']['name']);   

$extension = getExtension($filename); 

$extension = strtolower($extension); 

if (($extension != "jpg") && ($extension != "jpeg") && ($extension != "png") && ($extension != "gif"))

{ 

$error[] = "Incorrect format! Please make sure your image is a .jpg, .jpeg, .png or .gif file.";

} 

else

{ 

$size=getimagesize($_FILES['image']['tmp_name']); 

$sizekb=filesize($_FILES['image']['tmp_name']); 
  

if ($sizekb > $max_size*1024) 

{ 

$error[] = "Your image is too big! The maximum upload size is 1MB.";

}   

else

{

$image_name=time().'.'.$extension; 
 

$newname="uploads/" . $username . "/images/".$image_name; $copied = copy($_FILES['image']['tmp_name'], $newname); 

if (!$copied) 

{ 

$error[] = "There was an error uploading your image. Please try again!";

} 

else 

{ 

$thumb_name='uploads/' . $username . '/images/thumbs/thumb_'.$image_name; 
 

$thumb=make_thumb($newname,$thumb_name,$width,$height); 

}

}

}

}

else

{

$error[] = "Please select an image to upload!";

}

if(empty($error))

{

echo "Upload Successfully!<br />"; 

echo '<img src="'.$thumb_name.'">';

mysql_query("INSERT INTO images VALUES ('','$username','$image_name','','','','','uploads/$username/images/$image_name','uploads/$username/images/thumbs/thumb_$image_name','$type','$size')");

}

else

{
    

echo implode($error);

}
}
?>
<form method="post" enctype="multipart/form-data" action="upload_images.php">
<input type="file" name="image[]" /><br />
<input type="file" name="image[]" /><br />
<input type="file" name="image[]" /><br />
<input type="file" name="image[]" /><br />
<input type="file" name="image[]" /><br />
<input type="file" name="image[]" /><br />
<input type="submit" name="submit" value="Upload">
</form>


Thanks

Hello All

This was an online test, which I didn't do very well with, and am looking for guidance of where I went wrong.

The idea is you'd have a matrix $A with values -1, 0, 1 in each position, in rows M and column N where M,N can be up to 1000. The code is evaluated $k times where $k can be up to 1000 as well.
 
Starting at $A[0][0], evaluate the matrix where -1 means you go down a row and +1 means you go right one column. A 0 value means you continue along your previous direction.

After exiting a position, that positions value is multiplied by -1: 1 = (-1); (-1) = 1; 0 = 0
Once you exit the matrix either on the right or below, you stop and start over

The goal of the code is to return how many times you exited the bottom right corner, ending up below (not to the right) the matrix; When your spot is [M], [N+1]

Here is the code I wrote:

Code: [Select]
function eval_matrix( $A, $k ) {

$x = 0;
$y = 0;
// dir is x or y for up/down //
$dir = "y";

$ball_count = 0;

    // Get columns //
    $maxX = count($A);
    // Get rows //
    $maxY = count($A[0]);
   
// for $k times
for($i=0; $i<$k; $i++) {

// while we're within the matrix boundaries
while($x < $maxX && $y < $maxY) {

// get the value of our current position
$mode = $A[$x][$y];

// evaluate the value 0,1,-1
switch($mode) {
case 0:
if($dir =="y") {
$y++;
} else {
$x++;
}
break;

case -1:
// Change position value
$A[$x][$y] = 1;
$y++;
$dir = "y";
break;

case 1:
// Change position value
$A[$x][$y] = -1
$x++;
$dir = "x";

break;
}

}
if($x == $maxX-1 && $y == $maxY) {
$ball_count++;
}

$x = 0;
$y = 0;
$dir = "y";
}
return $ball_count;
}

It may be fairly ugly, but I'm looking for help with how to properly code this type of solution. Any pointers, resources, suggestions are greatly appreciated!

I am looking to allow people to also upload pdf and doc ad txt  files can you show me how to convert this script . Thanks.
Code: [Select]
[php]

// Check to see if the type of file uploaded is a valid image type
 function is_valid_type($file) 
 {
// This is an array that holds all the valid image MIME types 

     $valid_types = array("image/jpg", "image/jpeg", "image/bmp","image/gif"); 
     if (in_array($file['type'], $valid_types)) 

         return 1; 

     return 0; 
 }

[/php]

Good day:

I have a simple uploader form to upload video files. FLV and mpg files will upload without problems. But mp4, m4v, ogg and webm files will not upload. I get this error:

Warning: copy() [function.copy]: Unable to access in /mounted-storage/home7/sub007/sc30390-PTUD/**********.com/admin123/insertvideo.php on line 32
Error Video not loaded.

I have read about adding the files in config/mimes.php but I do not see that folder or files in my hosting service (which is servage).

Any help will be appreciated.


Thanks.

Good afternoon, I wrote a script for uploading files to ftp server. When I lunch this on local server(wamp) it does upload files into ftp server. When I run the same script on my host provider(iPage) it doesn't upload files. Any advices? Have a look at the code please.   form <code> <form enctype="multipart/form-data" action="upload.php" method="post">
    <table >

        <tr>
            <td>

            <input name="userfile" type="file">
            &nbsp;
            </td>                
        </tr>                

        <tr>        
            <td >
            <input type="Submit" value="Add file">
            </td>        
        </tr>

    </table>
</form> </code>   upload.php <code> <?php
######################################################################
/*
For remote servers
*/
ini_set("max_execution_time", 100000000000);  // execution time must be bigger for larger files
$file = basename($_FILES['userfile']['name']);


//$remote_file = "/home/users/web/b1729/ipg.marcinkopecnet/remote_upload/$file";
$remote_file = "/remote_upload/$file";

/*
Define ftp server connection
*/
$ftp_server = "right_server";
$ftp_user_name = "right_username";
$ftp_user_pass = "right_password";

// set up basic connection
$conn_id = ftp_connect($ftp_server, 21) or die ("Cannot connect to host");

// turn passive mode on
ftp_pasv($conn_id, true);

// login with username and password
$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass) or die("Cannot login");

// upload a file
if ($upload = ftp_put($conn_id, $remote_file, $_FILES['userfile']['tmp_name'], FTP_BINARY))
    {
    // check upload status:
    print (!$upload) ? 'Cannot upload' : 'Upload complete';
    print "\n";    
    
    //echo "successfully uploaded $file\n";
    }
else
    {
    echo "There was a problem while uploading $file\n";
    }

// close the connection
ftp_close($conn_id);
######################################################################
?> </code>
Edited by MarcinUK, 04 September 2014 - 05:39 PM.