PHP - Form Memory

What happens if you do not unset an array before the script is done executing?

I am running through thousands of CSV files, parsing data for hundreds of thousands of customers.  It works fine for the first 5/6 hours then starts bogging down bad. 

I run about 5-10 CSVs per execution...I'm wondering if unsetting the arrays in the script would help this or not...I thought they would be unallocated after the script ends. Am I wrong?

I have a 2-D array that holds a table of information that is stored in a $_SESSION array variable. If all the information in the table is correct, then it should be added to a database. If not correct, it will be sent to a page to be modified.

There are a total of three pages:
AddInformation.php

ProcessInformation.php

EditInformation.php
 
Table array is created from a form on AddInformation.php. ProcessInformation.php checks for the validity of the information. And EditInformation.php allows you to update the errored information. A $_SESSION array that is Array[5][9] stops at ProcessInformation.php and doesn't bothering using the header('Location: EditInformation.php') that it is supposed to use. The only difference between the start memory usage and the end memory usage is of 5kb. As far as I know there is enough memory on the operating machine. Is there a way to fix this?

Edit: I do not have direct access to the server or its terminal.

I have this method that puts files into a zip file from an array list, it looks like this:
public function zip($files = array(), $maxMB = 2, $flag = ZIP_NEW){
$this->memory_size = $maxMB * 1024 * 1024;
$zip_id = $this->zip_id;
$this->zip[] = new ZipArchive();
$this->zip_id++;
if($flag == ZIP_NEW){
$open = $this->zip[$zip_id]->open("php://temp/maxmemory:$this->memory_size", ZipArchive::CREATE);
$files = (array)$files;
if(!empty($files)){
foreach($files as $file){
if(is_file($file)){
$this->zip[$zip_id]->addFile($file);
}
}
}
$this->zip[$zip_id]->close();
}
$this->function_name = __FUNCTION__;
return $this;
}

I then download them like this:
public function download($download_data, $download_name, $type = DOWNLOAD_FILE){
header("Pragma: public;");
header("Expires: 0;");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0;");
header("Content-Type: application/force-download;");
header("Content-Type: application/octet-stream;");
header("Content-Type: application/download;");
header("Content-Transfer-Encoding: binary;");
header("Content-Disposition: attachment; filename=$download_name;");
if($type == DOWNLOAD_FILE){
header("Content-Length: ".filesize($download_data).";");
if($download_data == PHP_MEMORY){
$handle = fopen(PHP_MEMORY."/maxmemory:$this->memory_size", 'r+');
echo stream_get_contents($handle);
}else{
readfile($download_data);
}
}elseif($type == DOWNLOAD_STRING){
header("Content-Length: ".mb_strlen($download_data).";");
echo $download_data;
}
$this->function_name = __FUNCTION__;
return $this;
}

When I run this code:
$files = array("file1.php","file2.php");
$live->zip($files)->download(PHP_MEMORY, "download.zip");

It downloads the file, the only problem is, is that when I open the file I get an error saying the file can not be opened. I know when I actually save it to a file instead of in memory it works. So, I am not sure if it isn't working in the download portion or the create file portion, as I have never worked with php://temp before. Any thoughts on why this doesn't work when using php://temp?

Hello Guys,
I wanted to know what is memory leakage? Where we have to takecare of ? How to avoid this thing?

Hi,
I'm getting following error on my website
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 36000000 bytes) in /var/www/vhosts/celebratingart.com/1DEV_CELEBRATINGART/judges/phpimageeditor/classes/phpimageeditor.php on line 652

I tried to increase memory limit to 256M but invain.
Please Help

I have stepped into more than one project with a function similar to the following:

Code: [Select]
<?
/**
 * xml2array() will convert the given XML text to an array in the XML structure.
 * Link: http://www.bin-co.com/php/scripts/xml2array/
 * Arguments : $contents - The XML text
 *                $get_attributes - 1 or 0. If this is 1 the function will get the attributes as well as the tag values - this results in a different array structure in the return value.
 *                $priority - Can be 'tag' or 'attribute'. This will change the way the resulting array sturcture. For 'tag', the tags are given more importance.
 * Return: The parsed XML in an array form. Use print_r() to see the resulting array structure.
 * Examples: $array =  xml2array(file_get_contents('feed.xml'));
 *              $array =  xml2array(file_get_contents('feed.xml', 1, 'attribute'));
 */
function xml2array($contents, $get_attributes=1, $priority = 'tag') {
    if(!$contents) return array();

    if(!function_exists('xml_parser_create')) {
        //print "'xml_parser_create()' function not found!";
        return array();
    }

    //Get the XML parser of PHP - PHP must have this module for the parser to work
    $parser = xml_parser_create('');
    xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8"); # http://minutillo.com/steve/weblog/2004/6/17/php-xml-and-character-encodings-a-tale-of-sadness-rage-and-data-loss
    xml_parser_set_option($parser, XML_OPTION_CASE_FOLDING, 0);
    xml_parser_set_option($parser, XML_OPTION_SKIP_WHITE, 1);
    xml_parse_into_struct($parser, trim($contents), $xml_values);
    xml_parser_free($parser);

    if(!$xml_values) return;//Hmm...

    //Initializations
    $xml_array = array();
    $parents = array();
    $opened_tags = array();
    $arr = array();

    $current = &$xml_array; //Refference

    //Go through the tags.
    $repeated_tag_index = array();//Multiple tags with same name will be turned into an array
    foreach($xml_values as $data) {
        unset($attributes,$value);//Remove existing values, or there will be trouble

        //This command will extract these variables into the foreach scope
        // tag(string), type(string), level(int), attributes(array).
        extract($data);//We could use the array by itself, but this cooler.

        $result = array();
        $attributes_data = array();
       
        if(isset($value)) {
            if($priority == 'tag') $result = $value;
            else $result['value'] = $value; //Put the value in a assoc array if we are in the 'Attribute' mode
        }

        //Set the attributes too.
        if(isset($attributes) and $get_attributes) {
            foreach($attributes as $attr => $val) {
                if($priority == 'tag') $attributes_data[$attr] = $val;
                else $result['attr'][$attr] = $val; //Set all the attributes in a array called 'attr'
            }
        }

        //See tag status and do the needed.
        if($type == "open") {//The starting of the tag '<tag>'
            $parent[$level-1] = &$current;
            if(!is_array($current) or (!in_array($tag, array_keys($current)))) { //Insert New tag
                $current[$tag] = $result;
                if($attributes_data) $current[$tag. '_attr'] = $attributes_data;
                $repeated_tag_index[$tag.'_'.$level] = 1;

                $current = &$current[$tag];

            } else { //There was another element with the same tag name

                if(isset($current[$tag][0])) {//If there is a 0th element it is already an array
                    $current[$tag][$repeated_tag_index[$tag.'_'.$level]] = $result;
                    $repeated_tag_index[$tag.'_'.$level]++;
                } else {//This section will make the value an array if multiple tags with the same name appear together
                    $current[$tag] = array($current[$tag],$result);//This will combine the existing item and the new item together to make an array
                    $repeated_tag_index[$tag.'_'.$level] = 2;
                   
                    if(isset($current[$tag.'_attr'])) { //The attribute of the last(0th) tag must be moved as well
                        $current[$tag]['0_attr'] = $current[$tag.'_attr'];
                        unset($current[$tag.'_attr']);
                    }

                }
                $last_item_index = $repeated_tag_index[$tag.'_'.$level]-1;
                $current = &$current[$tag][$last_item_index];
            }

        } elseif($type == "complete") { //Tags that ends in 1 line '<tag />'
            //See if the key is already taken.
            if(!isset($current[$tag])) { //New Key
                $current[$tag] = $result;
                $repeated_tag_index[$tag.'_'.$level] = 1;
                if($priority == 'tag' and $attributes_data) $current[$tag. '_attr'] = $attributes_data;

            } else { //If taken, put all things inside a list(array)
                if(isset($current[$tag][0]) and is_array($current[$tag])) {//If it is already an array...

                    // ...push the new element into that array.
                    $current[$tag][$repeated_tag_index[$tag.'_'.$level]] = $result;
                   
                    if($priority == 'tag' and $get_attributes and $attributes_data) {
                        $current[$tag][$repeated_tag_index[$tag.'_'.$level] . '_attr'] = $attributes_data;
                    }
                    $repeated_tag_index[$tag.'_'.$level]++;

                } else { //If it is not an array...
                    $current[$tag] = array($current[$tag],$result); //...Make it an array using using the existing value and the new value
                    $repeated_tag_index[$tag.'_'.$level] = 1;
                    if($priority == 'tag' and $get_attributes) {
                        if(isset($current[$tag.'_attr'])) { //The attribute of the last(0th) tag must be moved as well
                           
                            $current[$tag]['0_attr'] = $current[$tag.'_attr'];
                            unset($current[$tag.'_attr']);
                        }
                       
                        if($attributes_data) {
                            $current[$tag][$repeated_tag_index[$tag.'_'.$level] . '_attr'] = $attributes_data;
                        }
                    }
                    $repeated_tag_index[$tag.'_'.$level]++; //0 and 1 index is already taken
                }
            }

        } elseif($type == 'close') { //End of tag '</tag>'
            $current = &$parent[$level-1];
        }
    }
   
    return($xml_array);
}
?>
The strange thing is, this function seems to be used a lot, I even started using it. The issue however, is memory.  When the xml is anymore than a small size, it causes out of memory errors while trying to use this function. Does anyone have a similar alternative that is better on the memory?

In the following code the memory usage balloons up to 500MB

       while ($priceRow = $pricestimes->fetch_assoc()) {

            //if another product
            if ($last_product_id != $priceRow['product_id']){
                // 100 products together
                if ($count % 100 == 0) {
                    echo 'MEM AFTER 100 products- ' . (memory_get_usage(true) / 1024 ). PHP_EOL;
                    //add_price_array_to_db($pricesArr) ;
                    $pricesArr = array();
                }
                $count++;
                $last_product_id = $priceRow['product_id'] ;
            }
            $pricesArr[$priceRow['product_id']][$priceRow['insertion_timestamp']] = $priceRow['price'];

        }

The 'echo MEM AFTER 100 products-' line shows that the memory of the script keeps on increasing. Any ideas why there is a memory leak here ? The $pricestimes is a huge mysqli resultset with 23 million rows, so I suspect that may be the issue here. Unless there is something wrong with the script logic.   btw the 'add_price_array_to_db($pricesArr)' is uncommented in the real code. But even with this line commented out the memory leak is there.   Thanks
Edited by nik_jain, 17 August 2014 - 01:30 PM.

Hi.  I am working with some rather large files.  Sometimes I am just opening a file and trying to find an entry on a single line.  Sometimes I am opening a file and working on each line.   Whats the most efficient way to work with a large file?   If I open a file, and place all lines into an array, I will use a lot of memory.   If I open a file, and work on it line by line, I should use much less memory, but will the process take longer?   Whats most efficient?   I have underlined my main question above.   Thanks :-)  

I have a PHP script, that should be simple ... it just reads some filenames from a CSV file and copies the associated files to a location,  but due to my lack of skill it is HUNGRY!  This sucker eats like 400M of RAM and then dies.

What am I doing wrong?  This seems like a simple thing ..

Here's the code:

<?php

ignore_user_abort(true);

/**
 * Tell WordPress we are doing the CRON task.
 *
 * @var bool
 */
if (!defined('ABSPATH')) {
    /** Set up WordPress environment */
    require_once('./wp-load.php');
}

if (count($argv) < 2) die();

$filename = $argv[1];

$filenamearr = explode('.', $filename);
$extensionarr = array('csv', 'CSV');

if (in_array($filenamearr[count($filenamearr) - 1], $extensionarr)) {
    $destination_path = ABSPATH . "vielerets/data/";
   
    if (!file_exists($destination_path)) {
        mkdir($destination_path, 0777);
    }
   
    $target_path = $destination_path . $filename;
    $csv_target_path = $target_path;
       
   $fd = fopen ($target_path, "rt");
   
   if (!$fd) die();   
   if (!feof ($fd)) $buffer = fgetcsv($fd, 4096);
   $rowcount = 0;
   $last_postid = 0;
   $pre = -1;
   while (!feof ($fd))   {
            try{
      $buffer = fgetcsv($fd, 4096);            
      if (count($buffer) < 4) continue;
      $rowcount++;
      
      if ($pre != $buffer[0]) {
         $pre = $buffer[0];
         //The Query
         query_posts('meta_key=sysid&meta_value='.$buffer[0]);

         //The Loop
         if ( have_posts() ) {
            the_post();
            $last_postid = get_the_ID();
            echo "update property images: $last_postid\n";
            wp_reset_query();
         }
         else {
            wp_reset_query();
            continue;
         }
      }
      
      $menu_order = 0;
      $image_folder_name = 'property/';
      
      if($buffer[3])
      {
         $image_name = $buffer[3];
         $menu_order = $buffer[1];
         $image_name_arr = explode('/',$image_name);
         $img_name = $image_name_arr[count($image_name_arr)-1];
         $img_name_arr = explode('.',$img_name);
         $post_img = array();
         $post_img['post_title'] = $img_name_arr[0];
         $post_img['post_status'] = 'attachment';
         $post_img['post_parent'] = $last_postid;
         $post_img['post_type'] = 'attachment';
         $post_img['post_mime_type'] = 'image/jpeg';
         $post_img['menu_order'] = $menu_order;
         $last_postimage_id = wp_insert_post( $post_img );
         update_post_meta($last_postimage_id, '_wp_attached_file', $image_folder_name.$image_name);               
         $post_attach_arr = array(
                        "width"   =>   580,
                        "height" =>   480,
                        "hwstring_small"=> "height='150' width='150'",
                        "file"   => $image_folder_name.$image_name,
                        );
         wp_update_attachment_metadata( $last_postimage_id, $post_attach_arr );
                        unset($post_img);
                        unset($post_attach_arr);
         echo " add images: $img_name\n";
      }
                unset($buffer);
            } catch (Exception $ex) {
                echo "$rowcount\n";
            }
   }
   echo "image csv uploaded successfully\n";   
   echo "Total of $rowcount records inserted\n";
   }
?>

Hi all,

When I run the script from browser, it loads fine but when I want it to run from cronjob, it gives memory exhausted error each time cron runs. What do you think about this? What are your solutions?

Here is the error:
"Allowed memory size of 67108864 bytes exhausted (tried to allocate 5307694 bytes) in ...."

Here are the server limits:
max_execution_time = 7200
max_input_time = 600
memory_limit = 1024M

Thanks.

Hello everyone,
I made this website using CakePHP, and it was working great, but then suddenly it started giving me this error

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 38 bytes) in /cake/libs/model/datasources/dbo/dbo_mysql.php on line 766

Here is the code that contains line number 766

Code: [Select]
/**
 * Fetches the next row from the current result set
 *
 * @return unknown
 */
function fetchResult() {
if ($row = mysql_fetch_row($this->results)) {
$resultRow = array();
$i = 0;
foreach ($row as $index => $field) {
list($table, $column) = $this->map[$index];
$resultRow[$table][$column] = $row[$index];
$i++;
}
return $resultRow;
} else {
return false;
}
}

/**

Hi, I am trying to create a login system in PHP, but I am not the greatest at PHP so I am using a source code which I found online as I found it to be more secure as it uses things like salted passwords. Anyway I am trying to add more fields to the register system so it adds them to the mysql, the source has a way to do this with arrays, but it is quite complicated so I am just using variables from the original file. There are 2 files: register.php and class.loginsys.php which contains all the functions. At first the query syntax was incorrect so I decided to use the variables created in register.php in the class.loginsys, but now it's giving me an out of memory error:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 28672 bytes) in C:\xampp\htdocs\ls\class.loginsys.php on line 34    Which I am unsure of how to fix. I have tried using different variable names, checking the line, checking the whole register.php file for anything rogue. Here is the code:  Top part of register.php 

<?php
include "config.php";
?>

Config.php: 

<?php
require "class.loginsys.php";
$LS=new LoginSystem();
?>

Then actual register part from register.php: 

<?php
	   	if( isset($_POST['submit']) ){
	   			$firstname2 = $_POST['firstname'];
	   			$lastname2 = $_POST['lastname'];
	    		$user2	 = $_POST['username'];
	    		$sex2 = $_POST['sex'];
	    		$country2 = $_POST['strCountryChoice'];
	    		$email2 = $_POST['email'];
	    		$pass2	 = $_POST['pass'];
	    		$pass3 = $_POST['pass2'];
	    		$birthdate2 = $_POST['birthdate'];
	    		$created2 = date("Y-m-d H:i:s");
	    		//need to add a lot more validation functions.. AKA Check if email exists and username. Password > 5 chars
	    		if( $user2=="" || $email2=="" || $pass2=='' || $pass3=='' || $firstname2=='' || $lastname2=='' || $sex2=='' || $country2=='' || $birthdate2=='' ){
	     			echo "Fields Left Blank","Some Fields were left blank. Please fill up all fields.";
	     			exit;
	    		}
	    		if( !$LS->validEmail($email2) ){
	     			echo "E-Mail Is Not Valid", "The E-Mail you gave is not valid";
	     			exit;
	    		}
	    		if( !ctype_alnum($user2) ){
	     			echo "Invalid Username", "The Username is not valid. Only ALPHANUMERIC characters are allowed and shouldn't exceed 10 characters.";
	     			exit;
	    		}
	    		if($pass2 != $pass3){
		     		echo "Passwords Don't Match","The Passwords you entered didn't match";
	     			exit;
	    		}
   	 		$createAccount2 = $LS->register($user2, $pass2,
	    			array(
	    				"email" 	 => $email2,
	    				"name" 	 => $firstname2,
	    				"lastname" => $lastname2,
	    				"gender" => $sex2,
	    				"country" => $country2,
	    				"DOB" => $birthdate2,
	    				"created" => date("Y-m-d H:i:s") // Just for testing

	    			)
	    		);
			//$createAccount = $LS->register($firstname,$lastname,$user,$sex,$country,$email,$pass,$birthdate,$created);
	    		if($createAccount2 === "exists"){
	     			echo "User Exists.";
	    		}elseif($createAccount2 === true){
	     			echo "Success. Created account.";
	    		}
	   	}
	   	?>

And the function from the class: 

	/* A function to register a user with passing the username, password and optionally any other additional fields. */
	public function register( $id, $password, $other = array() ){
		if( $this->userExists($id) && (isset($other['email']) && $this->userExists($other['email'])) ){
			return "exists";
		}else{
			$randomSalt	= $this->rand_string(20);
			$saltedPass	= hash('sha256', "{$password}{$this->passwordSalt}{$randomSalt}");
			
			if( count($other) == 0 ){
				/* If there is no other fields mentioned, make the default query */
				//old query: ("INSERT INTO `{$this->dbtable}` (`username`, `password`, `password_salt`) VALUES(:username, :password, :passwordSalt)");
				//new query: ("INSERT INTO `{$this->dbtable}` (`username`, 'email' , `password`, `password_salt` , 'name' , 'lastname' , 'gender' , 'country' , 'DOB') VALUES(:username, :email, :pass, :passwordSalt, :firstname, :lastname, :gender, :country, :DOB)");
				$sql = $this->dbh->prepare("INSERT INTO `{$this->dbtable}` (`username`, `password`, `password_salt`) VALUES(:username, :password, :passwordSalt)");
			}else{
				/* if there are other fields to add value to, make the query and bind values according to it */
				//old query: ("INSERT INTO `{$this->dbtable}` (`username`, `password`, `password_salt`, $columns) VALUES(:username, :password, :passwordSalt, :$colVals)");
				//new query: ("INSERT INTO `{$this->dbtable}` (`username`, 'email' , `password`, `password_salt` , 'name' , 'lastname' , 'gender' , 'country' , 'DOB') VALUES(:username, :email, :pass, :passwordSalt, :firstname, :lastname, :gender, :country, :DOB)");
				$keys	 = array_keys($other);
				$columns = implode(",", $keys);
				$colVals = implode(",:", $keys);
			//l= $this->dbh->prepare("INSERT INTO `{$this->dbtable}` (`username`, `password`, `password_salt`, $columns) VALUES(:username, :password, :passwordSalt, :$colVals)");
				//INSERT INTO MyGuests (firstname, lastname, email)cLUES ('John', 'Doe', 'john@example.com')
				$sql = $this->dbh->prepare("INSERT INTO `{$this->dbtable}` (username,email,password,password_salt,name,lastname,created,gender,country,DOB) VALUES ('$username2','$email2','$pass2','$saltedPass','$firstname2','$lastname2','$created2','$gender2','$country2','$birthdate2')");
				print($sql);
				foreach($other as $key => $value){
					$value = htmlspecialchars($value);
					$sql->bindValue(":$key", $value);
				}
			}
			/* Bind the default values */
			$sql->bindValue(":username", $id);
			$sql->bindValue(":password", $saltedPass);
			$sql->bindValue(":passwordSalt", $randomSalt);
			$sql->execute();
			return true;
		}
	}
	

Thanks for your help. I am doing this because for a hobby I am trying to create a browser based game in which I use this login system to login the user to a main page then code all of the other pages myself. I have posted on stackoverflow and someone on their suggested that I should use a framework. If this is the case, can someone point me in the right direction?   Thanks again, if you need any info ask.

// in this case, $teams_count = 16
for($j = $tournament->teams_count; $j != 1; $j/2)
{
echo $j;
}


What I expect:
16 | 8 | 4 | 2 | 1

What I get: memory exhaustion.
Code: [Select]
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 128716801 bytes)

Is it so late, I'm missing something?

For example, take a look at this: https://code.google....p/phpwebsocket/   When I run the server, it works fine and I am using the server for my Inventory system on my game. When users drag and drop items in their inventory to change position, it sends the data to the html 5 websocket server, and updates the positions accordingly/etc. And I will use it for a bunch of other stuff later.   My question is: What kind of VPS would I be looking for in terms of ram/cpu power to keep a websocket server like this up 24/7? Is it more memory intensive than socket/io/nodejs? Or what? How can I test these things? Looking at cmd.exe or php.exe in my task manager when having it run on my local windows machine isn't really a good way to test performance, or to estimate how much ram or VPS/CPU power of a server I'll need before launch.  I can get a idea, but I don't think it's going to be close to reality. That's why I'm asking you guys here if you would know.  Thanks
Edited by Monkuar, 17 January 2015 - 12:24 AM.

Hello,

So I am trying to reduce my memory footprint to the lowest possible value without sacrificing functionality.

I noticed that if I write a comment that is 2kb long, it costs me 2kb more memory to run the script.

How can I get PHP to free the memory taken up by comments? It seems like it would be a real pain in the butt to have to run php -w on all my source files each time I update my server.

Any ideas?

Thanks.

David