PHP - Efficiently Allocating Resources For Large Import Of Images
I'm hoping to get a little feedback on what you all believe is the best way to handle this efficiently in PHP. I am working on a script that imports a large amount of data from remote feeds; this facilitates the quick deployment of real estate web sites, but has to download a large number of images to each new site.
Assuming for right now that the bottleneck isn't in the method (fsock vs curl vs...) and that for each imported listing we're spending between .89439 and 17.0601 seconds on the image import process alone... what would you suggest for handling this over the space of 100-1000 occurrences? As of right now I have two ideas in mind, both fairly rudimentary in nature. The first idea is to shut the script down every 30-45 seconds, sleep for a second and fire off another asynchronous request to start the script again. The second idea is to fire off a new asynchronous to run the image imports separate from the main script. This would let the efficient ones clear out rather quickly while the slower imports would have their own process to run in. The only thing that worries me about this is the fact that 100 of these could be fired off every second. Even assuming half of them complete before the next round are fired off, they would still pile up. Similar TutorialsHi all So... I am creating an import script for putting contacts into a database. The script we had worked ok for 500kb / 20k row CSV files, but anything much bigger than that and it started to run into the max execution limit. Rather than alter this I wish to create something that will run in the background and work as efficiently as possible. So basically the CSV file is uploaded, then you choose if the duplicates should be ignored / overwritten, and you match up the fields in the CSV (by the first line being a field title row), to the fields in the database. The field for the email address is singled out as this is to be checked for duplicates that already exist in the system. It then saves these values, along with the filename, and puts it all into an import queue table, which is processed by a CRON job. Each batch of the CRON job will look in the queue, find the first import that is incomplete, then start work on that file from where it left off last. When the batch is complete it will update the row to give a pointer in the file for the next batch, and update how many contacts were imported / how many duplicates there were So far so good, but when checking for duplicity it is massively slowing down the script. I can run 1000 lines of the file in 0.04 seconds without checking, but with checking that increases to 14-15 seconds, and gets longer the more contacts are in the db. For every line it tries to import its doing a SELECT query on the contact table, and although I am not doing SELECT * its still adding up to a lot of DB activity. One thought was to load every email address in the contacts table into an array before hand, but this table could be massive so thats likely to be just as inefficient. Any ideas on optimising this process? My first post.
I am trying to import images related to a particular listing ID. Currently, I have to enter the number of available photos into a db table, and run the following loop;
<section id="photos">
<?php
$b = $row['PIX'];
for ( $a = 01; $a <= $b; $a++ ) { echo "<a href=\"../imgs/{ $row[ 'ID' ] }-" . { $a } . ".jpg\" ><img src=\"../imgs/{ $row[ 'ID' ] }-" . { $a } . "_tn.jpg\" /></a> " ; } ?> </section> I would like to find a way to import all the images that begin with the $row[ 'ID' ], but without me having to enter/store { $b } (the number of images recorded in the db). How can I accomplish this? _ I Can't resize large images with imagecreatefromjpeg() I can load small 38kb images fine, when they get up 780+- or 1.3 mb +- (with a width of 2500px * x) I get the below error I also can upload the same pics in another file with out resizing(using imagecreatefromjpeg()) them and the script works fine. my max file upload size with xampp is 128mb / php5 Warning: imagecreatefromjpeg() [function.imagecreatefromjpeg]: gd-jpeg, libjpeg: recoverable error: Corrupt JPEG data: 191 extraneous bytes before marker 0xd9 in C:\xampp\htdocs\ed\phpsol\ch08\work\includes\create_thumb.inc.php on line 35 Warning: imagecreatefromjpeg() [function.imagecreatefromjpeg]: 'C:\xampp\tmp\php9596.tmp' is not a valid JPEG file in C:\xampp\htdocs\ed\phpsol\ch08\work\includes\create_thumb.inc.php on line 35 I'm basicly using a switch switch($type) { case 1: $source = @ imagecreatefromgif($original); if (!$source) { $result = 'Cannot process GIF files. Please use JPEG or PNG.'; } break; case 2: $source = imagecreatefromjpeg($original); <---- LINE 35 where $original is $original = $_FILES['image']['tmp_name']; break; In the past, whenever I write an image upload script in php that needs to generate a thumbnail or resized version, I have had to make sure the image is a reasonable size before uploading otherwise you get the old 'allowed memory bytes exceeded' thing. What are my options if I want people to be able to upload a full size image from their camera i.e. a 15-20mb 4000x3000px image and then have a thumbnail and something like 500px wide version for displaying on the site? The large unaltered original needs to be stored as well as it will be used for prints. Is this just not possible with PHP? Or is it down to needing a dedicated server? i have my upload process working that uploads documents to the server and then displays in onto the user page like this:
<?php function find_all_files($dir) { $root = scandir($dir); foreach($root as $value) { if($value === '.' || $value === '..') {continue;} if(is_file("$dir/$value")) {$result[]="$dir/$value"; continue; } foreach(find_all_files("$dir/$value") as $value) { $result[]=$value; } } return $result; } $fileupload = 'fileupload'; $getem = find_all_files($fileupload); foreach($getem as $key => $value) { echo '<a href="'.$value.'">'.$value.'</a><br />'; } ?> <?php if($handle = opendir('members/')) { while (false !== ($entry = readdir($handle))) { if($entry != "." && $enrty != "..") { echo "<a href='download.php?file=".$entry."'>".$entry."</a>\n"; } } closedir($handle); } ?>here they can download the files to their computer however is there a way to only display the file of the user that is logged in through their session? I'll start by apologizing for the stupid decision that led to this question. A few years ago, I created a PHP/Myysql site with a login system and I created a field in the MySQL called "password" and it stored literally the exact password people entered (I know, I know).
The site has proven to have nice traffic potential, so I am going to re-vamp everything, including storing passwords properly (i.e. hashed).
My first question... Is there a way to convert regular text passwords to hashed passwords? For example, I could create a new field in the "User" table for "hashedpassword" and write a script that takes all the insecure passwords and turns them into hashed passwords. Then deleted the previous "bad" password field from the database. This would allow me to do it without the customer every knowing anything changed.
Quick googling appears to support that it IS doable rather easily, with something like...
UPDATE mytable SET password = MD5(password)If not, I guess I would have to create a thing where the first time omeone logged in after I put hashing in place, the site would force them to change their password. I'd rather not annoy the visitors if it all possible. Second question, what is the proper/recommended hashing method to use? Some people seem to poo-poo MD5. If you agree, should I use: MD5 SHA MD5 with a salt SHA with a salt Something else i never heard of NOTE: My site is a fantasy sports site, so the data involved is not overly important. Maybe a salt is overkill? Or is being overly safe never a bad thing? Lastly, don't need to address this, but if anyone can explain it like I'm 5 that would be great because i must be missing something... if you can easily turn a regular password into a hashed password, couldn't hackers easily do the reverse, which would render the hashing almost useless? I get that salting helps, but before salting (i.e. doing ONLY MD5), I don't see how hashing helped that much (if you could reverese figure out the password). What am I missing? Thanks! Greg Edited by galvin, 13 November 2014 - 09:44 AM. PHP Websockets Rate limiter (Thanks to @Kicken for the help) How do I setup let's say... no more than 5 requests per second? Any idea? // anti flood protection if (isset($users[$clientID]['last_session_request'])) { if($users[$clientID]['last_session_request'] > ''.microtime(true).'' - 1){ // users will be redirected to this page if it makes requests faster than 1 seconds echo "Limit Reached... Simmer down!"; wsClose($clientID); }else{ } $users[$clientID]['last_session_request'] = microtime(true); } Edited by Monkuar, 20 December 2014 - 03:28 PM. Since I'm new to programming I still don't know how to solve problems the most efficient way, I'd like to have your advice on this one. This is how I solved it and it works. I have my categories as input buttons (I'm planning to change it into a drop down menu later). Sort by Category: <form action="<?php echo $_SERVER['PHP_SELF']; ?>" method="post"> <input type='submit' name='All' value='All' /> <input type='submit' name='Smileys' value='Smileys' /> <input type='submit' name='Faces' value='Faces' /> <input type='submit' name='Love' value='Love' /> </form> When the input boxes are clicked the corresponding script below is being run, very simple and to a point primitive, definitely not really efficient, because if you have a lot of categories you'll end up with a lot of code and if you wanted to change one thing you'd have to change it in all the IF BLOCKS. Basically every input button has its own IF BLOCK. <?php if (isset($_POST['All']) OR (!isset($_POST['Smileys']) && !isset($_POST['Faces']) && !isset($_POST['Love']) { // Connect to the database $dbc = mysqli_connect(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME); // Retrieve the score data from MySQL $query = "SELECT * FROM asciiart"; $data = mysqli_query($dbc, $query); // Loop through the array of score data, formatting it as HTML while ($row = mysqli_fetch_array($data)) { echo '<table class="asciiartTable">'; // Display the score data echo '<tr><td class="asciiart_name">'; echo '<strong>' . htmlentities($row['asciiart_name']) . '</strong><br /></td></tr>'; echo '<tr><td class="asciiart_contribution"><pre>' . htmlentities($row['asciiart_contribution']) . '</pre><br /></td></tr>'; echo '<tr><td class="asciiart_categoryDate">' . htmlentities($row['asciiart_category']) . ' | ' . date('M d, Y', strtotime ($row['created_date'])) . ' </td></tr>'; echo '</table>'; } mysqli_close($dbc); } if (isset($_POST['Smileys'])) { // Connect to the database $dbc = mysqli_connect(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME); // Retrieve the chosen category from MySQL $query2 = "SELECT * FROM asciiart WHERE asciiart_category = 'Smileys'"; $data2 = mysqli_query($dbc, $query2); //Loop through the array of data while ($row2 = mysqli_fetch_array($data2)) { echo '<table class="asciiartTable">'; // Display the score data echo '<tr><td class="asciiart_name">'; echo '<strong>' . htmlentities($row2['asciiart_name']) . '</strong><br /></td></tr>'; echo '<tr><td class="asciiart_contribution"><pre>' . htmlentities($row2['asciiart_contribution']) . '</pre><br /></td></tr>'; echo '<tr><td class="asciiart_categoryDate">' . htmlentities($row2['asciiart_category']) . ' | ' . date('M d, Y', strtotime ($row2['created_date'])) . ' </td></tr>'; echo '</table>'; } mysqli_close($dbc); } if (isset($_POST['Faces'])) { // Connect to the database $dbc = mysqli_connect(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME); // Retrieve the chosen category from MySQL $query3 = "SELECT * FROM asciiart WHERE asciiart_category = 'Faces'"; $data3 = mysqli_query($dbc, $query3); //Loop through the array of data while ($row3 = mysqli_fetch_array($data3)) { echo '<table class="asciiartTable">'; // Display the score data echo '<tr><td class="asciiart_name">'; echo '<strong>' . htmlentities($row3['asciiart_name']) . '</strong><br /></td></tr>'; echo '<tr><td class="asciiart_contribution"><pre>' . htmlentities($row3['asciiart_contribution']) . '</pre><br /></td></tr>'; echo '<tr><td class="asciiart_categoryDate">' . htmlentities($row3['asciiart_category']) . ' | ' . date('M d, Y', strtotime ($row3['created_date'])) . ' </td></tr>'; echo '</table>'; } mysqli_close($dbc); } if (isset($_POST['Love'])) { // Connect to the database $dbc = mysqli_connect(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME); // Retrieve the chosen category from MySQL $query4 = "SELECT * FROM asciiart WHERE asciiart_category = 'Love'"; $data4 = mysqli_query($dbc, $query4); //Loop through the array of data while ($row4 = mysqli_fetch_array($data4)) { echo '<table class="asciiartTable">'; // Display the score data echo '<tr><td class="asciiart_name">'; echo '<strong>' . htmlentities($row4['asciiart_name']) . '</strong><br /></td></tr>'; echo '<tr><td class="asciiart_contribution"><pre>' . htmlentities($row4['asciiart_contribution']) . '</pre><br /></td></tr>'; echo '<tr><td class="asciiart_categoryDate">' . htmlentities($row4['asciiart_category']) . ' | ' . date('M d, Y', strtotime ($row4['created_date'])) . ' </td></tr>'; echo '</table>'; } mysqli_close($dbc); } My question is: is there a way to have just one block of script and the script automatically inserts the right INPUT BUTTON into the query in the script? Something similar to this, even though it doesn't work I'm just showing it for showcase purposes: if (isset($_POST['Smileys']) || (isset($_POST['Faces']) || (isset($_POST['Love'])) { // Connect to the database $dbc = mysqli_connect(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME); // Retrieve the chosen category from MySQL $query2 = "SELECT * FROM asciiart WHERE asciiart_category = 'Smileys' || 'Faces' || 'Love'"; $data2 = mysqli_query($dbc, $query2); Notice in the query, it is looking for THAT keyword that has been PRESSED on the INPUT BUTTON. How would I be able to create a logic like this so I can create a whole chain of categories? Thanks for advice. This topic is here for those looking for help on frequently asked questions, or just need some direction in where to find more information. Table of Contents for FAQs Can you do / write _________ for me? Call to undefined function "mysql_connect" I'm getting a "headers already sent error". What does that mean? What is the point of MD5 / SHA1 / SHA256 / etc.? Should I salt my user's passwords, and why? mysql* expects parameter 1 to be resource, boolean given I've asked this question before but still haven't really figured out how best to implement it, and have found myself hitting another roadblock. I have multiple types of charts (i.e. bar, pie, etc), and various endpoints to access them. I like to keep my routing script uncluttered, and feel the following does so. $app->get('/chart', function (Request $request, Response $response) { //List of chart with optional filtering $this->chartService->index($request->getQueryParams()); return $this->chartResponder->index($response, $index); }); $app->post('/chart', function (Request $request, Response $response) { //Create a new chart of type (i.e. bar, pie, etc) specified by type parameter $chart=$this->chartService->create($request->getParsedBody()); return $this->chartResponder->create($response, $chart); }); $app->get('/chart/{id:[0-9]+}', function (Request $request, Response $response, $args) { //View chart of given ID $chart=$this->chartService->detail((int)$args['id']); return $this->chartResponder->delete($response, $chart); }); $app->delete('/chart/{id:[0-9]+}', function (Request $request, Response $response, $args) { //Delete chart of given ID $this->chartService->delete((int)$args['id']); return $this->chartResponder->delete($response, null); }); $app->post('/chart/{id:[0-9]+}/series', function (Request $request, Response $response, $args) { //Add a new series to the collection for chart with given ID $chart=$this->chartService->addSeries((int)$args['id'], $request->getParsedBody()); return $this->chartResponder->update($response, $chart); }); $app->put('/chart/{id:[0-9]+}/series/{seriesPosition:[0-9]+}', function (Request $request, Response $response, $args) { //Modify series of given position in the collection for chart with given ID $chart=$this->chartService->updateSeries((int)$args['id'], (int)$args['seriesPosition'], $request->getParsedBody()); return $this->chartResponder->update($response, $chart); }); //More endpoints for chart of given ID... I then created the following service to support the endpoints. Off topic, but is this a service or a controller? My index() method is chart type agnostic. My create() method needs a means to determine what type of chart to create, and does so using the received chart "type" passed in the body to get the applicable repository. All the other methods receive the chart ID in the URL path and use it to first get the chart entity and then get the applicable repository based on the object. All seems good! <?php namespace NotionCommotion\ChartBuilder\Service; use NotionCommotion\ChartBuilder\Entity\Chart; class ChartService { protected $em; public function __construct(\Doctrine\ORM\EntityManager $em) { $this->em = $em; } public function index(array $params=[]):array { return $this->em->getRepository(Chart::class)->index($params); } public function create(array $params):Chart { //Get the specific repo based on $params['type]. Not perfect, but good enough $discriminatorMap=$this->em->getClassMetadata(Chart::class)->discriminatorMap; $repo = $this->em->getRepository($discriminatorMap[$params['type']]); //Validate data $chart=$repo->create($params); $this->em->persist($chart); $this->em->flush(); return $chart; } public function read(int $id):Chart { return $this->em->getRepository(Chart::class)->find($id); } public function delete(int $id):void { $this->em->remove($this->read($id)); $this->em->flush(); } public function addSeries(int $idPublic, array $id):Chart { $chart=$this->read($id); //Validate data $repo=$this->em->getRepository(get_class($chart)); $repo->addSeries($chart, $params); $this->em->persist($chart); $this->em->flush(); return $chart; } public function updateSeries(int $id, int $position, array $params):Chart { $chart=$this->read($id); $series=$chart->getSeries(); $seriesNode=$series->offsetGet($position); //Validate data $repo=$this->em->getRepository(get_class($seriesNode)); $repo->update($seriesNode, $params); $this->em->persist($chart); $this->em->flush(); return $chart; } } Until... I find myself needing to put non-database related functionality in the repository and violating the single responsibility principle. I am not a complete purist and might be willing to do so, however, there does not appear to be a clean way to inject dependencies in a Doctrine repository. One thought I had was to create specialized services maybe as follows: $c['chartService'] = function ($c) { return new ChartService( $c[EntityManager::class], [ 'bar'=>function ($c) {return new BarChartService($c[EntityManager::class]);}, 'pie'=>function ($c) {return new PieChartService($c[EntityManager::class], $c['someOtherObject']);}, //add more types... ] ); };
class ChartService { protected $em, $subServices=[]; public function __construct(\Doctrine\ORM\EntityManager $em, $subServices) { $this->em = $em; $this->subServices = $subServices; } public function index(array $params=[]):array {/* no change */} public function create(array $params):Chart {/* maybe no change */} public function __call($name, $args) { $chart=$this->em->getRepository(Chart::class)->find($args[0]); $subservice=$this->getSubservice($chart); $args[0]=$chart; return $subservice->$name(...$args); } } abstract class AbstractSpecificChartService { protected $em; public function __construct(\Doctrine\ORM\EntityManager $em) { $this->em = $em; } public function __call($name, $args) { throw new \Exception("Method $name not supported"); } protected function getSubservice(Chart $chart):self { //Haven't figured out but can do so if needed. } public function delete(Chart $chart):void { //Include methods common to all charts here $this->em->remove($chart); $this->em->flush(); } protected function helperMethods($foo) { //If necessary. } } class BarChartService extends AbstractSpecificChartService { //Override __construct if necessary public function updateSeries(Chart $chart, int $position, array $params):Chart { //Note that Chart and not $id is passed. //implement code as needed return $chart; } }
So, after this long story, how should I implement this? Thank you This topic has been moved to mod_rewrite. http://www.phpfreaks.com/forums/index.php?topic=347820.0 Having a find and replace nightmare and dumbfounded by a way to correct this. I am exporting a large inner join SQL statement in CSV format. I then want to be able to edit and replace values as needed from the other tables. ex: Orlando = 1 Miami = 2 Jacksonville = 3 Well if I want to go through and replace everything 'Orlando' with the id 1, so my inner joins dont break. I am doing this in the following method. // read the file $file = file_get_contents('data.csv'); // replace the data $result = $db->query('SELECT id_baseprices, baseprices FROM baseprices'); while ($line = $db->fetchNextObject($result)) { $file = str_replace(',"'.$line->baseprices.'",', ',"'.$line->id_baseprices.'",', $file); } $result = $db->query('SELECT id_cities, cities FROM cities'); while ($line = $db->fetchNextObject($result)) { $file = str_replace(',"'.$line->cities.'",', ',"'.$line->id_cities.'",', $file); } // write the file file_put_contents('revamped.csv', $file); Problem with the above is it replaces "Orlando" everywhere with 1. I really only want this to happen inside column 8. I am lost and google must hate me tonight, so thank you for any thoughts or comments. Hello, I'm having trouble with the following code $file = file_get_contents("database.sql"); $queries = explode(";",$file); for ($i=0,$c=count($queries);$i<$c;$i++){ mysql_query($queries[$i],$mlink); } the problem is the sql file I'm trying to import has ; alllll over the place, not just at the end of the queries, so the queries are not getting inputted correctly. I could see a way to fix this if each query was on a single line, but some of them are over 100 lines each. How do I make sure it only explodes using the ; at the end of query? and not the ; that are actually in the tables etc. thanks Hi, I have two xml files, one has 750 products, and the other one has 900 products (including product informations, stock information, sizes and photos) so shortly one product in xml file is like this;
product_name: x t-shirt And my tables are like this:
Product (product_id (numerical), product_code (alphanumerical), name) (So that I need to insert one product in my db, and take the id, and insert the xml-photos-data with the correct product_id to photos table, same goes with the stocks table) So I m thinking about the easiest and most healthy way to import this amount of data, and I need to update it daily or weekly (planning to do it by cron jobs for now). I have found some solutions but since I have not done this before could not decide which is the most secure way. Need to add that I can have other xml files from different companies to import later on, so I think I should code for long term. Thank you in advance for your help. Edited December 17, 2018 by RommeoI have mysql exported in the following sql format: Code: [Select] - CREATE TABLE IF NOT EXISTS `pma_history` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `username` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `db` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `table` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `timevalue` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `sqlquery` text COLLATE utf8_bin NOT NULL, PRIMARY KEY (`id`), KEY `username` (`username`,`db`,`table`,`timevalue`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='SQL history for phpMyAdmin' AUTO_INCREMENT=3761 ; -- -- Dumping data for table `pma_history` -- INSERT INTO `pma_history` (`id`, `username`, `db`, `table`, `timevalue`, `sqlquery`) VALUES (2, 'root', 'information_schema', 'FILES', '2010-03-04 19:26:33', 'SELECT * FROM `FILES`') Just as an example anyway. What I need to be able to do, is import this using PHP. Does anyone know of a suitable class for doing this? I could make my own, but I cant quite work out the best way to seperate the queries out? Thanks Hi, i have a script to import a csv into a database but some of my product descriptions have ' in the names, this causes mysql to error, how can i get around this? here is my code. Code: [Select] if(isset($_POST['submit'])) { $i=0; $fname = $_FILES['sel_file']['name']; $chk_ext = explode(".",$fname); if(strtolower($chk_ext[1]) == "csv") { mysql_query("truncate table products") ; $filename = $_FILES['sel_file']['tmp_name']; $handle = fopen($filename, "r"); while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) { if($i > 0) { $sql = "INSERT into products(prod_id,prod_name,prod_description,prod_cat,prod_sub_cat,tax,prod_price,active) values('$data[0]','$data[1]','$data[9]','$data[14]','$data[15]','$data[19]','$data[35]','$data[65]')"; mysql_query($sql) or die(mysql_error()); } $i++; } fclose($handle); echo "Successfully Imported<br>"; } else { echo "Invalid File"; } } I have a table I am trying to import to mySQL I am having trouble assigning the right variable which represents a column header to the correct column header in the mySQL table. <?php /********************************/ /* Code at http://legend.ws/blog/tips-tricks/csv-php-mysql-import/ /* Edit the entries below to reflect the appropriate values /********************************/ $databasehost = "xxx"; $databasename = "xxx"; $databasetable = "names"; $databaseusername ="xxx"; $databasepassword = "xxx"; $fieldseparator = "\t"; $lineseparator = "\n"; $csvfile = "AddEnt2.txt"; $con = @mysql_connect($databasehost,$databaseusername,$databasepassword) or die(mysql_error()); @mysql_select_db($databasename) or die(mysql_error()); $fcontents = file ('AddEnt2.txt'); # expects the csv file to be in the same dir as this script for($i=0; $i<sizeof($fcontents); $i++) { $line = trim($fcontents[$i],'\t'); // '\t' for tab delimeted $arr = explode('\t', $line); // '\t' for tab delimeted $name=implode("','",$arr); $PA=implode("','",$line); # if your data is comma separated # instead of tab separated # change the '\t' above to ',' $sql = "insert into $databasetable values ('$PA','$name')"; $sql = str_replace("''", mysql_escape_string("NULL"), $sql); $sql = str_replace("' '", mysql_escape_string("NULL"), $sql); mysql_query($sql); echo $sql ."<br>\n"; if(mysql_error()) { echo mysql_error() ."<br><b><font color=red>\n</b>"; } } ?> Hello all - Currently I have functions that will parse a csv file and insert the values into a database. That code is working. However, I want to take it a step further and ONLY insert selected fields from the csv file. For example, I have 14 rows in the csv and of those I only want to import values into the database of the rows that I define. So, gName, fName and id -> take those and pump those values into the database. I'm sure this a possible but not sure of the direction I should take. Any help on my problem would be excellent! Here's the code: Code: [Select] csv_file_to_mysql_table('tim 3.csv', 'growers'); function csv_file_to_mysql_table($source_file, $target_table, $max_line_length = 10000) { if (($handle = fopen("$source_file", "r")) !== FALSE) { $columns = fgetcsv($handle, $max_line_length, ","); foreach ($columns as &$column) { $column = str_replace(".", "", $column); } $insert_query_prefix = "INSERT INTO $target_table (" . join(",", $columns) . ")\nVALUES"; while (($data = fgetcsv($handle, $max_line_length, ",")) !== FALSE) { while (count($data) < count($columns)) array_push($data, NULL); $query = "$insert_query_prefix (" . join(",", quote_all_array($data)) . ");"; mysql_query($query) or die(mysql_error()); } fclose($handle); } } function quote_all_array($values) { foreach ($values as $key => $value) if (is_array($value)) $values[$key] = quote_all_array($value); else $values[$key] = quote_all($value); return $values; } function quote_all($value) { if (is_null($value)) return "NULL"; $value = "'" . mysql_real_escape_string($value) . "'"; return $value; } Here's some sample data: Code: [Select] Long,Lat,id,gName,fId,fName,fldId,fldName,fldAcers,featureID,objID,fInsu,fFSA,fBid -82.38306422,40.38439870,2,Norris| Tim,3,Hallinan,4,H1 - 10.0,9.900,1,1, , ,29 -82.22279060,40.42760230,2,Norris| Tim,4,Holt,5,Ho1,11.50,1,1, , ,30 -82.21917211,40.42838107,2,Norris| Tim,4,Holt,6,Ho2,15.10,1,1, , ,31 -82.21710436,40.42454375,2,Norris| Tim,4,Holt,7,Ho3,17.90,1,1, , ,32 -82.21833571,40.42367314,2,Norris| Tim,4,Holt,7,Ho3,17.90,2,2, , ,32 -82.21595345,40.42200315,2,Norris| Tim,4,Holt,9,Ho4,9.100,1,1, , ,34 -82.36538195,40.37711617,2,Norris| Tim,5,Home,10,H1 - 36.3A,36.20,1,1, , ,35 -82.36159625,40.37804250,2,Norris| Tim,5,Home,11,H2 - 3.9A,3.900,1,1, , ,36 -82.36196265,40.38085335,2,Norris| Tim,5,Home,12,H3 - 6A,7.000,1,1, , ,37 -82.41030962,40.38997625,2,Norris| Tim,2,Kenyon,13,K10-17A,17.00,1,1, , ,38 -82.38584288,40.35998635,2,Norris| Tim,2,Kenyon,14,K11-14A,14.52,1,1, , ,39 -82.41644710,40.37927258,2,Norris| Tim,2,Kenyon,15,K3 - 18.2A,18.20,1,1, , ,40 -82.40744700,40.37788250,2,Norris| Tim,2,Kenyon,16,K4 - 26.2A,26.12,1,1, , ,41 -82.40390048,40.37467350,2,Norris| Tim,2,Kenyon,17,K5 - 8.9A,8.874,1,1, , ,42 -82.38605720,40.36920760,2,Norris| Tim,2,Kenyon,18,K6 - 19.3A,18.22,1,1, , ,43 -82.39960597,40.38844915,2,Norris| Tim,2,Kenyon,19,K7 - 18.4A,18.37,1,1, , ,44 -82.39515150,40.39498212,2,Norris| Tim,2,Kenyon,20,K8 - 7.3A,7.324,1,1, , ,45 -82.38817795,40.39458225,2,Norris| Tim,2,Kenyon,21,K9 - 40.4A,40.28,1,1, , ,46 -82.38836722,40.39172487,2,Norris| Tim,2,Kenyon,21,K9 - 40.4A,40.28,2,2, , ,46 -82.40294059,40.35565598,2,Norris| Tim,10,Lane,2,1,6.900,1,1, , ,47 -82.40579843,40.35399913,2,Norris| Tim,10,Lane,22,2,7.200,1,1, , ,48 -82.38322795,40.37619695,2,Norris| Tim,6,Leach,23,L1 - 1.5,1.500,1,1, , ,49 -82.38334655,40.38060737,2,Norris| Tim,6,Leach,24,L2 - 17.6,17.60,1,1, , ,50 -82.38032235,40.38354262,2,Norris| Tim,7,Robinson,25,R1 - 7.5,7.400,1,1, , ,51 -82.39919331,40.35353322,2,Norris| Tim,9,Shorey,26,SH1,5.757,1,1, , ,52 -82.40033216,40.35715336,2,Norris| Tim,9,Shorey,28,SH2,13.39,1,1, , ,54 -82.37072642,40.31789197,2,Norris| Tim,8,Stream,27,S1,17.80,1,1, , ,53 include "connect.php"; if(isset($_POST['submit'])) { $filename=$_POST['filename']; $handle = fopen("$filename", "r"); while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) { $import="UPDATE isc_products(prodavailability,prodinvtrack,prodcurrentinv) values('$data[1]','$data[2]','$data[3]') where vendor_id = '($data[0])' "; mysql_query($import) or die(mysql_error()); } fclose($handle); print "Import done"; } else { print "<form action='inv_update.php' method='post'>"; print "Type file name to import:<br>"; print "<input type='text' name='filename' size='20'><br>"; print "<input type='submit' name='submit' value='submit'></form>"; } ?> I am building this script to import some data via a csv that has three fields. The where statement gives me this error - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(prodavailability,prodinvtrack,prodcurrentinv) values('35','1','35') where vendo' at line 1. I'm know my syntax is flawed but can't find a solution to this.... any one point me in the right direction? I know just enough php to get myself in trouble!! lol I have a problem with my project. i want to import Excel sheet directly into mysql database for PHP language without any conversion of Excel sheet into another format. i am very fed about this from last two weeks. So Please help me How I Do it. any expert please help me |