PHP - Server Timeout Scraping Multiple Pages
Hi,
I have some code which scrapes data from a page. However there are around 1200 product pages on the site I need to scrape, when I attempt to loop through all the pages I get a server timeout. I can only get to around 40 without timeout. Has anyone else had this problem? Similar TutorialsI have a page that you click to from your email to validate your account. Whenever you click that link and it goes to this page, the server connection times out. What is in this code that would make it timeout (it does not give an error, just says connecting...then times out) session_start(); include "../incl/connectdb.php"; $key1 = $_GET['id']; $key2= $_GET['id2']; $query = "select * from users where passkey = '$key1' and pass2 = '$key2' and activation= 'pending' LIMIT 1"; $result = mysql_query ($query) or die (mysql_error()); $row = mysql_fetch_array ($result); if (mysql_num_rows ($result) < 1){ $_SESSION['message'] = "Invalid link"; header ("Location: ../"); exit(); } $query = "update table-name set activation= 'active' where id = '".$row['id']."' "; mysql_query($query); $_SESSION['message'] = "Account validated."; Hi, I have created a script that connects to an FTP SERVER... downloads a movie file to the web server then uploads it to a target FTP server. If I move a small file of say 10mb it works well! But if I move a larger moviefile such as a 700mb file it doesn't... The only thing I can think is there is some sort of timeout... Any pointers would be a massive help! I have tried to increase the FTP timeout and also enabled passive mode Code: [Select] ftp_set_option($conn2, FTP_TIMEOUT_SEC, 600); Code: [Select] <?php $movefile = "movie 2.avi"; $server1 = array( "Name" =>"Downloads", "Host" =>"10.0.1.3", "User" =>"Paulio", "Pass" =>"lol", "Path" =>"/Download/Completed"); $server2 = array( "Name" =>"files", "Host" =>"10.0.1.2", "User" =>"Paulio", "Pass" =>"lol", "Path" =>"/Television"); ////////////////////////////////// Download file to be moved. ////////////////////////////////// // Connect to server $conn1 = ftp_connect($server1['Host']); // Open a session to an external ftp site $login1 = ftp_login ($conn1, $server1['User'], $server1['Pass']); ftp_pasv($conn1, true); // Check open if ((!$conn1) || (!$login1)) { echo "Ftp-connect failed!"; die; } else { echo "Connected to " . $server1['Name'] . " FTP server.<br><br>"; } [b]ftp_set_option($conn1, FTP_TIMEOUT_SEC, 600);[/b] ftp_chdir($conn1, $server1["Path"]); // Moves file to be moved to web NAS drive. ftp_get($conn1, $movefile, $movefile, FTP_BINARY); ////////////////////////////////// Upload file to be moved. ////////////////////////////////// // Connect to server $conn2 = ftp_connect($server2['Host']); // Open a session to an external ftp site $login2 = ftp_login ($conn2, $server2['User'], $server2['Pass']); ftp_pasv($conn2, true); // Check open if ((!$conn2) || (!$login2)) { echo "Ftp-connect failed!"; die; } else { echo "Connected to " . $server2['Name'] . " FTP server.<br><br>"; } [b]ftp_set_option($conn2, FTP_TIMEOUT_SEC, 600);[/b] ftp_chdir($conn2, $server2["Path"]); // Uploads moved file from web NAS drive to destination ftp_put($conn2, $movefile, $movefile, FTP_BINARY); //Deletes source file ftp_delete($conn1, $movefile); //Deletes temp file unlink($movefile); ftp_close($conn1); ftp_close($conn2); echo "Complete."; ?> Thanks, Paul. The html file i'm scraping from has 11 boxevent div classes and I want to loop through them and grab the results and team name for each. Am I doing it the right way? Also how can I loop through all float-right result and team classes grabbing the data. <div class="boxevent"> <div class="row"> <span class="float-right result">4</span> <span class="team" title="Pittsburgh Power ">PIT</span> </div> <div class="row"> <span class="float-right result">12</span> <span class="team" title="Orlando Predators ">ORL</span> </div> </div> <div class="boxevent"> <div class="row"> <span class="float-right result">24</span> <span class="team" title="Atlanta ">ATL</span> </div> <div class="row"> <span class="float-right result">6</span> <span class="team" title="Miami ">MIA</span> </div> </div> **php** <?php include 'includes/simple_html_dom.php'; $html = new simple_html_dom(); $html = file_get_html('http://score...', false, $context); $score0 = $html->find('span[class=float-right result]', 0); $score1 = $html->find('span[class=float-right result]', 0); $team0 = $html->find('span[class=team]', 0); $team1 = $html->find('span[class=team]', 0); /* out */ echo '<pre>'; print_r($score0); print_r($score1); print_r($team0); print_r($team1); echo '</pre>'; $html->clear(); unset($html); ?> I have 2 pages in my flash site that call some PHP files. The pages are a feedback page and a locator page. When my server went from PHP4 to PHP5 both pages stopped displaying the content. When I load the flash pages with Fiddler open, fiddler shows a 500 error on the PHP files that the flash is calling. I'm certain my database connections are good, file paths have not changed since the server maintenance, and I have .htaccess files except they are blank. Can anyone advise what I can do to find where the problem is? I have the PHP code below that showed the 500 errors. Any help would be really appreciated. This is the locator page: Code: [Select] <?php include_once('db_connect.inc.php'); $query = "SELECT state FROM stores group BY state ORDER BY state"; $result = mysql_query($query); if(mysql_num_rows($result)==NULL){ $r_string = '&error=1&msg=No Records Found.'; }else{ $r_string = '&error=0&n='.mysql_num_rows($result) . '&r_states='; $i = 0; $r=''; while ($row = mysql_fetch_assoc($result)) { if($r!='') $r .= '||'; $r .= $row['state']; $i++; } $r_string .= $r; // add extra & to prevent returning extra chars at the end $r_string .='&'; } echo $r_string; ?> This is the feedback page: Code: [Select] <?php include_once('inc/feedbackconn.inc.php'); echo '&rsult='; $query = "SELECT * FROM messages where active!=0"; $result = mysql_query($query); $num = @mysql_num_rows($result); $cfeed=$_POST['sFeed']; $cfeed--; $query = "SELECT * FROM messages where active='1' ORDER BY id LIMIT " . $cfeed . ',1'; $result = mysql_query($query); //$row = mysql_fetch_array($result); if($num==NULL){ echo "No Records."; exit(); } while($row = @mysql_fetch_array($result)){ $b_name = stripslashes($row[strName]); $b_loc = stripslashes($row[strLocation]); $b_mes = stripslashes($row[strMessage]); $b_id = $row[id]; $b_active = $row[active]; $feedArray[] = array("name"=>$b_name,"location"=>$b_loc,"feedback"=>$b_mes,"id"=>$b_id,"active"=>$b_active); } /* /////////////// DISPLAY THE RECORDS ///////////////// */ $numOfMessages = sizeOf($feedArray); for($i=0;$i<$numOfMessages;$i++){ //------------------------------------------------------------------ //echo $feedArray[$i]['id']."<br>\n"; echo '<b><i>' . $feedArray[$i]['feedback']."</i><br><br>"; echo $feedArray[$i]['name']."<br>"; echo $feedArray[$i]['location']."</b>"; //--------------------------------------------------------------------- } echo '&tFeeds=' . $num; echo '&cFeed=' . $_POST['sFeed']; ?> Great conversation, wrong board. This topic has been moved to Application Design. http://www.phpfreaks.com/forums/index.php?topic=357967.0 I don't think php is installed/enabled?! Hey guys, how is everyone today? I am needing a little help with this one. I have a database that updates itself and the display page for it with every form submission on the website. I am wondering how I go about getting it to display across multiple pages? For example, once it hits 25 submissions, it would make a page 2, once page two hits 25 submissions, it creates page 3, etc. etc. Much like you see on something like a forum, it hits so many replies before it creates a new page, to limit one long page of results. Thanks in advance!! Hello. I'm coding myself an small webpage, In the internet you can see that there is pages like index.php?id=223923 <- for example or index.php?=news. So, I'm trying to create similar to that myself. I tried googling and searching youtube how to do this but didn't really find anything. I figured it out that it needs some database etc. I tried myself doing some table in my mysql db. And in the table some 'id, title, content' and in the id would be the url, (index.php?='id') the title would be the <title> </title> and the content would be all the code inside the webpage. I got no idea how to link these to an php or whatever it should be done So would anyone kindly tell me howto do this or give some link to an tutorial? HY I have index.php and pictures.php. In index.php I have 3 columns: - left (for menu) - right (for advertising) - center (where I want to include pictures.php) What is the best way to include pictures.php in center of index.php Code: [Select] if($_get[pictures]){ include ("pictures.php"); } I ask this because I have multiple variable like "pictures" and I will have multiple "If". Or to include in DB all this variables "pictures" and just add ".php" extension. It is secure in this way ? Thanks I've read through a few examples of how to paginate query results, but with my currently php skills I don't understand them, or they do not fit into my code. I wondered if I could make my own, and the logic behind this code makes sense to me, but it's still not working. Here is the code: <?php if (!$Start) { $Start = 0; } else { $Start=$_POST['Start']; } $crittercount = 120; $numpages = $crittercount / 30; $pagei = 1; $pagereit = 1; echo '<p><center> <FORM method=POST action=testpagination.php> <SELECT NAME="Start"> <OPTION>Page 1'; while ($pagereit < $numpages) { $pagei = $pagei + 30; echo '<OPTION value='.$pagei.'>'; $pagereit = $pagereit + 1; echo "Page: " .$pagereit; } echo '</SELECT>'; echo'<input type=submit value=Go></FORM><p>'; echo $Start; ?> The test site is he http://lab-lib.com/felishorns/Felishorns/websitetest/testpagination.php $crittercount is a variable that changes depending on the main $query. This query is invisible, and is used to pull a count for all the results in total. I am modifying it with $Start to create a second query which the viewer sees. $queryF = $query . " LIMIT " . $Start . ", 30"; The idea is that $Start changes depending on the value selected, thereby changing the query. In this test code, I've simply echoed the $Start value of that page, to see if the code is working, and it's not changing. What's wrong with this code? Is it not possible to have a form lead to the same link of the page it's on? Does the variable not become updated? Thanks in advance. I am creating a webpage for my golf league and some of the information(variables) such as Name, Address and email, must be reused across multiple pages for different outputs. May question is "Can I, or is there a way to reUse variables across multiple pages. "Do I need to create a session and make everything session variables"? "Should I link to a database and call the information when needed"? Any explanation would be greatly appreciated. Hi guys, I have a php file that will go to a site and scrape the data I need. However this site is setup to use pagination so when I try to scrape all the players names I have to do separate queries to search each page. Is there a way to find out by using code how many pages there are and query all the pages at same time? I use this code Code: [Select] <?php //first page //turn error reporting on libxml_use_internal_errors(true); //get data from this page $dom = new DOMDocument; $dom->loadHTMLFile('http://www.gametracker.com/server_info/76.73.3.42:1716/top_players/?searchipp=50#search'); $xpath = new DOMXPath($dom); // Get the total player count $rows2 = $xpath->query('//div[@class="block774"]/div'); // Get the rows from the search list $rows = $xpath->query('//table[@class="table_lst table_lst_spn"]/tr'); for ($i=1; $i<$rows->length-1; $i++) { $row = $rows->item($i); // Get the columns for a row $cols = $row->getElementsByTagName('td'); // Get the player rank (1st column) echo 'Rank:'.trim($cols->item(0)->textContent).PHP_EOL; // Get the player name (2nd column) echo 'Name:'.trim($cols->item(1)->textContent).PHP_EOL; // Get the player score (3rd column, actually 4th but number 3 is hidden) echo 'Sco '.trim($cols->item(3)->textContent).PHP_EOL; echo "<br />"; } ?> <?php //secondpage //turn error reporting on libxml_use_internal_errors(true); //get data from this page $dom = new DOMDocument; $dom->loadHTMLFile('http://www.gametracker.com/server_info/76.73.3.42:1716/top_players/?searchipp=50&searchpge=2#search'); $xpath = new DOMXPath($dom); // Get the rows from the search list $rows = $xpath->query('//table[@class="table_lst table_lst_spn"]/tr'); for ($i=1; $i<$rows->length-1; $i++) { $row = $rows->item($i); // Get the columns for a row $cols = $row->getElementsByTagName('td'); // Get the player rank (1st column) echo 'Rank:'.trim($cols->item(0)->textContent).PHP_EOL; // Get the player name (2nd column) echo 'Name:'.trim($cols->item(1)->textContent).PHP_EOL; // Get the player score (3rd column) echo 'Sco '.trim($cols->item(3)->textContent).PHP_EOL; echo "<br />"; } ?> I also have to go to that website first to see how many pages there are so I can have enough queries. Hello Everyone was wondering if I could get some help with the following code? I am querying a database for results of listings that are in a database these listings are displayed on the page in a form. I am wanting each listing to be on a different page. Below is my code. Code: [Select] $lim=1; if (!isset($s) || $s < 1 || !is_numeric($s)) { $s = 1; } $start = ($s - 1) * $lim; $sql = "select id,bussimg,imagewidth,imageheight,email,usridm,company,businesscategory,address1,address2,state,city,zip,website,email,repname,description,phonenumber,country,status from $approvecheckbusinesses where usridm='$user_id'"; $result=db_query($sql); $countpages = $sql; $sql = $sql . " order by id asc limit $start, $lim"; $result=db_query($sql); $pages = ceil(mysql_num_rows(mysql_query($countpages)) / $lim); $result=db_query($sql); for ($i = 0; $i < mysql_num_rows($result); $i++) { $Listid= mysql_result($result, $i, "id"); $usridm= mysql_result($result, $i, "usridm"); $CompanyName= mysql_result($result, $i, "company"); $realname= mysql_result($result, $i, "repname"); $email= mysql_result($result, $i, "email"); $BusinessCategory= mysql_result($result, $i, "businesscategory"); $status= mysql_result($result, $i, "status"); echo ("FORM IS TO BE DISPLAYED HERE"); } if ($pages > 1) { echo("<p align=left style='font-size: 85% color=white'>"); for ($i = 1; $i <= $pages; $i++) { echo("["); if ($i == $s) {echo("<b>");} else {echo("<a id=home_offerLink href='index.html?EditMemberListing&user_id=$user_id&s=$i'>");} echo("Page $i"); if ($i == $s) {echo("</b>");} else {echo("</a>");} echo("] "); } echo("</p>") Page Numbers here using the above code.. The problem I seem to be running into is that it only displats the first record. The page numbers show up page 1 page 3 page 2 and three are blank there is no mysql error or anything for some reason I only get that first result out of three Hello everyone, Very new to coding - enjoying it but struggling! I think I'm trying to do something pretty common but I seem to have come up against a complete wall now and after hours/days searching the internet and reading books I'm completely stuck! I'm trying to write some code to search a MySQL database of products, then display the results. For some search results there will be lots of products so I want to display 10 products on the first page then allow visitors to go to the next page to see another 10, and so on - a type of pagination, as they should then be able to click back to see the last page etc. I've got to the point of being able to display the first 10 search results, but I can't figure out at all how to create some kind of page scrolling/pagination system. Please, does anybody have any ideas?? I've attached my code, I hope this is the correct way of doing things here. Many thanks for your time! The PHP search code... Code: [Select] <?php //opens connection to mysql server $dbc = mysql_connect('localhost'); if (!$dbc) { die('Not connected :' . mysql_error()); } echo "Connected to mysql database<br />"; //select database $db_selected = mysql_select_db("NAME_OF_DATABASE", $dbc); if (!$db_selected) { die ("Cannot connect :" . mysql_error()); } echo "Connected to database<br /><hr />"; echo "Here are your results"; $term = $_POST['term']; $category = $_POST['category']; $brand = $_POST['brand_name']; $sql = mysql_query("SELECT * FROM products where product_name like '%$term%' AND category_name like '%$category%' AND brand_name like '%$brand%' LIMIT 0, 10"); { while ($row = mysql_fetch_array($sql)){ echo "<table border='1' width='100%'> "; echo "<tr>"; echo "<td style='vertical-align:top' width='25%'>" . '<img src="', $row['image_url'], '" alt="', $row['product_name'], '" width="100" height="100" />' . "</td>"; echo "<td style='vertical-align:top' width='50%'>" . $row['product_name']; echo "<br />"; echo "<span style='font-size: 10px'>" . $row['description'] . "</span>" . "</td>"; echo "<td style='vertical-align:top' width='25%'>" . $row['price']; echo "<br />"; echo "<br />"; echo "<hr />"; echo "$row['merchant_name'] </td>"; echo "</tr>"; } echo "</table>"; } ?> Hey Im always trying to remove code and cut corners to reduce work in the long run, soIim wondering how I could link my menu bar from say a template to ALL my php pages for my site so I don't have to write/change links on every page when I need to. Thanks I have an ordering process that consists of 3 pages: 1) Page1: Enter name/address 2) Page2: Based on address, determine availability of products, and let user choose which products, how many, etc. 3) Page3: Enter credit card info to send to the processor What is the best way to store information fro page 1 to page 2, then from page 2 to page 3? I thought about just using hidden fields. Is this OK? Is there another preferred way? Thanks! I'm am somewhat new to PHP and am trying to set up a website for my cousin's wedding. Her idea is to have the guests sign in with a user/pass that she provides, and once they sign in, they will be taken to a page that has their name on it (i.e. "Mr. and Mrs. So and So, you are invited...). I have come to the conclusion that I will need to make an image for each guest's name (she wants to use a font for their names that nobody will have on their computer) so what I need to know is: How do I link each user name to their own personalized webpage, where the image of their name on the next page will change based on what username is entered? I have been told to use Sessions (which I don't yet have in this code), but I'm clueless as to how to make that work for multiple users. Where do I put the coding, what does the coding look like, etc. Thanks in advance for any help! The php code I have right now is this (i'm sorry it's so long, I just don't want to leave anything out that might be important): Code: [Select] $LOGIN_INFORMATION = array( 'steve' => 'password', 'rick' => 'password', 'tom'=> 'password' ); // request login? true - show login and password boxes, false - password box only define('USE_USERNAME', true); // User will be redirected to this page after logout define('LOGOUT_URL', 'http://www.example.com/'); // time out after NN minutes of inactivity. Set to 0 to not timeout define('TIMEOUT_MINUTES', 0); // This parameter is only useful when TIMEOUT_MINUTES is not zero // true - timeout time from last activity, false - timeout time from login define('TIMEOUT_CHECK_ACTIVITY', true); ################################################################## # SETTINGS END ################################################################## /////////////////////////////////////////////////////// // do not change code below /////////////////////////////////////////////////////// // show usage example if(isset($_GET['help'])) { die('Include following code into every page you would like to protect, at the very beginning (first line):<br><?php include("' . str_replace('\\','\\\\',__FILE__) . '"); ?>'); } // timeout in seconds $timeout = (TIMEOUT_MINUTES == 0 ? 0 : time() + TIMEOUT_MINUTES * 60); // logout? if(isset($_GET['logout'])) { setcookie("verify", '', $timeout, '/'); // clear password; header('Location: ' . LOGOUT_URL); exit(); } if(!function_exists('showLoginPasswordProtect')) { // show login form function showLoginPasswordProtect($error_msg) { ?> <html> <head> <title>Please enter password to access this page</title> <META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE"> <META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <style type="text/css"> body,td,th { font-family: Verdana, Geneva, sans-serif; font-size: 10px; color: #666; } body { background-color: #FFFFFB; } </style> </head> <body> <div align="center"> <style> input { border: 1px solid black; } </style> <div style="width:600px; margin-left:auto; margin-right:auto; text-align:center"> <form method="post"> <h4>Please sign in using the information provided on the invitation</h4> <font color="red"><?php echo $error_msg; ?></font><br /> <?php if (USE_USERNAME) echo 'Login:<br /><input type="input" name="access_login" /><br />Password:<br />'; ?> <input type="password" name="access_password" /><p></p><input type="submit" name="Submit" value="Submit" /> </form> <br /> <a style="font-size:9px; color: #B0B0B0; font-family: Verdana, Arial;" href="http://www.zubrag.com/scripts/password-protect.php" title="Download Password Protector">Powered by Password Protect</a> </div> </body> </html> <?php // stop at this point die(); } } // user provided password if (isset($_POST['access_password'])) { $login = isset($_POST['access_login']) ? $_POST['access_login'] : ''; $pass = $_POST['access_password']; if (!USE_USERNAME && !in_array($pass, $LOGIN_INFORMATION) || (USE_USERNAME && ( !array_key_exists($login, $LOGIN_INFORMATION) || $LOGIN_INFORMATION[$login] != $pass ) ) ) { showLoginPasswordProtect("Incorrect password."); } else { // set cookie if password was validated setcookie("verify", md5($login.'%'.$pass), $timeout, '/'); // Some programs (like Form1 Bilder) check $_POST array to see if parameters passed // So need to clear password protector variables unset($_POST['access_login']); unset($_POST['access_password']); unset($_POST['Submit']); } } else { // check if password cookie is set if (!isset($_COOKIE['verify'])) { showLoginPasswordProtect(""); } // check if cookie is good $found = false; foreach($LOGIN_INFORMATION as $key=>$val) { $lp = (USE_USERNAME ? $key : '') .'%'.$val; if ($_COOKIE['verify'] == md5($lp)) { $found = true; // prolong timeout if (TIMEOUT_CHECK_ACTIVITY) { setcookie("verify", md5($lp), $timeout, '/'); } break; } } if (!$found) { showLoginPasswordProtect(""); } } ?> I am using mod_php with Apache/2.4.6 (CentOS). I currently do not have multiple versions of PHP running on my server, but wish to do so now. Will I need to use php-fpm? Even if multiple instances can be accomplished with mod_php, is it typically preferred to use php-fpm? Any unusual gotcha's that I should be aware of? Will I need to uninstall mod_php before installing php-fpm? I previously used remi-php73 repo. Better to install from source? Thanks Edited August 25, 2019 by NotionCommotion |