PHP - Reading Several Remote Urls Simultaneously
I'm trying to create a simple search engine bot and have found the biggest issue is that of performance, it takes several hours to index all the links and ideally I need to run it once a day, the bottleneck occurs when pulling the data from the remote urls in terms of network transfer time. This could obviously be reduced massively if I could read from several url's simultaneously. Does anyone know how I could do this. I'm aware that in PHP5 there is thread functionality which may solve the problem but this is only available from the CLI. I heard that it could be done with CURL without having to go down the threaded route, does anyone know how nthis can be done or know of any examples on the net that I could look at?
As a cheeky follow up, I was also thinking a potential problem could be the script getting out of hand and opening too many connections and I don't to cause a DOS attack! Is there a way that the number of connections could be limited to 10 for example. Is there a way of detecting when a CURL connection has finished reading a file so that a connection manager class could then start the next connection asap? Thanks for any helpn people, I've been looking for this stuff for days!! Similar TutorialsHello all, I am trying to read a remote xml file on a file server which requires username and password. I can access the xml file fine through HTTP authentication like \\111.11.11.111\\pdrs\fileserver\FullText\12345\12345_CLEANUP.xml But when i try to read it with PHP, I recieved a warning message: "Warning: simplexml_load_file() [function.simplexml-load-file]: I/O warning : failed to load external entity" allow_url_fopen is ON This is part of my code. Code: [Select] $test=rawurldecode("http://username:password@\\\\111.11.11.111\\pdrs\\fileserver\\" . $number . "\\FullText\\" . $number . "_CLEANUP.xml"); echo $test; $xml_filename = simplexml_load_file(rawurldecode("($test)")); echo $xml_filename; print_r($xml_filename); Am I missing something here? Is there any other way to read/save/load a remote xml file which requires authentications? I am new to php so any help is appreciated! Thanks, Hey guys I m developing a download system for my server that reads files from a third party server and enables MY USER to download it via my URL. What mechanisms should i use to manage this efficiently ??? file_get_contents(); fopen(); Or should I use some kind of buffer mechanism ??? Please do provide a code sample as I'm stuck here very badly. Note: The file sizes may be >500MB Hi all, I am looking a developing a script that instead of using something like mysite.com/showproduct.php?id=1 I would like it to appear as mysite.com/php-editor Is their a simple way of doing this? I would still need to pull info from a database so the page would still need to be dynamic, I just want it to appear static! I am used to using the get id function of php, what would the workaround be? would the id be hidden from the url but still usable in a query? I have had a look at the apache mod_rewrite, but quite frankly, I dont understand it! Cheers Hello, Greetings! I have php application for restaurant, basically there are two types of orders Kitchen Items and Bar items. A cashier enters a order from customers from computer A (say). We have two computers in kitchen, Computer B and in Bar, Computer C. When cashier from computer A clicks a button, depending upon the order type i need to print a chit on Computer B and/or Computer C. COmputer A, B, C are in same network. All computers have laser printer which can print any type of content. Well i tried to search the forum, i couldn't find anything that would be of my help. Can anyone tell me what would be the starting point for this?? Thanks watsmyname is it possible to have two open connections to two different mysql dbs at the same time? when i tried it, only the one on the bottom of the list was active. my config file looks like this: //---------------------------------------------// $dbname = 'xxx'; # Database Name $dbuser = 'xxx'; # Database Username $dbpass = 'xxx'; # Database Password $dbhost = 'xxx'; # Database Host $conn2 = mysql_connect($dbhost,$dbuser,$dbpass) or die ("Could not connect to $dbname: ".mysql_error()); mysql_select_db($dbname) or die ("Could not access the database: ".mysql_error()); $dbname5 = 'yyy'; # Database Name $dbuser5 = 'yyy'; # Database Username $dbpass5 = 'yyy'; # Database Password $dbhost5 = 'yyy'; # Database Host $conn = mysql_connect($dbhost5,$dbuser5,$dbpass5) or die ("Could not connect to $dbname5: ".mysql_error()); mysql_select_db($dbname5) or die ("Could not access the database: ".mysql_error()); //--------------------------------------// so i want to be able to do mysql_query($query,$conn2) when i need to access xxx db, but it doesn't seem to work that way. am i doing something incorrectly? any help would be greatly appreciated. This topic has been moved to MySQL Help. http://www.phpfreaks.com/forums/index.php?topic=322398.0 Ok so I have a form to submits to a .php file that creates an image. This works fine, but I also want the info from the form to be sent to MySQL database. The problem is that something in a script I "required" called core.inc.php is interfering and so no image is output. Here is core.inc.php: Code: [Select] <?php ob_start(); session_start(); $current_file = $_SERVER['SCRIPT_NAME']; if(isset($_SERVER['HTTP_REFERER'])&&!empty($_SERVER['HTTP_REFERER'])) { $http_referer = $_SERVER['HTTP_REFERER']; } function loggedin() { if(isset($_SESSION['user_id'])&&!empty($_SESSION['user_id'])) { return true; } else { return false; } } function getuserfield($field) { $query="SELECT `$field` FROM `Users` WHERE `id`='".$_SESSION['user_id']."'"; if($query_run=mysql_query($query)) { if($query_result = mysql_result($query_run, 0, $field)) { return $query_result; } } } ?> Here is the code for the image creating .php file: Code: [Select] <?php require 'connect.inc.php'; require 'core.inc.php'; $title = $_REQUEST['title'] ; $user_id = getuserfield('id'); $query = "INSERT INTO `---` VALUES ('".mysql_real_escape_string($title)."','".mysql_real_escape_string($user_id)."')"; $query_run = mysql_query($query); //There is some more code in here obviously, but it's irrelevant header('Content-type: image/png'); imagepng($image_f); imagedestroy($image_f) Can anybody give me some idea of what is conflicting or what I can do to fix it? I require a page to be added to my website. This page will facilitate a refined search via Google, Bing and Yahoo search engine simultaneously , show the top 5 of each engine. Is this possible using php ? Thank's Is their something I can do in PHP (like some sort of function/config/array etc...) which would make the following process easier (examples appreciated)... I have an .htaccess file where im rewriting urls to seo friendly ones...now in the PHP script is their something I can place for urls are meant to be displayed/linked (like some sort of placeholder) which makes it easier for me to modify the urls within the PHP script, without constantly editing countless php sourcecode to just change the url? Example scenario: Say I had a php file (called submit.php): <?php echo 'Hey! Thanks for submiting the form, <a href="submit.php?form">click here</a> to go back!'; ?> and my rewrite rule for submit.php?form was: /form How would I easily change the url within submit.php without editing submit.php directly? Hello, Ive got many URLs like these which im trying to just get the name for example "Ace Lightning" Quote http://www.wupload.com/file/1374721/Ace Lightning # GBA.rar http://www.wupload.com/file/1374722/Action Man - Robotatak # GBA.rar http://www.wupload.com/file/1374726/Activision Anthology # GBA.rar http://www.wupload.com/file/1374761/Advance Guardian Heroes # GBA.rar http://www.wupload.com/file/1374764/Agassi Tennis Generation # GBA.rar This is my code Code: [Select] <?php if($_POST['newlinks']) { $newlinks = $_POST["newlinks"]; $newlinks = preg_split("/.rar/", $newlinks); foreach ($newlinks as $newlink) { $r_name = substr("$newlink", 36, 100); echo "<strong>Game Name: $r_name</strong> <br />"; } } echo "<form method=\"POST\" action=\"index.php?page=mass-gba\">"; echo "<br /> <strong>Mass Add GBA ROMs</strong> <br />"; echo "<textarea rows=\"16\" name=\"newlinks\" cols=\"84\"></textarea>"; echo "<input type=\"submit\" value=\"Mass Add GBA Links\" name=\"B1\">"; echo "</form>"; ?> This results are like this, I cant understand why it works for the first game but not the second or any after that. Quote True Name: Ace Lightning # GBA True Name: 2/Action Man - Robotatak # GBA True Name: 6/Activision Anthology # GBA True Name: 1/Advance Guardian Heroes # GBA True Name: 4/Agassi Tennis Generation # GBA Okay, i know there are lots of these questions out there and tons of ways to do this. I do have it working to replace URLs with <a href="URLs">URLs</a> but now i need some help detecting <a href=""></a> so the preg_replace will not double up the code and mess up the link. here is my code so far. Code: [Select] <?php echo 'COMMENTS<br /><br />'; while($rows = mysql_fetch_array($comres)){ list($comid, $menu_title, $post_id, $comdate, $comname, $comment) = $rows; $comment = nl2br($comment); $comment = preg_replace('/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(\/\S*)?/', '<a href="$0" target="_new">$0</a>', $comment); $comment = str_replace("´", "'", $comment); $comdate = date("g:ia - M j, Y",strtotime("$comdate")); echo 'User: '.$comname.'<font color="#B20303"> - '.$comdate.'</font><br />'; echo ' - '.$comment.'<br /><br />'; } ?> so with this code, if someone commented a link like this http://example.com it will turn it this <a href="http://example.com" target="_new">http://example.com</a> but now if someone knew some basic HTML coding, and they used this <a href="http://example.com">Example.com</a> it will look like this... <a href=<a href="http://example.com" target="_new">http://example.com</a>>Example.com</a> Say my URL is http://www.example.com/catalog.php?category=hats&prodID=53 and I use Mod-rewrite to make it http://www.example.com/catalog/hats/53/ Can I still use $_GET['category'] to get the parameters? If so, how? Hello! First... I'm sorry about my english skills.. Seems that with my own language I don't get a correct or good answer So.. I have a site where visitors can put there favorite links to other sites and show them to others. My problem is multiple urls to same locations in sites-table. example: http://google.com http://google.com/ http://www.google.com http://www.google.com/ http://www.google.com/index.php (?) they all lead to same... is there any light and powerfull way to check these? I'm trying to use query with "like" but it only works if there is exactly same url. I use fopen to check that url really exicst.. table: Sites (basic information for link) -id -url -title +some meta tags if found -timestamp I'm not asking to do me this code. But if someone can tell me just what functions and others I'm looking for and what I need for this... My coal is to keep sites-table as clean as it can be. If visitor adds a url that is already in sites-table only thing that happens is updating timestamp. Thank you all for helping me! -Roosterr Hi, What is the best way to change: www.mysite.com/product.php?id=1&sort=2&cat=z to: www.mysite.com/1/product-name-here I would need the other variables in the URL (sort & cat...) to also pass through to the new address. Is this an .htaccess change or via PHP code? Any help appreciated, thanks. Hi All, Currently, in my application, my links use index.php?id=14 (or whatever ID it is). Should I somehow encrypt those so the exact ID is not known or is this OK? Can someone help me to generate SEO friendly URLs? I dont want my URLs to be presented like: www.example.com/listprod.php?categ=1&subcateg=7 I want it to dynamically do the following: www.example.com/listprod/cameras/slr I would love to google my question, but I don't know how this is called.... I want to create URLs for each user of a site with this format: www.mysite.com/user1 Facebook does this, for example. In a static page I'd do it with a folder and index file for each user, but that's not practical if you have too many users. Could you explain how this is done or at least how it is called so I can find myself something to read? Thanks!!! I am using seo friendly urls so when someone makes a post named "this is a post" the url will point to www.example.com/topic/this_is_a_post But when the user enters a character in their post name that means something in a url(? /) it obviously breaks. How can i make the urls safe from this without str_replace as i want to keep the characters. Hi Everyone,
I am really struggling with this.....Cannot find a solution anywhere.
I have a website with SSL installed.
I would like to force SSL on the whole website but not on 2 urls.
If my whole site is https://mysite.com
and the 2 urls to disable from the SSL a
http://mysite.com/hotels
http://mysite.com/weather
How can I do it via htaccess please?
Thank you so much,
Ben
im sure i am missing something simple. but this is killing me. i get this to work: Code: [Select] $url = (isset($_POST['url'])) ? $_POST['url'] : ''; $BlockedNames = array('nepwk', 'teleworm'); $BlockedDomains = array('nepwk.com', 'teleworm.com', 'yopmail.com', 'adf.ly', 'www.nepwk.com'); @list($name, $domain) = explode("http://", $url); else if(in_array($name, $BlockedNames) || in_array($domain, $BlockedDomains)) { $mesaj = "<div class=\"msg\"<div class=\"error\">Your URL has been blocked by our system!</div></div>"; } now the above works if i type the whole domain for every single url i want to block but i want to know is how would i add a catch all of sorts to the domain so say the domain is: http://www.google.com i want to block everything google so all of the following would be blocked as well http://www.google.com/whatever http://www.google.com/anything http://www.google.com/something i want all of those to be blocked automatically without having to add each individual one im trying to block adf.ly links which change for each person and each url they mask and i want to take proactive measure to just block them from being added into my form instead of having them added to my database then having to go and manually delete them. any questions or if this doesn't make sense let me know. thanks in advance (and maybe this is bad code im a noob so go easy ) |