PHP - Curl Connections
Hi,
I would just like to know what limitations apply for opening X amount of connections to different websites with Curl? (specifically, I'm using Curl multi and have my own way of regulating how many connections occur at a given time). I ask this because it seems I can open 100 connections and get the info from the websites quite quickly. If I were to open say 1000 would that crash my router or what would happen? Thanks in advance! Similar Tutorialsgood day dear community, i am workin on a Curl loop to fetch multiple pages: i have some examples - and a question: Example: If we want to get information from 3 sites with CURL we can do it like so: $list[1] = "http://www.example1.com"; $list[2] = "ftp://example.com"; $list[3] = "http://www.example2.com"; After creating the list of links we should initialize the cURL multi handle and adding the cURL handles. $curlHandle = curl_multi_init(); for ($i = 1;$i <= 3; $i++) $curl[$i] = addHandle($curlHandle,$list[$i]); Now we should execute the cURL multi handle retrive the content from the sub handles that we added to the cURL multi handle. ExecHandle($curlHandle); for ($i = 1;$i <= 3; $i++) { $text[$i] = curl_multi_getcontent ($curl[$i]); echo $text[$i]; } In the end we should release the handles from the cURL multi handle by calling curl_multi_remove_handle and close the cURL multi handle! If we want to another Fetch of sites with cURL-Multi - since this is the most pretty way to do it! Well I am not sure bout the string concatenation. How to do it - Note I want to fetch several hundred pages: see the some details for this target-server sites - /(I have to create a loop over several hundred sites). * siteone.example/?show_subsite=9009 * siteone.example/?show_subsite=9742 * siteone.example/?show_subsite=9871 .... and so on and so forth Question: How to appy this loop into the array of the curl-multi? <?php /************************************\ * Multi interface in PHP with curl * * Requires PHP 5.0, Apache 2.0 and * * Curl * ************************************* * Writen By Cyborg 19671897 * * Bugfixed by Jeremy Ellman * \***********************************/ $urls = array( "siteone", "sitetwo", "sitethree" ); $mh = curl_multi_init(); foreach ($urls as $i => $url) { $conn[$i]=curl_init($url); curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout curl_multi_add_handle ($mh,$conn[$i]); } do { $n=curl_multi_exec($mh,$active); } while ($active); foreach ($urls as $i => $url) { $res[$i]=curl_multi_getcontent($conn[$i]); curl_multi_remove_handle($mh,$conn[$i]); curl_close($conn[$i]); } curl_multi_close($mh); print_r($res); ?> I look forward to your ideas. Wasn't really sure what to call this topic, and I'm also not sure how to ask this question. Here is what I have, and the problem I'm having. I have an html form on a webpage that sends information into 2 different tables in a database I have. One table is not important to the topic, the other table is where i'm having my problem. The user submits their information from the form and the info goes into the tables. There is another page that displays the names of the users on a "wall" that populates from the second table in the database. The second table takes information from the form they filled out and displays their name on the wall based on that (most importantly whether or not they can be e-mailed from the wall or not). The issue I am having (and this is probably the long way of asking this) is that when some people fill out the form their info does not go into either database. I know the functionality works, because their have been a couple of users that have signed up and everything worked fine for them. I'm thinking this may be a connection issue possibly due to a firewall or something. Any ideas on this? The website is http://itgetsbetter.tsagl.org the form to fill out is under the button "Take the Pledge" and the names are displayed under "View the Wall". If someone wouldn't mind to try out the functionality of the form for me (I need people that have some technical experience to do this because most of the people that I work with have no idea what I am talking about when trying to troubleshoot this with them). I assure you that if everything works, I will remove your information from my database. This site is only out to a few people at the moment, but I am planning on taking it public in the next couple of days. I would display code here, but I'm not sure what code you would need to see in order to help with this. I just think this is a connection issue; I just need to know how to test it or how to rectify the issue. Thanks for your help (and please don't hate too much on the design on the site; i'm more of a programmer than a designer). On my new site on average every page that loads opens 9 MySql connections and runs 14 MySql Queries, And as I use Ajax to dynamically update the pages, so this repeats every 60 seconds per page. Is this too many connections? I am not having any problems at the moment but the site is getting more busy, and I don't want the site to start running slow, or the MySql server to crash. Any comment would be much appreciated. OK, so I dont think this is a code issue as such, but I am completely stumped as to where the issue could be. I have a page with some standard include() functions at the top. When I access the page from home, everything operates as it should, when I access it from work (on multiple machines), I get the standard "unable to open stream" and "unable to access" warnings for the include statements. I have tried hard refreshing and deleting cache (not that it should make a shred of difference) etc, but the error is still there after a week or so. Does anyone know what could be causing this issue? I've looked every where, probably googled every variation of my question I could come up with and I've tried all of the suggestions... I'm trying to connect to two different DBs in two different locations ($local and $remote) and only the second one works. Here is a sample of my code ("..." = hidden): //-------------Local DB Connection: $local = mysql_connect("localhost","root","..."); if (!$local) { die('Could not connect: ' . mysql_error()); } $sel1 = mysql_select_db("new", $local); //-------------Remote DB Connection: $remote = mysql_connect("...","...","...",true); if (!$remote) { die('Could not connect: ' . mysql_error()); } $table = "..."; //---------function selecting from local: function fncGrabNemsis($ele,$val){ mysql_select_db("new", $local); $result = mysql_query("SELECT * FROM new.tblvalues WHERE fldelement='$ele' AND fldcode='$val'",$local); $tmprow = mysql_fetch_array($result); return (isset($tmprow['fldvariable'])?$tmprow['fldvariable']:$val); } //----------Select run from Remote: mysql_select_db("ImdxTest", $remote); $result = mysql_query("SELECT * FROM ImdxTest.$table WHERE ClientID = ... AND IncidentNum = '$fldINCID'", $remote) or die(mysql_error()); $row = mysql_fetch_array($result); I've tried moving the mysql_select_db() function calls everywhere you can think of and just about everything else... What happens is, I get php errors saying that $local is not defined or that the mysql function that are trying to use the $local connection are expecting parameters to be resources!? I know for a fact that both connections work because individually they both work. Only the second connection ($remote) works... Thanks a lot for any suggestions! Hey, I would there be a way i could create a form in which upon entering a domain name/URL it will tell you how many connections there are to the site and how many (if any) MySQL queries are running? If I add this line to a wordpress footer it displays the number of sql queries. Code: [Select] <?php echo get_num_queries(); ?> queries in <?php timer_stop(1); ?> seconds. Any help is greatly appreciated. Thanks. I was curious.. I read that after using a PDO connection you should set the object to null to close it out. In this instance: Code: [Select] function IsOwner($regionid, $user) { global $Username, $SQL_Server, $SQL_User, $SQL_Pass; $pdo = new PDO("mysql:host=$SQL_Server;dbname=$SQL_Database", $SQL_User, $SQL_Pass); $sql = $pdo->prepare("SELECT count(region_id) FROM region_players WHERE region_id = ? AND user_id = ? AND owner = 1"); $sql->execute(array($regionid, $user)); if ($sql->rowCount() == 0) { return false; } else { return true; } $pdo = null; } Since the return is called prior to the nulling, does it leave a connection hanging or does PHP do some sort of auto-closeout when a function ends? Should I instead be doing: Code: [Select] function IsOwner($regionid, $user) { global $Username, $SQL_Server, $SQL_User, $SQL_Pass; $ownerstatus = false; $pdo = new PDO("mysql:host=$SQL_Server;dbname=$SQL_Database", $SQL_User, $SQL_Pass); $sql = $pdo->prepare("SELECT count(region_id) FROM region_players WHERE region_id = ? AND user_id = ? AND owner = 1"); $sql->execute(array($regionid, $user)); if ($sql->rowCount() > 0) $ownerstatus = true; $pdo = null; return $ownerstatus; } ? Hello guys, I hope you can help. I can't get my page to switch between two database connections properly It seems to ignore the second connection. I'm trying to go through each group in database one and find out how many members they have registered in database two by checking the field "tsName". This is a valid query for database one. $fetchUsers = mysql_query("SELECT * FROM $db_corps_table ORDER BY cName;", $con1); This is a valid query for database two. $fetchUsers = mysql_query("SELECT * FROM 'users' WHERE tsName LIKE $cTicker%;", $con2); However despite me specifying which database link to use it tries to use every query on database one ($con1). So I can never query database two ($con2) because it just breaks with the result below. :S $fetchUsers = mysql_query("SELECT * FROM $db_corps_table ORDER BY cName;", $con1); gives Quote AREA 43 [A-43] (11): result result result (there are 3 groups in database one) But: $fetchUsers = mysql_query("SELECT * FROM 'users' WHERE tsName LIKE $cTicker%;", $con2); gives Quote AREA 43 [A-43] (11): Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource in /home/sever/public_html/mav/tools/ts_corp_registrations.php on line 51 <?php // ===== Connect to database one (group) ===== $con1 = mysql_connect($db_host,$db_corps_user,$db_corps_pass); if (!$con1) { die("Could not connect:" . mysql_error()); } $db_select1 = mysql_select_db($db_corps_db, $con1); if (!$db_select1) { die ("Could not select database:" . mysql_error()); } // ===== Connect to database two (members) ===== $con2 = mysql_connect($db_host,$db_ts_user,$db_ts_pass, true); if (!$con2) { die("Could not connect:" . mysql_error()); } $db_select2 = mysql_select_db($db_ts_db, $con2); if (!$db_select2) { die ("Could not select database:" . mysql_error()); } $fetchCorps = mysql_query("SELECT * FROM $db_corps_table ORDER BY cName;", $con1); while ($row = mysql_fetch_array($fetchCorps)) { // For each of the results, gather details from database one, then see how many members we can find in database two $cName = $row["cName"]; $cTicker = mysql_real_escape_string($row["cTicker"],$con1); $cMembers = $row["cMembers"]; echo $cName." [".$cTicker."] (".$cMembers."):<br />"; // ========================== HERE BE PROBLEMS ========================== //$fetchUsers = mysql_query("SELECT * FROM $db_corps_table ORDER BY cName;", $con1); $fetchUsers = mysql_query("SELECT * FROM 'users' WHERE tsName LIKE $cTicker%;", $con2); // =================================================================== while ($rowTS = mysql_fetch_array($fetchUsers)) { echo "result<br />"; } } // all done, close DB connections mysql_close($con2); mysql_close($con1); ?> Hi, Just a quick query. I call my db connection and configuration files on every script. I also have just made a functions script where i'm storing some functions except when i call the function, my script fails because of a missing database connection. But on the page that fails, i call the configuration files including the db connection at the start of the script. But when i get to where i include the function they are all on the same script so should it fail?? The page did work fine when the function was inbedded in the script. But as i'm using the function on a few pages, i thought it would be better practise and more efficent to just include the function on each page i need, rather than just copy and paste all the php in the required pages. Or do i have to declare the db connection inside the function page as well??? Thanks Hello, I'm trying to update my database with current statistics from my gameservers, but running the loop of socket connections to all of the servers on page load takes too long. For every server in the array it takes roughly 4 seconds to complete each one.. with a massive list of say 42 it takes awhile. However, I'm running from shared hosting and was wondering if that was the reason each server takes so long to query. I was thinking about running a cron job every 1 minute to update the information so it's still relativity new and current. Will the 1 minute cron job affect anything from the shared hosting? I'm using 1and1 web-hosting on a 1&1 Business Package. Thanks Hi everybody, I am facing "Too many connections" problem on my site. in fact the server is moved from one location to,another and after the movement this problem arise... Pls help. Thanks At the moment I am writing an application that supports plugins. Every plugin implements an interface so that the main application can control the plugin with the interface methods. There could be plugins that need a database connection and others that do not. What I am doing at the moment is loading every plugin with a __construct($databaseHandler). For plugins that don't need a database connection this is a little overhead. I could also use a Singleton or something else "staticish" so that the plugin can get the database instance whenever it wants, but then the plugin can only be used for my application where this specific Singleton is present and maybe I would like to use the plugin for other purposes later in another environment without changing it. Do you have any better ideas or how do you normally manage your database connections? Hi Have an issue with a script that connects to multiple Access databases to extract data. There is one master database and then numerous small databases (I take no responsibility for the design!). The master is opened and then the sub databases are looped around, opened process and closed in turn. However after about 20 connections I get the error [ODBC Microsoft Access Driver] Too many client tasks for any new connections. It is related to the number of connections rather than the number of operations on databases (ie, if I comment out one of the pieces of SQL run on each database it makes no difference). I am closing the connection and unsetting the variable that stores the connection. As such there shouldn't be an excess of connections open at any one time. Any suggestions? All the best Keith Hi, I am quite new to using php classes so it maybe something simple here, however I am trying to establish to connection to two different databases using one PHP class; The problem is that this will work for one but when I create a new connection they will both fail.
<?php putenv("TZ=Europe/London"); // start database connection $funky_db = new Database('xxxx', 'xxxx', 'xxxx', 'xxxx'); // start database connection $funky_cc = new Database('xxxx', 'xxxx', 'xxxx', 'xxxx'); class Database { private $host; private $user; private $pass; private $name; private $link; private $error; private $errno; private $query; function __construct($host, $user, $pass, $name = "", $conn = 1) { $this -> host = $host; $this -> user = $user; $this -> pass = $pass; if (!empty($name)) $this -> name = $name; if ($conn == 1) $this -> connect(); } function __destruct() { @mysql_close($this->link); } public function connect() { if ($this -> link = mysql_connect($this -> host, $this -> user, $this -> pass, TRUE)) { if (!empty($this -> name)) { if (!mysql_select_db($this -> name)) $this -> exception("Could not connect to the database!"); } } else { $this -> exception("Could not create database connection!"); } } public function close() { @mysql_close($this->link); } public function query($sql) { if ($this->query = @mysql_query($sql)) { return $this->query; } else { $this->exception("Could not query database!".$this->name); return false; } } public function num_rows($qid) { if (empty($qid)) { $this->exception("Could not get number of rows because no query id was supplied!"); return false; } else { return mysql_num_rows($qid); } } public function fetch_array($qid) { if (empty($qid)) { $this->exception("Could not fetch array because no query id was supplied!"); return false; } else { $data = mysql_fetch_array($qid); } return $data; } public function fetch_array_assoc($qid) { if (empty($qid)) { $this->exception("Could not fetch array assoc because no query id was supplied!"); return false; } else { $data = mysql_fetch_array($qid, MYSQL_ASSOC); } return $data; } public function fetch_object($qid) { if (empty($qid)) { $this->exception("Could not fetch object assoc because no query id was supplied!"); return false; } else { $data = mysql_fetch_object($qid); } return $data; } public function fetch_all_array($sql, $assoc = true) { $data = array(); if ($qid = $this->query($sql)) { if ($assoc) { while ($row = $this->fetch_array_assoc($qid)) { $data[] = $row; } } else { while ($row = $this->fetch_array($qid)) { $data[] = $row; } } } else { return false; } return $data; } public function last_id() { if ($id = mysql_insert_id()) { return $id; } else { return false; } } private function exception($message) { if ($this->link) { $this->error = mysql_error($this->link); $this->errno = mysql_errno($this->link); } else { $this->error = mysql_error(); $this->errno = mysql_errno(); } if (PHP_SAPI !== 'cli') { ?> <div class="alert-bad"> <div> Database Error </div> <div> Message: <?php echo $message; ?> </div> <?php if (strlen($this->error) > 0): ?> <div> <?php echo $this->error; ?> </div> <?php endif; ?> <div> Script: <?php echo @$_SERVER['REQUEST_URI']; ?> </div> <?php if (strlen(@$_SERVER['HTTP_REFERER']) > 0): ?> <div> <?php echo @$_SERVER['HTTP_REFERER']; ?> </div> <?php endif; ?> </div> <?php } else { echo "MYSQL ERROR: " . ((isset($this->error) && !empty($this->error)) ? $this->error:'') . "\n"; }; } } ?> Does anyone have an example of establishing two MySQL connections (one local & one external) to pass data from local DB to external using PHP? Objective: I need to query the table of local data (for email addresses) then the same for external db table and if email not present, insert into that db. I will use CRON to run the script (will figure that part out later but suggestions welcome if you have that info as well....) I have been reading in Larry Ullmans book "Visual QuickPro Guide PHP 6 and MySQL 5" and I find it well-written. Since it is from 2008, it does not contain anything about MySQL PDO, but rather does it in the mysqli_* way. Larry suggest placing the secret database password and more along with a database connection script in a connect.php file, placed above the webroot if possible. Then later, when he is creating queries and executing them in other php files, he includes the connect.php file before making the queries.
Now I know it is very important to be careful with the Error handling, so the script won't output errors, which could reveal something about the database making it less secure. Therefore I am wondering how to structure things when using PDO. I need to write error-handling scripts for the following situations:
a) Connection to the database doesn't succeed
b) The execution of the queries doesn't succeed
c) User input in HTML forms are not appropriate
and probably more. The recommended way of handling errors when using PDO seems to be writing some try-catch code. But then I don't see how I can keep the connection to the database completely inside the connect.php file. Either I will need to use a die() or exit() inside this file or I will need to give up my idea to keep everything which concerns the connection to the database in the file mentioned AND write nested try-catch sentences - first make sure the connection works, then make sure the query will execute properly.
I don't like either of those approaches. Firstly I have been told that using exit() is bad programming and secondly it seems to get more complicated using nested try-catch code and to let database connection take part in diverse php-files.
Maybe somebody have a smart, convenient and secure way to do it?
Erik
after inserting data i get 0 sometimes i created a mysql_connect for it Code: [Select] <?php $maincon = mysql_connect('local', "root", "root"); mysql_select_db("root"); then //insert and $success = mysql_query($query); echo $ref_id = mysql_insert_id($maincon); ?> $ref_id is returned as 0, although the connection is there var_dumping the query returns the values How do I code a persistent connection with mysqli and PHP? I know back in mysql it was mysqli_pconnect, what is it for mysqli? Please do not refer to this page (http://www.php.net/manual/en/mysqli.persistconns.php) as it gives no example and makes little sense. Many thanks in advance. Over the past few days I have been running into some issues with my server crashing due to apache max connections issues. I am running my site off of a hosted Cloud VPS with 200GB of storage, 8192MB Memory, 8TB of transfer, Apache, MySQL, PHP5, and CentOS.
I am afraid the issue doesn't necessarily lay in the configuration of Apache, but the way I have scripted the php on my site, the reason I am reaching out here. My site isn't your average website, it is more of a web-based customer management program. There are currently only 2 pages you can actually access via the url bar (signin.php and index.php). All other content is loaded via AJAX and JQuery processes (.load and $.getScript).
All AJAX requests are pointed toward a single file called functions.php where a _POST parameter contains the function name and any additional _POST data required by the function.
FOR EXAMPLE:
AJAX Call:
$.post('functions/functions.php',{func:'myFunctionName',ops:'whatever',a:'whatever',b:'whatever'},function(data){ DO WHATEVER I WANT WITH THE RETURN DATA HERE },"json");PHP (functions.php) require 'dbcon.php'; include 'main_class.php'; include 'f_customerdetail.php'; include 'f_listoptions.php'; include 'f_route.php'; include 'f_useractions.php'; if (isset($_POST['func'])){ $userfunc = $_POST['func']; $funcops = $_POST['ops']; if ($funcops != ''){ $userfunc($funcops); }else{ $userfunc(); } } Hi All
I have not really played around with PHP in ages so I am having a hard time trying to figure out the best way to proceed. Anyhow... what I need to get done is to take a singleton pattern core database class that has a extended driver based class (ie; the type of database server the connection is connecting to), and build a new class that dynamically handles as many driver based connections that are called using a single instance. As a side note, I tried PDO but it only supports a single driver per class instance, and then each of those connections cannot not have their on set of properties that relate to each connection. Anyway, I was thinking that the best way to handle all the driver specific connections, is to hand out a 'unique hash reference' that points to an array of connection objects and the object properties. then when the client runs any sql function they pass the 'unique hash reference' which the class then returns the object and it properties that will be used by the class to call up the driver specific sql function that was requested by the client. So what do you all thing about that....
TIA stephanieT
PS...
I reason i need this is because I need to update up 25 different databases based on data stored on all the databases and not all of the database servers are of the same type, (ie; MSSQL, Oracle, MySQL, MariaDB, berkeley DB, postgres, sqlite2,3, etc, etc)! Edited January 16, 2019 by StephanieT |