PHP - File_get_contents And Curl Not Working For Some Url
Hi everyone
I've look all other the web for a work around that.
basically my problem is:
if i use file_get_contents or curl to get www.google.fr, there is NO problem. i get the data i want
but when i try to use it on another target it doesn't work. it loads and loads for couple seconds and finally says that it could not retrieve informations.
the target is not blocked as i can personnally use it on my webhost, but some people on other webhosts can't.
A friend of mine told me that it was because some webhosts blocks unsecure targets.
Is there any way around it?
$context = stream_context_create(array( 'http' => array( 'header' => "Authorization: Basic " . base64_encode("admin:$password") ) )); $data = file_get_contents("http://TARGETHOST:29015/status.json", false, $context); $status = json_decode($data); var_dump($status);(And again this works on my webhost, i can query like that any servers i want, but others can't use this they get a timeout trying it (when doing a file_get_contents on google works)) Similar TutorialsA German DB that collects all the data from all German Foundations... see: http://www.suche.stiftungen.org/index.php?strg=87_124&baseID=129 Here we find all Foundations in Germany: : 8074 different foundations You get the full results if you choose % as wildcard in the Search-field. But if we do that - then you get some kind of overflow... 350 results are the limit. More is not possible to show. So the question is: How can we create a spider that runs across the site and asks step by step - that we get all : 8074 results. The way to get through this database is to search combinations of letters eg "ac" and select search only titles. Then go through every pair of letters. If you still get too many results for a particular pair, use 3 letters. aca, acb,... Can i do this with File_get_contents_ or with Curl!? (eg -MultiCurl ) Well - i want to make a little script thatdoes this - i need to create a little automation - that does this task automatically. Regarding the destination database, it's all going into sqlite, which we believe can handle large enough sets of data without any problems. We can download the database as a file too. For capabilities, see he http://www.sqlite.org/limits.html The question is _ how to create the first approach of the parser...!`? Can any body assist! All, I'm trying to use some code to get some file contents from another website and am looking at performance. What is the quicker way and better for my resources to get out to websites, is it cURL or by using file_get_contents? Thanks in advance. I would like to spoof the useragent as googlebot through the implementation of file_get_contents, or curl (which ever is easiest). How to do this? Googlebot useragent = Googlebot/2.1 (+http://www.google.com/bot.html) Hi guys, I have a very simple php script that's supposed to work but doesn't. The script looks like this... <?php $response = file_get_contents("http://localhost:8090/MyWebApp/MyServlet", false, NULL) echo $response ?> which seems to return nothing. This is supposed to call a java servlet on the local machine. However, the servlet does not get called (I put a breakpoint in the servlet so I'd know if it were called). When I navigate to this php from a web browser, I get back an empty page (as if there's an error or something). However, if I change the above line to use http://www.google.com as the URL, it works (I can hit the php page from my browser and get back a google-looking page, minus some pictures and what not). For some reason, the call to a localhost URL does not seem to work, even though the servlet is, in fact, running. I have tested this by calling "curl http://localhost:8090/MyWebApp/MyServlet" from my linux terminal and correctly getting the servlet response. Does anyone know why this does not work in PHP? Thanks, --Ronen Hello there, The error i have is that my server can't find itself and it gives me this nice error. Quote php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in... You maybe thinking why don't i just include the file or use its relative location (./file.php) but the answer is simple. It doesn't fit in with my framework and its too hard to expand. How else can I execute a file on the same domain using PHP? Thank-you Paul I have an integration with a third party that utilizes a SOAP API. On my current server AND my local environment, this works fine. I recently deployed a new server and it seems that cURL isn't doing anything. *The code is identical on all three installations. The "error" that I am seeing is no response at all (no response and no error, however, I can see that it is a "Resource"). After much trial and error, I tried creating a simple cURL example to test try to rule out other issues as the code itself is quite complex. My simple example, that also fails, is as follows: $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "google.com"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $output = curl_exec($ch); $err = curl_error($ch); print_r($err); echo $output; curl_close($ch); This does the same thing. On local, I get a "301 Moved" notice, on the new server, it is just a white screen. I have confirmed that php has curl installed in via command line (php -m) and using phpinfo(); in my code. It is definitely enabled. What would cause cURL to not have a response AND not have an error? My thought is a server setting, but I can't seem to find what might be causing this. Edited July 7, 2020 by rklocknerHi, I want to fetch some xml data from google website to my website using curl library. i tried. it display nothing to me even my error display is enabled. can any one guide me with small example how can i get sml content using curl. Thanks I have never used cURL before but I need to use it for developing a facebook application. What I am trying to do is create a status update scheduler. There are two tables in my database. One table has generic updates for each day of the week an the other had special events. My script checks the day, checks to see if there is a special event in the future on that day, and if there is reminds guests of the upcoming event. If there is no event it posts the generic status. Code: [Select] <?php include('config.php'); $result2 = mysql_query("select facebook_id,facebook_access_token from admin where username='users'"); $row = mysql_fetch_array($result2); $facebook_id = $row['facebook_id']; $facebook_access_token = $row['facebook_access_token']; $day = date('N'); $result2 = mysql_query("select * from events where date > CURDATE() order by date ASC") or die(mysql_error()); $num_rows = mysql_num_rows($result2); if($num_rows == 0) { $result2 = mysql_query("select * FROM alerts WHERE id='$day'") or die(mysql_error()); $row1 = mysql_fetch_array($result2); echo 'There are no events just alerts'; } else { while($row = mysql_fetch_array($result2)){ if(date('N',strtotime($row[date])) == $day) { $id = $row['id']; $result1 = mysql_query("SELECT * FROM events WHERE id = $id") or die(mysql_error()); $row1 = mysql_fetch_array($result1); } else { $result1 = mysql_query("SELECT * FROM alerts WHERE id = $day") or die(mysql_error()); $row1 = mysql_fetch_array($result1); } } } $params = array('access_token' => $facebook_access_token, 'message' => $row1[alert_message], 'name'=> $row1[alert_name], 'caption' => $row1[alert_caption], 'link' => $row1[alert_link], 'description' => $row1[alert_description], 'picture' => $row1[alert_picture] ); $url = "https://graph.facebook.com/$facebook_id/feed"; $ch = curl_init(); curl_setopt_array($ch, array( CURLOPT_URL => $url, CURLOPT_POSTFIELDS => $params, CURLOPT_RETURNTRANSFER => true, CURLOPT_SSL_VERIFYPEER => false, CURLOPT_VERBOSE => true )); $result = curl_exec($ch); ?> The problem seems to be occurring when the While loop is accessed. I have used 'echo' to follow the variables through the script and it all seems to be working. However if I try and execute the cURL and the While Loop has run then it does not seem to work. I know the cURL part works becuase if the variables cause the first IF statement default to ELSE then it runs fine. Any help would be GREATLY appreciated. Users make entries into a form on my site which are then sent to a remote server which generates a variable length page of results for the users perusal. I want the results to be displayed on my site but I discover that iframe height is a problem so I am trying PHP in a separate file to avoid an iframe. Then maybe I can cache the result pages and display them without any height difficulties and not having to fiddle with javascript. The first PHP code I try is prevented from working and I discover that the apparent cause is the issue of fopen and fsockopen etc. being set to off by the host that I use because of security concerns: $url="http://remote_server.cgi"; foreach($_POST as $key => $value) {$url .="$key=" . urlencode($value) . "&";} $array = file($url);] The host has cURL so I try that but the form entries are not being accepted as they are with a direct post from the form: $URL="http://remote_server.cgi"; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $URL); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST); curl_exec ($ch); curl_close ($ch); In case useragent is a problem at the remote server, I added: curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'); I know all entries are received by the separate file by using: print_r($_POST); But the remote server still says "Error in form found. You are not authorized etc. etc... " Apparently the post data is not being sent like the form sends it. Any suggestions? Ive tried everything and cannot get this POST to work, page comes up blank. The output of curlgetinfo() shows: Code: [Select] Array ( [url] => http://app.alliedinsurance.com/find_agent/calcpage4_1_popup.cfm?RequestTimeout=180 [content_type] => text/html; charset=UTF-8 [http_code] => 200 [header_size] => 329 [request_size] => 450 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0.172622 [namelookup_time] => 0.02064 [connect_time] => 0.092116 [pretransfer_time] => 0.092244 [size_upload] => 0 [size_download] => 509 [speed_download] => 2948 [speed_upload] => 0 [download_content_length] => 0 [upload_content_length] => 0 [starttransfer_time] => 0.172563 [redirect_time] => 0 ) The whole code Im using: Code: [Select] <h2>Please Enter Zip Code</h2> <p> <form method="POST" action="<?php echo $_SERVER['PHP_SELF'];?>"> <input type="text" size="10" maxlength="10" name="zipcode" tabindex="1" value="<?php echo $_POST['zipcode'];?>" /> <input type="submit" value="Search" name="submit" tabindex="2" /> </form> <br /> <?php if(isset($_POST['submit'])) { $zipcode = $_POST['zipcode']; $userAgent = 'Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1'; $mercurl = "http://app.alliedinsurance.com/find_agent/calcpage4_1_popup.cfm?RequestTimeout=180"; $postcom = "City=&State=AR&ZipCode=" . urlencode($zipcode) . "&Miles=" . urlencode('20') . "&SubmitThis=Submit"; $ch = curl_init(); curl_setopt($ch, CURLOPT_USERAGENT, $userAgent); curl_setopt($ch, CURLOPT_URL,$mercurl); curl_setopt($ch, CURLOPT_FAILONERROR, true); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER,true); curl_setopt($ch, CURLOPT_TIMEOUT, 100); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_HEADER_OUT, true); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true); curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); curl_setopt($ch, CURLOP_REFERRER, "http://app.alliedinsurance.com/find_agent/find_an_agent_popup.cfm"); curl_setopt($ch,CURLOPT_POST, 1); curl_setopt($ch,CURLOPT_POSTFIELDS,$postcom); $html = curl_exec($ch); print_r(curl_getinfo($ch)); $html = @mb_convert_encoding($html, 'HTML-ENTITIES', 'utf-8'); curl_close( $ch ); echo $html; ?> I am trying to create a remote login to one website using mine. The users will need to enter their username and password on my site, and if they are registered to my website, their login credentials will be sent to another website and a page will be retrieved.
I am stuck at sending the users' data to the original site. The original site's viewsource is this..
<form method=post> <input type="hidden" name="action" value="logon"> <table border=0> <tr> <td>Username:</td> <td><input name="username" type="text" size=30></td> </tr> <tr> <td>Password:</td> <td><input name="password" type="password" size=30></td> </tr> <td></td> <td align="left"><input type=submit value="Sign In"></td> </tr> <tr> <td align="center" colspan=2><font size=-1>Don't have an Account ?</font> <a href="?action=newuser"><font size=-1 color="#0000EE">Sign UP Now !</font></a></td> </tr> </table>I have tried this code, but not works. <?php $username="username"; $password="password"; $url="http://www.example.com/index.php"; $postdata = "username=".$username."&password=".$password; $ch = curl_init(); curl_setopt ($ch, CURLOPT_URL, $url); curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); curl_setopt ($ch, CURLOPT_TIMEOUT, 60); curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt ($ch, CURLOPT_REFERER, $url); curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata); curl_setopt ($ch, CURLOPT_POST, 1); $result = curl_exec ($ch); header('Location: track.html'); //echo $result; curl_close($ch); ?>Any help would be appreciated, Thanks in advance. I have a CURL code that fetches some XML content from a remote server. I get the expected output on my localhost (XAMPP on windows). But from the live site(hosted on Linux), I get the CURL error:'couldn't connect to host'. function DownloadUrl($Url, $p){ // is curl installed? if (!function_exists('curl_init')){ die('CURL is not installed!'); } // create curl resource $ch = curl_init(); // set url curl_setopt($ch, CURLOPT_URL, $Url); curl_setopt ($ch, CURLOPT_POSTFIELDS, $p); curl_setopt ($ch, CURLOPT_POST, 1); //return the transfer as a string curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // $output contains the output string $output = curl_exec($ch); if($output === false) echo 'Curl error: ' . curl_error($ch); else print $output; // close curl resource to free up system resources curl_close($ch); } Hello PHP Freaks! I have the following PHP Curl code which is supposed to fetch the twitter timeline feed for a specific username and parse and display it on a page, but for some reason Curl is unable to fetch data from twitter: Code: [Select] <?php function get_data($url) { $ch = curl_init(); $timeout = 5; curl_setopt($ch,CURLOPT_URL,$url); curl_setopt($ch,CURLOPT_RETURNTRANSFER,1); curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout); $data = curl_exec($ch); curl_close($ch); return $data; } $json = get_data("https://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&screen_name=digioz&count=2"); if ($json != false) { $obj = json_decode($json); foreach($obj as $var => $value) { echo "Message number: $var <br/>"; echo "Name: " . $obj[$var]->user->name; echo "Handle: " . $obj[$var]->user->screen_name . "<br/>"; echo "Message: " . $obj[$var]->text; echo "Created" . $obj[$var]->created_at . "<br/>"; echo "URL" . $obj[$var]->user->url . "<br/>"; echo "Location" . $obj[$var]->user->location . "<br/>"; echo "<br/>"; } } else { echo "Could not fetch Twitter Data"; } ?> Anyone have any idea why the data is not being fetched? If I copy and paste the URL into my browser window it returns results just fine, so I know the problem is not with the URL. Thanks, Pete Hi All, My goal is to process all redirects serverside and simply return the user to the 'final' URL. Here's my function to get the final URL: PHP Code: function get_final_url($url){ global $final_host; $redirects = get_all_redirects($url); if (count($redirects)>0){ $final_url = array_pop($redirects); if(substr($final_url,0,7) != 'http://' && !empty($final_host)) { $final_url = 'http://'.$final_host.$final_url; } return $final_url; } else { return $url; } Here's the cURL execution: $ch2 = curl_init(); curl_setopt($ch2,CURLOPT_URL,$url); curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, true); curl_setopt($ch2,CURLOPT_POST,''); curl_setopt($ch2,CURLOPT_POSTFIELDS,''); curl_exec($ch2); curl_close($ch2); For some reason, this works no problem, posts any data over to the URLs serverside, redirects serverside, and passes the client to the final URL, with none of the redirection displayed via HTTPWatch or any similar debug utility. Sometimes, however, it does show phases of redirection. Any insight into what I might be doing incorrectly? Thanks SO much in advance. E good day dear community, i am workin on a Curl loop to fetch multiple pages: i have some examples - and a question: Example: If we want to get information from 3 sites with CURL we can do it like so: $list[1] = "http://www.example1.com"; $list[2] = "ftp://example.com"; $list[3] = "http://www.example2.com"; After creating the list of links we should initialize the cURL multi handle and adding the cURL handles. $curlHandle = curl_multi_init(); for ($i = 1;$i <= 3; $i++) $curl[$i] = addHandle($curlHandle,$list[$i]); Now we should execute the cURL multi handle retrive the content from the sub handles that we added to the cURL multi handle. ExecHandle($curlHandle); for ($i = 1;$i <= 3; $i++) { $text[$i] = curl_multi_getcontent ($curl[$i]); echo $text[$i]; } In the end we should release the handles from the cURL multi handle by calling curl_multi_remove_handle and close the cURL multi handle! If we want to another Fetch of sites with cURL-Multi - since this is the most pretty way to do it! Well I am not sure bout the string concatenation. How to do it - Note I want to fetch several hundred pages: see the some details for this target-server sites - /(I have to create a loop over several hundred sites). * siteone.example/?show_subsite=9009 * siteone.example/?show_subsite=9742 * siteone.example/?show_subsite=9871 .... and so on and so forth Question: How to appy this loop into the array of the curl-multi? <?php /************************************\ * Multi interface in PHP with curl * * Requires PHP 5.0, Apache 2.0 and * * Curl * ************************************* * Writen By Cyborg 19671897 * * Bugfixed by Jeremy Ellman * \***********************************/ $urls = array( "siteone", "sitetwo", "sitethree" ); $mh = curl_multi_init(); foreach ($urls as $i => $url) { $conn[$i]=curl_init($url); curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout curl_multi_add_handle ($mh,$conn[$i]); } do { $n=curl_multi_exec($mh,$active); } while ($active); foreach ($urls as $i => $url) { $res[$i]=curl_multi_getcontent($conn[$i]); curl_multi_remove_handle($mh,$conn[$i]); curl_close($conn[$i]); } curl_multi_close($mh); print_r($res); ?> I look forward to your ideas. <?php $1 = file_get_contents(""); $2 = json_decode($1); $3 = $2->result; var_dump($myData1); foreach($3 as $key => $item): echo $item->Users->UserObject->UserName; endforeach; ?> { "result": [ { "ID": 1, "Users": [ { "UserObject": { "UserName": "test1" }, "...": 2 }, { "UserObject": { "UserName": "test2" }, "...": 2 } ], "IsActive": true } ], "error": false }
I get this error Notice: Trying to get property 'UserObject' of non-object in C:\xampp\htdocs\dashboard-master\chart.php on line 10 This works great but I only need part of the page for example the right column...can I do this with a class or some how. What this is is pulling information from another site. Code: [Select] <?php $g = file_get_contents("http://www.site.com/index.php"); echo ($g); ?> Okay, so I'm trying to make a service to detect proxies. Heres what I need to know.. This is how I have it set up: 1. I have a file on mysite.com/check.php 2. My clients create a page to load my check.php file on THEIR website using file_get_contents(); Like: file_get_contents("http://mysite.com/check.php?ip=$_SERVER['REMOTE_ADDR']"); 3. My system checks for a proxy using HTTP vars, how would I get them?? Like.. if I echo $_SERVER['REMOTE_ADDR']; out on the page on MY end, it will echo the servers IP to check is using step #2 with file_get_contents(); Can anyone please help me? I am willing to pay a little, if needed :/ Thanks in advance! Hello I wanted to query something from an API, but now I have 2 times money see example below. How can I only output one of them I have already looked for a solution but cannot find any or I am looking wrong { "result": [ { "Money": 45674569.88764 }, { "Money": 546.6544356456 } ], "test": 0 } <?php $myData = file_get_contents(""); $myObject = json_decode($myData); $myObjectMap = $myObject->result; ?> <?php foreach($myObjectMap as $key => $item): ?> <?PHP echo $item->Money; ?>
|