PHP - Php Curl Proxy Problem

Hi,   I'd stumbled upon a problem around doing curl on an asp website which use cookie.   I use username n session id to determine the cookie file, on a case when a second browser open up it should have different session id, hence it may create new cookies instead of using the active previously-made cookie file, please do correct me if I'm wrong.   So question is....how to detect a user is logged or not in such situation ? Or perhaps how is the proper way to determine a cookie file in such situation ?     Thanks in advance,

Hi
I made a curl and don't know what to do with the string I got back, I would need the id and the name of each, should I explode, don't know how to do that, here is what I got as answer:

Quote

string(596) "{"data":[{"name":"Esoterik Forum","category":"Community","id":"180883525283576"},{"name":"Hellsehen und Wahrsagen","category":"Company","id":"177967172214131"},{"name":"Spirituelle Lebensberatung","category":"Website","id":"164411270264299"},{"name":"Esoterik","category":"Website","id":"164524456919432"},{"name":"Kartenlegen und Hellsehen","category":"Company","id":"173933135951523"},{"name":"Exchange","category":"Application","id":"190533567633445"},{"name":"Astrologie","category":"Website","id":"164928140213920"},{"name":"Esoterik Forum","category":"Application","id":"184577371575355"}]}"

I have this code


curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_URL, $link);
$wall = curl_exec($ch);


but the $link is a facebook wall page that takes a couple of seconds to load fully. and you have to scroll down the page before more loads or you have to click older posts

when i run the script the $wall doesnt have the full page.

Is there a way that i can delay the $link from saving the file so that it reads more of the page??

Hey guys,
I have developed an script which opens or fetches a page using curl, now the problem is the url i am trying to fetch follows lot of redirects(say 302) and in between it stores several cookies also which i don't know or guess.
So i was just thinking, is there any way so that i can automatically store the cookies in a particular file which server wants to store, and let it sent in request as per the asking of a request  in request header?

I hope you would have got what i mean to say here, for example
suppose i want to open http://www.example.com
and now that above url faces lot of redirects probably 5-6 and then ends up on http://www.test.com

now in between it stores some cookies in clients computer and then it fetches them also. Since i don't know which cookie it gonna store, I must put this responsibility to PHP itself to store and send the requested cookies in request headers.

Also i want to send the referrer also in request header, and it must be the last page that or request sent to the server, no matter whether it's in betwen redirects or to the final page.

In short i mean to create the complete emulation of browser for this URL.

good day dear community,

i am workin on a Curl loop to fetch multiple pages:

i have some examples - and a question:

Example: If we want to get information from 3 sites with CURL we can do it like so:


$list[1] = "http://www.example1.com";
$list[2] = "ftp://example.com";
$list[3] = "http://www.example2.com";


After creating the list of links we should initialize the cURL multi handle and adding the cURL handles.


$curlHandle = curl_multi_init();
for ($i = 1;$i <= 3; $i++)
 $curl[$i] = addHandle($curlHandle,$list[$i]);


Now we should execute the cURL multi handle retrive the
content from the sub handles that we added to the cURL multi handle.

ExecHandle($curlHandle);
for ($i = 1;$i <= 3; $i++)
{
 $text[$i] =  curl_multi_getcontent ($curl[$i]);
 echo $text[$i];
}

In the end we should release the handles from the cURL multi handle by calling curl_multi_remove_handle and close the cURL multi handle!

If we want to another Fetch of sites with cURL-Multi - since this is the most pretty way to do it!
Well I am not sure bout the string concatenation. How to do it - Note I want to fetch several hundred pages:
see the some details for this target-server sites - /(I have to create a loop over several hundred sites).

    * siteone.example/?show_subsite=9009
    * siteone.example/?show_subsite=9742
    * siteone.example/?show_subsite=9871

.... and so on and so forth Question: How to appy this loop into the array of the curl-multi?



<?php
/************************************\
* Multi interface in PHP with curl  *
* Requires PHP 5.0, Apache 2.0 and  *
* Curl         *
*************************************
* Writen By Cyborg 19671897         *
* Bugfixed by Jeremy Ellman         *
\***********************************/

$urls = array(
   "siteone",
   "sitetwo",
   "sitethree"
   );

$mh = curl_multi_init();

foreach ($urls as $i => $url) {
       $conn[$i]=curl_init($url);
       curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string 
       curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects
       curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects
       curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout
       curl_multi_add_handle ($mh,$conn[$i]);
}

do { $n=curl_multi_exec($mh,$active); } while ($active);

foreach ($urls as $i => $url) {
       $res[$i]=curl_multi_getcontent($conn[$i]);
       curl_multi_remove_handle($mh,$conn[$i]);
       curl_close($conn[$i]);
}
curl_multi_close($mh);


print_r($res);

?>





I look forward to your ideas.

Hi All, I have been having issues for several days uploading a simple pdf file via Curl to one of our service providers servers.  I'm doing an is_uploaded_file check and getting a successful response, but the document is not actually being uploaded to the server. Can anyone help?? Please see code below:

Code: [Select]
$filename = $_FILES['activity_doc']['name']; // Name of the file (including file extension).
$upload_path = https://upload_url

//Curl session initialized
$session = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($session, CURLOPT_URL, $upload_path);
curl_setopt($session, CURLOPT_USERPWD, 'username:password');
curl_setopt($session, CURLOPT_POST, 1);


$post = array(
        "file_box"=>"@" . $filename,
    );


curl_setopt($session, CURLOPT_POSTFIELDS, $post);
$response = curl_exec($session);

if (is_uploaded_file($_FILES['activity_doc']['tmp_name'])) {
   echo "File ". $_FILES['activity_doc']['name'] ." uploaded successfully.\n";
} else {
   echo "Error occurred, file uploaded unsuccessful";
}

$req = new SimpleXMLElement($response);
print_r($req);
curl_close($session);

Hello,
   First of all ,I have been a regular reader of the forum and it has increased my knowledge in the PHP a lot. 
 Till now i had just been a guest user and used to go through the queries and tips posted here.

Today , for the first time i am posting on the forum

Coming to a problem:
 I have to design a function for a client from travel domain. Where he wants the script/function to go and extract data from an airline website on the available fare for a specific date and destination. The problem is that only a part of fare(Base fare) is visible on the site and you have to select one of the given flight option(s) (after selecting the option through AJAX the full fare is displayed on the same page on the left hand site) to get the complete fare including fare and surcharges.

Now i want my function/script to extract the full fare for all the available flight options. I have written the below piece of code. But its not efffective as it gets only the base fare part of the result page and not the complete fare.. 

The airline booking url is : http://book.spicejet.com/skylights/cgi-bin/skylights.cgi

My code:

//---- Spicejet start
// ENGINE URL http://book.spicejet.com/skylights/cgi-bin/skylights.cgi?page=homecomp

//Taking some sample search data - just to test the script
$event="search" ;
$module="SB" ;
$page="SEARCH" ;
$mode="empty" ;
$travel="1" ;     // ONE WAY OR 2 WAY
$from1="HYD" ;
$to1="DEL" ;
$departDay1="13" ;
$departDay2="";
$departMonth1="201010" ;
$ADULT="1" ;
$CHILD="0" ;
$INFANT="0" ;
$toCity1="DEL" ;
$numberMarkets="1" ;

// the page needs data in POST form
$curlPost = 'event='  . urlencode($event) . '&module=' . urlencode($module) .'&page=' . urlencode($page) .'&mode=' . urlencode($mode) .'&travel=' . urlencode($travel) .'&from1=' . urlencode($from1) .'&to1=' . urlencode($to1) .'&departDay1=' . urlencode($departDay1) .'&departMonth1=' . urlencode($departMonth1) .'&ADULT=' . urlencode($ADULT) .'&CHILD=' . urlencode($CHILD) .'&INFANT=' . urlencode($INFANT) .'&toCity1=' . urlencode($toCity1) .'&numberMarkets=' . urlencode($numberMarkets).'&submitSearch=submitSearch';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://book.spicejet.com/skylights/cgi-bin/skylights.cgi');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost);
$data = curl_exec($ch);
curl_close($ch);

echo($data); 



Also one more query Is there something like beautiful soup (As in Python) to parse the data. I want to extract the flight timing and total price from the page and insert into a database.

Thanxs in advance

Hi everybody !

I have this current problem .. I need to login into a website via cUrl .. website : www.v-tac [dot] ro/

Now based on the headers and based on the input fields I wrote a php function, but I hit a wall with the token .

HEADERS :

username=username&password=password&Submit=Conectare&option=com_users&task=user.login&return=aW5kZXgucGhwP0l0ZW1pZD0yMTY%3D&0dbf64fe20e2395a7d72ed5b64b3cf7c=1

FORM FIELDS - copy paste - this is the login form

<fieldset class="userdata">
    
<p id="form-login-username">
        <label for="modlgn-username">Nume Utilizator</label>
        <input id="modlgn-username" type="text" name="username" class="inputbox" size="18">
    </p>
    <p id="form-login-password">
        <label for="modlgn-passwd">Parola</label>
        <input id="modlgn-passwd" type="password" name="password" class="inputbox" size="18">
    </p>
        <p id="form-login-remember">
        <label for="modlgn-remember">Retine utilizator</label>
        <input id="modlgn-remember" type="checkbox" name="remember" class="inputbox" value="yes">
    </p>
    
    <input type="submit" name="Submit" class="button" value="Conectare">
    <input type="hidden" name="option" value="com_users">
    <input type="hidden" name="task" value="user.login">
    <input type="hidden" name="return" value="aW5kZXgucGhwP0l0ZW1pZD0yMTY=">
    <input type="hidden" name="11b09608b3184e6258012d44846c81ed" value="1">    

</fieldset>

And this is the function I wrote to do the cUrl login :

function login_to_website($targetURL){ 

    global $browser_user_agent; 
    if(empty($targetURL)) { return; } 
    if(empty($login_url)) { $login_url = $targetURL; } 
    $url = $login_url; 

    $login_user     = "loginusername"; 
    $login_password = "loginpassword"; 
    $thetoken       = "this-is-my-problem-the-token-from-the-hidden-input";         

    $post_data = array();    
    $post_data['username']  = "$login_user";  
    $post_data['password']  = "$login_password";  
    $post_data['Submit']    = "Conectare"; 
    $post_data['option']    = "com_users"; 
    $post_data['task']      = "user.login"; 
    $post_data['return']    = "aW5kZXgucGhwP0l0ZW1pZD0yMTY%3D"; 
    $post_data[$thetoken]   = "1";           

    $postthis = http_build_query($post_data); 

    $login = curl_init(); 

    curl_setopt($login, CURLOPT_COOKIEJAR, dirname(__FILE__) . "/cookie.tmpz"); 
    curl_setopt($login, CURLOPT_COOKIEFILE, dirname(__FILE__) . "/cookie.tmpz"); 
    curl_setopt($login, CURLOPT_VERBOSE, true); 
    curl_setopt($login, CURLOPT_URL, $url); 
    curl_setopt($login, CURLOPT_USERAGENT, random_user_agent()); 
    curl_setopt($login, CURLOPT_FOLLOWLOCATION, TRUE); 
    curl_setopt($login, CURLOPT_RETURNTRANSFER, TRUE);   
    curl_setopt($login, CURLOPT_POST, TRUE); 
    $timeout = 5; 
    curl_setopt( $login, CURLOPT_CONNECTTIMEOUT, $timeout ); 
    curl_setopt( $login, CURLOPT_TIMEOUT, $timeout ); 
    curl_setopt( $login, CURLOPT_MAXREDIRS, 10 );    

    curl_setopt($login, CURLOPT_POSTFIELDS, $postthis); // POST vars 

    curl_setopt($login, CURLOPT_HEADER, 0); // debug headers sent - 1 

      $data = curl_exec ($login); 

      curl_setopt($login, CURLOPT_URL, $targetURL); 

      $datax = curl_exec ($login); 
      return $datax; 

      // close cURL resource, and free up system resources 
      curl_close($login); 
}  

The problem is this the last array input. the token is generated each time the page is loaded, located on the page as an input hidden field .

So the question is how do I get a fresh token that will work ?

Also I have tried to get the token with a xpath extract like this :

    $htmlx = file_get_contents('http://www.v-tac.ro'); 
    $htmlx = mb_convert_encoding($htmlx, 'UTF-8', mb_detect_encoding($htmlx)); //make sure this is utf8 
    if(!strlen($htmlx)) {echo "No HTML here . stoping execution ."; return;} 
    $doc = new DomDocument; 
    @$doc->loadHTML($htmlx); 
    $xpath = new DOMXPath($doc); 

    echo $xpath->query('//fieldset[@class="userdata"]/input[5]')->item(0)->getAttribute("name"); 
    $thetoken = $xpath->query('//fieldset[@class="userdata"]/input[5]')->item(0)->getAttribute("name");  

Help !?

Hello,   I want to download flight from Wizzair page. I've checked all headers which are sended when browser "create" request (the same which Ryanair few months ago).   It looks like this in my browser:

Hi,   Well, I've been wondering if I can get answer to my problem here...   I'd used curl to fetch information from other site and then I need to parse the data to be displayed in our own. However I found something odd...I can't parse it to get what I want...however should I put the result into a html file, n parse the file itself...it's fine.   Basically I just need to find whether a word/phrase is there or not...n I'm using preg_match to do it so far. Since the time it first arises...I'd been googling n saw some posts...something similar to how regex is doing poorly in doing such thing...   However it does work when I just make a php file just to do that regex from the curl-result html file, it just not work on the actual php file which handle the process even if it's from the same curl-result html file. FYI, I'm using codeigniter. I wonder if it got something to do with codeigniter or something...or is there any other better way to parse a html file from a curl result. In my opinion it's just the same...be it a curl result or just a normal html...so I really can't figure out where it went wrong.   I appreciate if there's anyone who can shed a light on this problem, even a little bit is a great help to me       Thanks in advance,

Code: [Select]
<html>
<title>MyBB PM Exporter</title>
<body>
<form action ="<?php $_SERVER['PHP_SELF'] ?>" method="post">
URL to site: <input type ="text" name="site" value='community.mybb.com' /><br>
Full path to user:pwd list: <input type ="text" name="path" value='/home/www/list.txt' /><br>
<input type = "submit" value="Export"  />
</form>
</body>
</html>

<?php


if(isset($_POST['path']) && isset($_POST['site']))
{
$list = $_POST['path'];
$url = $_POST['site'];

$file = file($list);
foreach($file as $line){
$explode = explode(":",$line);
$username = $explode[0];
$password = $explode[1];



   $crl = curl_init();

    curl_setopt($crl, CURLOPT_URL, $url."/member.php");
    curl_setopt($crl, CURLOPT_COOKIEFILE, "/tmp/cookie.txt");
    curl_setopt($crl, CURLOPT_COOKIEJAR, "/tmp/cookie.txt");
    curl_setopt($crl, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($crl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($crl, CURLOPT_POST, 1);


    // This array will hold the field names and values.
    $postdata=array(
      "username"=>"$username", 
      "password"=>"$password",
      "remember"=>"no", 
      "submit"=>"Login",
  "action"=>"do_login", 
  "url"=>$url."/member.php"
    );
    // Tell curl we're going to send $postdata as the POST data
    curl_setopt ($crl, CURLOPT_POSTFIELDS, $postdata);

    $result=curl_exec($crl);
$find_key = preg_match('/var my_post_key = "(.*?)"/',$result,$match);
$my_post_key = $match[1];

curl_setopt($crl, CURLOPT_URL, $url."/private.php");
curl_setopt($crl, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($crl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($crl, CURLOPT_POST, 1);

    // This array will hold the field names and values.
$postdata=array(
"my_post_key" => $my_post_key,
"exportfolders[]" => "all",
"dayway" => "disregard",
"exportunread" => "1",
"deletepms" => "0",
"exporttype" => "html",
"action" => "do_export",
"submit" => "Export Private Messages"
);
curl_setopt ($crl, CURLOPT_POSTFIELDS, $postdata);

    
$fh = fopen($username.".html", 'w');
curl_setopt($crl, CURLOPT_FILE, $fh);
curl_exec($crl);
fclose($fh);
curl_close($crl);


}


}
?>
So , basically this code just opens the file where username:password is located and then it logins to community.mybb.com with each combination in that file and saves the content of pm's of logged in user.
Problem is that this script allways saves pm's of only one user and it saves other usernames pm's but with the same content of the first user.

Hi, i'm currently building essentially an API for my URL shortener site so other sites can implement it. But i've hit a stumbling block on the last script of the lot(the others merely collate the data and store it)

In this last script, i need to find all this data, stored in cookies and inout it into the database, as the main site does(i'm confident this bit works, as it's the code from the main site just copied across, and i can see the data in the database). Importantly, i then need o look in another table for the callback URL to send the data back to


$urltoshorten = $_COOKIE["URLtoShorten"];
$alias = $_COOKIE["urlalias"];
$key = $_COOKIE["key"];
//connection details go in here
$alias =mysql_real_escape_string($alias);
$urltoshorten = mysql_real_escape_string($urltoshorten);
$base = "http://s-url.co.uk/i/?id=";
$url = $base . $alias ;
//this is the main bit, where the URL shortening occurs
mysql_query("INSERT INTO table (id, name) VALUES('$alias', '$urltoshorten')")or die( mysql_error());
//this is where the callback url is looked for
$result= mysql_query("SELECT apiaddress FROM apistuff WHERE apinumber='$key'")or die( mysql_error());
$row = mysql_fetch_array($result);
$new = $row[apiaddress]; 
//I then start a CURL session to POST data to the page in question, as found above
$Curl_Session = curl_init($new);
curl_setopt ($Curl_Session, CURLOPT_POST, 1);
curl_setopt ($Curl_Session, CURLOPT_POSTFIELDS, "shortenedurl=$url");
curl_exec ($Curl_Session); 
curl_close ($Curl_Session);

annoyingly, no errors get thrown, as far as i can see, i just end at a blank page without being returned to the callback page.

I'm rather stumped on this one, where am i likely to be going wrong/how do i go about correcting it.

Cheers
Joe