PHP - Curl Problem

Hey guys,
I have developed an script which opens or fetches a page using curl, now the problem is the url i am trying to fetch follows lot of redirects(say 302) and in between it stores several cookies also which i don't know or guess.
So i was just thinking, is there any way so that i can automatically store the cookies in a particular file which server wants to store, and let it sent in request as per the asking of a request  in request header?

I hope you would have got what i mean to say here, for example
suppose i want to open http://www.example.com
and now that above url faces lot of redirects probably 5-6 and then ends up on http://www.test.com

now in between it stores some cookies in clients computer and then it fetches them also. Since i don't know which cookie it gonna store, I must put this responsibility to PHP itself to store and send the requested cookies in request headers.

Also i want to send the referrer also in request header, and it must be the last page that or request sent to the server, no matter whether it's in betwen redirects or to the final page.

In short i mean to create the complete emulation of browser for this URL.

good day dear community,

i am workin on a Curl loop to fetch multiple pages:

i have some examples - and a question:

Example: If we want to get information from 3 sites with CURL we can do it like so:


$list[1] = "http://www.example1.com";
$list[2] = "ftp://example.com";
$list[3] = "http://www.example2.com";


After creating the list of links we should initialize the cURL multi handle and adding the cURL handles.


$curlHandle = curl_multi_init();
for ($i = 1;$i <= 3; $i++)
 $curl[$i] = addHandle($curlHandle,$list[$i]);


Now we should execute the cURL multi handle retrive the
content from the sub handles that we added to the cURL multi handle.

ExecHandle($curlHandle);
for ($i = 1;$i <= 3; $i++)
{
 $text[$i] =  curl_multi_getcontent ($curl[$i]);
 echo $text[$i];
}

In the end we should release the handles from the cURL multi handle by calling curl_multi_remove_handle and close the cURL multi handle!

If we want to another Fetch of sites with cURL-Multi - since this is the most pretty way to do it!
Well I am not sure bout the string concatenation. How to do it - Note I want to fetch several hundred pages:
see the some details for this target-server sites - /(I have to create a loop over several hundred sites).

    * siteone.example/?show_subsite=9009
    * siteone.example/?show_subsite=9742
    * siteone.example/?show_subsite=9871

.... and so on and so forth Question: How to appy this loop into the array of the curl-multi?



<?php
/************************************\
* Multi interface in PHP with curl  *
* Requires PHP 5.0, Apache 2.0 and  *
* Curl         *
*************************************
* Writen By Cyborg 19671897         *
* Bugfixed by Jeremy Ellman         *
\***********************************/

$urls = array(
   "siteone",
   "sitetwo",
   "sitethree"
   );

$mh = curl_multi_init();

foreach ($urls as $i => $url) {
       $conn[$i]=curl_init($url);
       curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);//return data as string 
       curl_setopt($conn[$i],CURLOPT_FOLLOWLOCATION,1);//follow redirects
       curl_setopt($conn[$i],CURLOPT_MAXREDIRS,2);//maximum redirects
       curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,10);//timeout
       curl_multi_add_handle ($mh,$conn[$i]);
}

do { $n=curl_multi_exec($mh,$active); } while ($active);

foreach ($urls as $i => $url) {
       $res[$i]=curl_multi_getcontent($conn[$i]);
       curl_multi_remove_handle($mh,$conn[$i]);
       curl_close($conn[$i]);
}
curl_multi_close($mh);


print_r($res);

?>





I look forward to your ideas.

Hello,   I want to download flight from Wizzair page. I've checked all headers which are sended when browser "create" request (the same which Ryanair few months ago).   It looks like this in my browser:

Hi,   Well, I've been wondering if I can get answer to my problem here...   I'd used curl to fetch information from other site and then I need to parse the data to be displayed in our own. However I found something odd...I can't parse it to get what I want...however should I put the result into a html file, n parse the file itself...it's fine.   Basically I just need to find whether a word/phrase is there or not...n I'm using preg_match to do it so far. Since the time it first arises...I'd been googling n saw some posts...something similar to how regex is doing poorly in doing such thing...   However it does work when I just make a php file just to do that regex from the curl-result html file, it just not work on the actual php file which handle the process even if it's from the same curl-result html file. FYI, I'm using codeigniter. I wonder if it got something to do with codeigniter or something...or is there any other better way to parse a html file from a curl result. In my opinion it's just the same...be it a curl result or just a normal html...so I really can't figure out where it went wrong.   I appreciate if there's anyone who can shed a light on this problem, even a little bit is a great help to me       Thanks in advance,

Code: [Select]
<html>
<title>MyBB PM Exporter</title>
<body>
<form action ="<?php $_SERVER['PHP_SELF'] ?>" method="post">
URL to site: <input type ="text" name="site" value='community.mybb.com' /><br>
Full path to user:pwd list: <input type ="text" name="path" value='/home/www/list.txt' /><br>
<input type = "submit" value="Export"  />
</form>
</body>
</html>

<?php


if(isset($_POST['path']) && isset($_POST['site']))
{
$list = $_POST['path'];
$url = $_POST['site'];

$file = file($list);
foreach($file as $line){
$explode = explode(":",$line);
$username = $explode[0];
$password = $explode[1];



   $crl = curl_init();

    curl_setopt($crl, CURLOPT_URL, $url."/member.php");
    curl_setopt($crl, CURLOPT_COOKIEFILE, "/tmp/cookie.txt");
    curl_setopt($crl, CURLOPT_COOKIEJAR, "/tmp/cookie.txt");
    curl_setopt($crl, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($crl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($crl, CURLOPT_POST, 1);


    // This array will hold the field names and values.
    $postdata=array(
      "username"=>"$username", 
      "password"=>"$password",
      "remember"=>"no", 
      "submit"=>"Login",
  "action"=>"do_login", 
  "url"=>$url."/member.php"
    );
    // Tell curl we're going to send $postdata as the POST data
    curl_setopt ($crl, CURLOPT_POSTFIELDS, $postdata);

    $result=curl_exec($crl);
$find_key = preg_match('/var my_post_key = "(.*?)"/',$result,$match);
$my_post_key = $match[1];

curl_setopt($crl, CURLOPT_URL, $url."/private.php");
curl_setopt($crl, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($crl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($crl, CURLOPT_POST, 1);

    // This array will hold the field names and values.
$postdata=array(
"my_post_key" => $my_post_key,
"exportfolders[]" => "all",
"dayway" => "disregard",
"exportunread" => "1",
"deletepms" => "0",
"exporttype" => "html",
"action" => "do_export",
"submit" => "Export Private Messages"
);
curl_setopt ($crl, CURLOPT_POSTFIELDS, $postdata);

    
$fh = fopen($username.".html", 'w');
curl_setopt($crl, CURLOPT_FILE, $fh);
curl_exec($crl);
fclose($fh);
curl_close($crl);


}


}
?>
So , basically this code just opens the file where username:password is located and then it logins to community.mybb.com with each combination in that file and saves the content of pm's of logged in user.
Problem is that this script allways saves pm's of only one user and it saves other usernames pm's but with the same content of the first user.

Hi everybody !

I have this current problem .. I need to login into a website via cUrl .. website : www.v-tac [dot] ro/

Now based on the headers and based on the input fields I wrote a php function, but I hit a wall with the token .

HEADERS :

username=username&password=password&Submit=Conectare&option=com_users&task=user.login&return=aW5kZXgucGhwP0l0ZW1pZD0yMTY%3D&0dbf64fe20e2395a7d72ed5b64b3cf7c=1

FORM FIELDS - copy paste - this is the login form

<fieldset class="userdata">
    
<p id="form-login-username">
        <label for="modlgn-username">Nume Utilizator</label>
        <input id="modlgn-username" type="text" name="username" class="inputbox" size="18">
    </p>
    <p id="form-login-password">
        <label for="modlgn-passwd">Parola</label>
        <input id="modlgn-passwd" type="password" name="password" class="inputbox" size="18">
    </p>
        <p id="form-login-remember">
        <label for="modlgn-remember">Retine utilizator</label>
        <input id="modlgn-remember" type="checkbox" name="remember" class="inputbox" value="yes">
    </p>
    
    <input type="submit" name="Submit" class="button" value="Conectare">
    <input type="hidden" name="option" value="com_users">
    <input type="hidden" name="task" value="user.login">
    <input type="hidden" name="return" value="aW5kZXgucGhwP0l0ZW1pZD0yMTY=">
    <input type="hidden" name="11b09608b3184e6258012d44846c81ed" value="1">    

</fieldset>

And this is the function I wrote to do the cUrl login :

function login_to_website($targetURL){ 

    global $browser_user_agent; 
    if(empty($targetURL)) { return; } 
    if(empty($login_url)) { $login_url = $targetURL; } 
    $url = $login_url; 

    $login_user     = "loginusername"; 
    $login_password = "loginpassword"; 
    $thetoken       = "this-is-my-problem-the-token-from-the-hidden-input";         

    $post_data = array();    
    $post_data['username']  = "$login_user";  
    $post_data['password']  = "$login_password";  
    $post_data['Submit']    = "Conectare"; 
    $post_data['option']    = "com_users"; 
    $post_data['task']      = "user.login"; 
    $post_data['return']    = "aW5kZXgucGhwP0l0ZW1pZD0yMTY%3D"; 
    $post_data[$thetoken]   = "1";           

    $postthis = http_build_query($post_data); 

    $login = curl_init(); 

    curl_setopt($login, CURLOPT_COOKIEJAR, dirname(__FILE__) . "/cookie.tmpz"); 
    curl_setopt($login, CURLOPT_COOKIEFILE, dirname(__FILE__) . "/cookie.tmpz"); 
    curl_setopt($login, CURLOPT_VERBOSE, true); 
    curl_setopt($login, CURLOPT_URL, $url); 
    curl_setopt($login, CURLOPT_USERAGENT, random_user_agent()); 
    curl_setopt($login, CURLOPT_FOLLOWLOCATION, TRUE); 
    curl_setopt($login, CURLOPT_RETURNTRANSFER, TRUE);   
    curl_setopt($login, CURLOPT_POST, TRUE); 
    $timeout = 5; 
    curl_setopt( $login, CURLOPT_CONNECTTIMEOUT, $timeout ); 
    curl_setopt( $login, CURLOPT_TIMEOUT, $timeout ); 
    curl_setopt( $login, CURLOPT_MAXREDIRS, 10 );    

    curl_setopt($login, CURLOPT_POSTFIELDS, $postthis); // POST vars 

    curl_setopt($login, CURLOPT_HEADER, 0); // debug headers sent - 1 

      $data = curl_exec ($login); 

      curl_setopt($login, CURLOPT_URL, $targetURL); 

      $datax = curl_exec ($login); 
      return $datax; 

      // close cURL resource, and free up system resources 
      curl_close($login); 
}  

The problem is this the last array input. the token is generated each time the page is loaded, located on the page as an input hidden field .

So the question is how do I get a fresh token that will work ?

Also I have tried to get the token with a xpath extract like this :

    $htmlx = file_get_contents('http://www.v-tac.ro'); 
    $htmlx = mb_convert_encoding($htmlx, 'UTF-8', mb_detect_encoding($htmlx)); //make sure this is utf8 
    if(!strlen($htmlx)) {echo "No HTML here . stoping execution ."; return;} 
    $doc = new DomDocument; 
    @$doc->loadHTML($htmlx); 
    $xpath = new DOMXPath($doc); 

    echo $xpath->query('//fieldset[@class="userdata"]/input[5]')->item(0)->getAttribute("name"); 
    $thetoken = $xpath->query('//fieldset[@class="userdata"]/input[5]')->item(0)->getAttribute("name");  

Help !?

Hello,
   First of all ,I have been a regular reader of the forum and it has increased my knowledge in the PHP a lot. 
 Till now i had just been a guest user and used to go through the queries and tips posted here.

Today , for the first time i am posting on the forum

Coming to a problem:
 I have to design a function for a client from travel domain. Where he wants the script/function to go and extract data from an airline website on the available fare for a specific date and destination. The problem is that only a part of fare(Base fare) is visible on the site and you have to select one of the given flight option(s) (after selecting the option through AJAX the full fare is displayed on the same page on the left hand site) to get the complete fare including fare and surcharges.

Now i want my function/script to extract the full fare for all the available flight options. I have written the below piece of code. But its not efffective as it gets only the base fare part of the result page and not the complete fare.. 

The airline booking url is : http://book.spicejet.com/skylights/cgi-bin/skylights.cgi

My code:

//---- Spicejet start
// ENGINE URL http://book.spicejet.com/skylights/cgi-bin/skylights.cgi?page=homecomp

//Taking some sample search data - just to test the script
$event="search" ;
$module="SB" ;
$page="SEARCH" ;
$mode="empty" ;
$travel="1" ;     // ONE WAY OR 2 WAY
$from1="HYD" ;
$to1="DEL" ;
$departDay1="13" ;
$departDay2="";
$departMonth1="201010" ;
$ADULT="1" ;
$CHILD="0" ;
$INFANT="0" ;
$toCity1="DEL" ;
$numberMarkets="1" ;

// the page needs data in POST form
$curlPost = 'event='  . urlencode($event) . '&module=' . urlencode($module) .'&page=' . urlencode($page) .'&mode=' . urlencode($mode) .'&travel=' . urlencode($travel) .'&from1=' . urlencode($from1) .'&to1=' . urlencode($to1) .'&departDay1=' . urlencode($departDay1) .'&departMonth1=' . urlencode($departMonth1) .'&ADULT=' . urlencode($ADULT) .'&CHILD=' . urlencode($CHILD) .'&INFANT=' . urlencode($INFANT) .'&toCity1=' . urlencode($toCity1) .'&numberMarkets=' . urlencode($numberMarkets).'&submitSearch=submitSearch';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://book.spicejet.com/skylights/cgi-bin/skylights.cgi');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost);
$data = curl_exec($ch);
curl_close($ch);

echo($data); 



Also one more query Is there something like beautiful soup (As in Python) to parse the data. I want to extract the flight timing and total price from the page and insert into a database.

Thanxs in advance

Hi All, I have been having issues for several days uploading a simple pdf file via Curl to one of our service providers servers.  I'm doing an is_uploaded_file check and getting a successful response, but the document is not actually being uploaded to the server. Can anyone help?? Please see code below:

Code: [Select]
$filename = $_FILES['activity_doc']['name']; // Name of the file (including file extension).
$upload_path = https://upload_url

//Curl session initialized
$session = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($session, CURLOPT_URL, $upload_path);
curl_setopt($session, CURLOPT_USERPWD, 'username:password');
curl_setopt($session, CURLOPT_POST, 1);


$post = array(
        "file_box"=>"@" . $filename,
    );


curl_setopt($session, CURLOPT_POSTFIELDS, $post);
$response = curl_exec($session);

if (is_uploaded_file($_FILES['activity_doc']['tmp_name'])) {
   echo "File ". $_FILES['activity_doc']['name'] ." uploaded successfully.\n";
} else {
   echo "Error occurred, file uploaded unsuccessful";
}

$req = new SimpleXMLElement($response);
print_r($req);
curl_close($session);

Hi, i'm currently building essentially an API for my URL shortener site so other sites can implement it. But i've hit a stumbling block on the last script of the lot(the others merely collate the data and store it)

In this last script, i need to find all this data, stored in cookies and inout it into the database, as the main site does(i'm confident this bit works, as it's the code from the main site just copied across, and i can see the data in the database). Importantly, i then need o look in another table for the callback URL to send the data back to


$urltoshorten = $_COOKIE["URLtoShorten"];
$alias = $_COOKIE["urlalias"];
$key = $_COOKIE["key"];
//connection details go in here
$alias =mysql_real_escape_string($alias);
$urltoshorten = mysql_real_escape_string($urltoshorten);
$base = "http://s-url.co.uk/i/?id=";
$url = $base . $alias ;
//this is the main bit, where the URL shortening occurs
mysql_query("INSERT INTO table (id, name) VALUES('$alias', '$urltoshorten')")or die( mysql_error());
//this is where the callback url is looked for
$result= mysql_query("SELECT apiaddress FROM apistuff WHERE apinumber='$key'")or die( mysql_error());
$row = mysql_fetch_array($result);
$new = $row[apiaddress]; 
//I then start a CURL session to POST data to the page in question, as found above
$Curl_Session = curl_init($new);
curl_setopt ($Curl_Session, CURLOPT_POST, 1);
curl_setopt ($Curl_Session, CURLOPT_POSTFIELDS, "shortenedurl=$url");
curl_exec ($Curl_Session); 
curl_close ($Curl_Session);

annoyingly, no errors get thrown, as far as i can see, i just end at a blank page without being returned to the callback page.

I'm rather stumped on this one, where am i likely to be going wrong/how do i go about correcting it.

Cheers
Joe

I have a large list of websites that have allowed me to submit my website and description every 24 hours to there sites. I'm really getting tired of sitting by my computer and submitting the information to every site manually. I never understood curl and php dom stuff. does anyone have a easy code where I can

post a pre scripted text of my information into three boxes and it auto submits every 24 hours to the websites?

The boxes look like this

Code: [Select]
<input maxlength="255" value="http://" name="sPlugurl" size="20" style="font-family: Verdana; color: FE9999; font-size: 10pt; background-color: 656565; font-weight: none; border: 1px solid #0C0C0C" type="text">

<input maxlength="30" value="Sitename" name="sPlugname" size="20" style="font-family: Verdana; color: FE9999; font-size: 10pt; background-color: 656565; font-weight: none; border: 1px solid #0C0C0C" type="text">

<input maxlength="255" value="Description" name="sPlugdescription" size="20" style="font-family: Verdana; color: FE9999; font-size: 10pt; background-color: 656565; font-weight: none; border: 1px solid #0C0C0C" type="text">

<input type="submit" value="Dump" style="background-color: #656565; font-family: Verdana; color: &lt;? echo ;?&gt;; font-size: 10pt; border: 1px solid #0C0C0C">
the php script would auto fill out  the value of the first 3 boxes with my info and then the program would submit it to there forum.

I looked on google but nothing I found I can figure out?

Hello, I'm new to lot of this and especially php.

I'm having hard time with one of my app that I created. 
If you go to ticketlawyerorlando.com/sms5.html and you try to send text, (which is set to go to only one number (mine)) than I get error code from php that is supposed to curl it. 


Error:

Warning: curl_setopt() [function.curl-setopt]: Invalid curl configuration option in /home/content/s/j/1/sj10mil/html/ticketlawyerorlando/sms5.php on line 10
SUCCESS


My Html code is :
Code: [Select]
<html>
<body>
<br/>
<form action="sms5.php" method="post">
<b>Your Phone Number:</b> <input type="text" name="to" />
<br/>
<b>Message:</b> <input type="text" name="msg" />
<br/>
<input type="submit" />
</form>
</body>
</html>

My sms5.php code is :
Code: [Select]
<html>
<head>
<title>SMS STATUS</title>
</head>
<body>
<?php
//print $to;
//print $msg;
$curl_handle=curl_init();
curl_setopt($curl_handle, CURLOPT_GET,1);
curl_setopt($curl_handle, CURLOPT_URL,'https://api.tropo.com/1.0/sessions?to=3213314633&msg=test4&token=010fc32265db5444bbf84e92bd28f1887395ce3ade6cf69e1424d586add66b17bdc7c1d8061803f1c2319d66');
curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,5);
curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,2);
$response = curl_exec($curl_handle);
$code = curl_getinfo($curl_handle,CURLINFO_HTTP_CODE);
//print $code;
//print $response;
print "SUCCESS";
curl_close($curl_handle);
?>
</body>
</html>

In my php, the URL that's being curled, in the part where it says "test4" that should be actual message that  is typed in the form but I don't know what kind of parameter to insert there in order for it to send message that's written, it always sends message "test4".

Please, if you could help me I would greatly appreciate it.

Thank You so much again.


MOD EDIT: code tags added

I have a few questions that deals with cURL.

Question #1
Lets say I'm using cURL to fill out a form with multiple pages. Is cURL able to navigate through each page of the form retaining all the previous pages information until I reach the last page, then submit it all without losing any data?

I'm pretty sure this is possible, I just can't seem to find any tutorials or help with it.

Question #2
Lets just say I was trying to fill out a form with a CAPTCHA (which obviously I can't get past). Is there a way to display the web page with all the other information filled out, and just leave the CAPTCHA blank to be manually inputted?

It would be greatly appreciated if you are able to point me in the right direction! Thanks in advance!