PHP - Getting Data From Curl Imported Xml Page?
I'm getting data from a remote XML file using the following as a result of a form:
Code: [Select] <?php header('Content-type: text/xml'); //extract data from the post extract($_POST); //set POST variables $url = 'https://www.*******'; $fields = array( 'ESERIES_FORM_ID'=>urlencode($ESERIES_FORM_ID), 'MXIN_USERNAME'=>urlencode($MXIN_USERNAME), 'MXIN_PASSWORD'=>urlencode($MXIN_PASSWORD), 'MXIN_VRM'=>urlencode($MXIN_VRM), 'MXIN_TRANSACTIONTYPE'=>urlencode($MXIN_TRANSACTIONTYPE), 'MXIN_PAYMENTCOLLECTIONTYPE'=>urlencode($MXIN_PAYMENTCOLLECTIONTYPE), 'MXIN_CAPCODE'=>urlencode($MXIN_CAPCODE) ); //url-ify the data for the POST foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; } rtrim($fields_string,'&'); //open connection $ch = curl_init(); //set the url, number of POST vars, POST data curl_setopt($ch,CURLOPT_URL,$url); curl_setopt($ch,CURLOPT_POST,count($fields)); curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string); //execute post $result = curl_exec($ch); //close connection curl_close($ch); ?> Which is displaying what was the remote XML page, but has brought it to my domain. How would I go about taking the results of the displaying XML page and displaying them on the next page rather than just displaying the XML data? Many thanks in advanced! Similar TutorialsI have a list of names in a table and I am trying to search a website with each of these names. Can someone point me in the right direction if I want to POST each name to the site and use cURL to get the information(did this part already). Will a simple Loop or While command do the job? I am trying to go to http://lirr42.mta.info/ and get the data from the submitted form. I have rebuilt the form exactly as needed, and done a few adjustments. The posts that I am sending from MY form, are the same as the one on the site. This site allows public data mining by the way. So does someone seem something wrong with my code? I know the basics are working. I have another page that is working fine. It's just this one ends up returning something..it returns part of the data but it's something wrong with it. Any advice is appreciated. Code: [Select] <?php define( 'DEBUG', true ); define( 'DEFAULT_STOP', 'Broadway' ); // report all errors during development/debug mode if ( DEBUG ) error_reporting( E_ALL ); else error_reporting( 0 ); include( 'simplehtmldom/simple_html_dom.php' ); $mta_post_url = 'http://lirr42.mta.info/schedules.php'; // GTFS Specification File Definitions $gtfs_files = array ( 'agency' => 'agency.txt', // required 'stops' => 'stops.txt', // required 'routes' => 'routes.txt', // required 'trips' => 'trips.txt', // required 'stop_times' => 'stop_times.txt', // required 'calendar' => 'calendar.txt', // required 'calendar_dates' => 'calendar_dates.txt', 'fare_rules' => 'fare_rules.txt', 'fare_attributes' => 'fare_attributes.txt', 'shapes' => 'shapes.txt', 'frequencies' => 'frequencies.txt', 'transfers' => 'transfers.txt' ); $gtfs_pickup_codes = array ( 0 => 'Regularly scheduled pickup', 1 => 'No pickup available', 2 => 'Must phone agency to arrange pickup', 3 => 'Must coordinate with driver to arrange pickup' ); $gtfs_dropoff_codes = array ( 0 => 'Regularly scheduled drop off', 1 => 'No drop off available', 2 => 'Must phone agency to arrange drop off', 3 => 'Must coordinate with driver to arrange drop off' ); // load stops file $stopsFile = fopen( 'data/lir/' . $gtfs_files[ 'stops' ], "r" ); if ( $stopsFile ) { // get (and toss) header row $stopsHeader = fgetcsv( $stopsFile, 1000 ); // print_r( $stopsHeader ); // will hold HTML output for Select dropdown for stops $stopSelectOptions = ''; // build array from file data while ( $data = fgetcsv( $stopsFile, 1000 ) ) { $stopsData[ $data[1] ] = $data[0]; } // get a sorted array of the stop names ( yes this could be done with array_multisort or usort but I didnj't feel like it right now ) $stopNames = array_keys( $stopsData ); sort( $stopNames ); // loop through sorted array and create SELECT options with default SELECTED at Grand Central Terminal foreach( $stopNames as $stop ) $stopSelectOptions .= sprintf( '<option %s value="%d">%s</option>', ( DEFAULT_STOP == $stop ) ? 'selected="selected"' : '' , $stopsData[ $stop ], $stop ) . "\n\r"; } fclose( $stopsFile ); // load ads file $adsFile = fopen( 'ads.txt', "r" ); if ( $adsFile ) { // get (and toss) header row $adsHeader = fgetcsv( $adsFile, 1000 ); // will hold ads available for stops $ads = array(); // build array from file data while ( $data = fgetcsv( $adsFile, 1000 ) ) { $ads[ $data[0] ] = array ( 'filename' => $data[1], 'url' => $data[2], 'text' => $data[3] ); } fclose( $adsFile ); } // printout the html header require_once( 'header.php' ); if ( 'POST' == $_SERVER[ 'REQUEST_METHOD' ] ) { $from_stop = $_POST[ 'FromStation' ]; // @todo Filter! http://www.php.net/manual/en/filter.filters.validate.php $orig_station = $from_stop; $to_stop = $_POST[ 'ToStation' ]; // @todo Filter! http://www.php.net/manual/en/filter.filters.validate.php $dest_station = $to_stop; if ( $to_stop == $from_stop ) { $error = 'Originating and Destination stops are the same.'; } else { print_ad( $orig_station, $dest_station, $location='top' ); $travel_date = $_POST[ 'RequestDate' ]; $requestTime = $_POST[ 'RequestTime' ]; $am_pm = $_POST[ 'RequestAMPM' ]; $filter = $_POST[ 'sortBy' ]; /* * Lets post this to MTA.info */ $mta_curl = curl_init(); $mta_curl_options = array ( CURLOPT_FAILONERROR => true, CURLOPT_FOLLOWLOCATION => false, // CURLOPT_MUTE => true, CURLOPT_POST => true, CURLOPT_RETURNTRANSFER => true, CURLOPT_CONNECTTIMEOUT => 30, CURLOPT_TIMEOUT => 30, CURLOPT_URL => $mta_post_url, CURLOPT_REFERER => 'http://lirr42.mta.info/', CURLOPT_USERAGENT => 'MTA Info Scrape/1.0', CURLOPT_POSTFIELDS => http_build_query( $_POST ) ); curl_setopt_array( $mta_curl, $mta_curl_options ); $mta_response = curl_exec( $mta_curl ); if ( false === $mta_response ) { curl_close( $mta_curl ); die( sprintf( 'Response Code:%s, Curl Error No:%s, Curl Error Message:%s', curl_getinfo( $mta_curl, CURLINFO_HTTP_CODE ),curl_errno( $mta_curl ), curl_error( $mta_curl ) ) ); } else { curl_close( $mta_curl ); echo '<pre>' . htmlentities( $mta_response ) . '</pre>'; $html = new simple_html_dom(); $html->load( $mta_response ); $schedule_table = $html->find( 'table', 0 ); // find second table in the response $row_count = 0; // will hold HTML output for table $stopSchedule = '<table><th>Departs</th><th>Arrives</th><th>Minutes</th><th>Transfer</th><th>Fare</th></tr>'; foreach ( $schedule_table->find('tr') as $schedule_row ) { $row_count++; // check for and skip header row if ( 1 == $row_count ) continue; // check for and skip last row which first TD has rowspan attribute if ( $schedule_row->children(0)->colspan ) { continue; } // extract table data for this row $depart_time = $schedule_row->children( 0 )->innertext; $arrive_time = $schedule_row->children( 2 )->innertext; $minutes_traveled = $schedule_row->children( 4 )->innertext; $transfer = $schedule_row->children( 5 )->innertext; $fare = $schedule_row->children( 6 )->plaintext; // add table row to html output $stopSchedule .= sprintf( '<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>', $depart_time, $arrive_time, $minutes_traveled, $transfer, $fare ) . "\n\r"; } // end foreach schedule_row // add table end tag to html output $stopSchedule .= '</table>'; // output table to browser echo $stopSchedule; print_ad( $orig_station, $dest_station, $location='bottom' ); } // end if mta response } // end if not same orig and dest } // end if POST if ( 'GET' == $_SERVER[ 'REQUEST_METHOD' ] || ! empty( $error ) ) { print_ad( null, null, $location = 'default' ); ?> <form method="post"> <?php if ( ! empty ( $error ) ) { echo sprintf( '<div class="error">%s</div>', $error ); } ?> From:<br /> <select id="orig_station" name="FromStation" tabindex="2"> <?php echo $stopSelectOptions; ?> </select> <br /> <br /> To:<br /> <select id="dest_station" name="ToStation" tabindex="3"> <?php echo $stopSelectOptions; ?> </select> <br /> <br /> <input id="date1" name="RequestDate" size="12" maxlength="10" tabindex="4" value="<?php echo date( 'm/d/Y' ); ?>" /> <?php $currentHour = trim( date( 'h' ) ); $currentMinutes = trim( date( 'i' ) ); $currentMinutesFloor = sprintf( '%02d' , $currentMinutes - ( $currentMinutes % 30 ) ); $currentAMPM = trim ( date( 'A' ) ); $start_time = strtotime( '1:00 AM' ); $end_time = strtotime( '1:00 PM' ); ?> <select name="RequestTime" tabindex="5"> <?php while( $start_time < $end_time ) { $option_time = date( 'h:i', $start_time ); $option_hour = trim( date( 'h', $start_time ) ); $option_minutes = trim( date( 'i', $start_time ) ); $option_selected = ( $option_hour == $currentHour && $option_minutes == $currentMinutesFloor ); echo sprintf( '<option %s>%s</option>', ( $option_selected ) ? 'selected="selected"' : '', $option_time ) . "\r\n"; $start_time = $start_time + 30*60; } ?> </select> <select name="RequestAMPM" tabindex="6"> <option value="AM" <?php echo ( 'AM' == $currentAMPM ) ? 'selected="SELECTED"' : ''; ?>>AM</option> <option value="PM" <?php echo ( 'PM' == $currentAMPM ) ? 'selected="SELECTED"' : ''; ?>>PM</option> </select> <input name="sortBy" type="hidden" value="1" /> <input type="submit" name="submit" value="Check Schedule" /> </form> <!-- images/base/ ic_menu_back.png ic_menu_home.png --> <?php } require_once( 'footer.php' ); ?> Hi there, OK, there is what I'm trying to accomplish - I want to send POST instantly when the page is loaded without "Submit" button or anything like that. Here how it looks using JavaScript: Code: [Select] <form name="postdata" action="http://www.mywebsite.com/post.php" method="post"> <input name="data" type="text" value="1"/> </form> <script type="text/javascript" language="JavaScript"> document.postdata.submit(); </script> Works fine with JavaScript, however i can't use it. Here is reference for CURL: http://php.net/manual/en/book.curl.php I found this curl_post function at php.net to simplify things up. <?php /** * Send a POST requst using cURL * @param string $url to request * @param array $post values to send * @param array $options for cURL * @return string */ function curl_post($url, array $post = NULL, array $options = array()) { $defaults = array( CURLOPT_POST => 1, CURLOPT_HEADER => 0, CURLOPT_URL => $url, CURLOPT_FRESH_CONNECT => 1, CURLOPT_RETURNTRANSFER => 1, CURLOPT_FORBID_REUSE => 1, CURLOPT_TIMEOUT => 4, CURLOPT_POSTFIELDS => http_build_query($post) ); $ch = curl_init(); curl_setopt_array($ch, ($options + $defaults)); if( ! $result = curl_exec($ch)) { trigger_error(curl_error($ch)); } curl_close($ch); return $result; } $test = curl_post("http://www.mywebsite.com/post.php", array("data" => "1")); print($test); ?> Looks like it uses GET instead POST (used sniffer to verify this). I'm trying to send POST data to other host, if that matters. Any help would be appreciated. Thanks in advance. First of all... this is a GREAT forum. Been lurking around here for a long time and searched all over but still can't find a solution to a problem I'm facing. Here's my cURL code - I am basically retrieving data from a web site: <?php $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,'http://myprimemobile.com/update.html'); curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2); curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1); $buffer = curl_exec($curl_handle); curl_close($curl_handle); if (empty($buffer)) { print "Connection Timeout<p>"; } else { print $buffer; } ?> If you take a look at the site http://myprimemobile.com/update.html -- it will begin to update new information. I want to basically retrieve the data from that page but in my own way. I want to loop through the page and search for all the names, addresses, and phone numbers only (not store hours or anything else), and want to put it in a variable so I can create a table with all that information on my page. Based on the code, all the information is stored in the $buffer variable. I want to break this variable down to extract certain data in a 'div' or a 'table'. Please help me out. Thanks! OK, I have the initial cURL working but need to figure out how to extract data I want off that webpage to display or store in a database, I tried using dom and xpath, but because of the way the page displays using css, i think its not picking it up. Here is my cURL script: <?php $userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)'; $target_url = "www.test.com"; $ch = curl_init(); curl_setopt($ch, CURLOPT_USERAGENT, $userAgent); curl_setopt($ch, CURLOPT_URL,$target_url); curl_setopt($ch, CURLOPT_FAILONERROR, true); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER,true); curl_setopt($ch, CURLOPT_TIMEOUT, 10); $html = curl_exec($ch); if (!$html) { echo "<br />cURL error number:" .curl_errno($ch); echo "<br />cURL error:" . curl_error($ch); exit; } // parse the html into a DOMDocument $dom = new DOMDocument(); $dom->loadHTML($html); // grab all the on the page $xpath = new DOMXPath($dom); $hrefs = $xpath->evaluate("/html/body//td"); for ($i = 0; $i < $hrefs->length; $i++) { $href = $hrefs->item($i); $url = $href->getAttribute('href'); storeLink($url,$target_url); echo "<br />Link stored: $url"; } ?> and here is a snippet of the source of the page I am getting: <span id="lblTest"><h1 id='surrZipTitle'>Agents in Surrounding Zip Codes</h1><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span></td><td valign='top'> </td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>4444 MANZANITA AVE STE 6</span><br /><span>CARMICHAEL , CA 95608-1488</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415199904150295&lat=38.646142&lng=-121.327623' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact & Directions</a> <a class='faaBlueLink' id='lnkWebSite' style='display: none;' href='http://' target='_blank' onclick="return trackEvent('/External-Link/AgentWebsite/ ','PROFESSIONAL INS ASSOC, INC. ');">Website</a></div></td><td valign='top'> </td></tr></table></td></tr></table><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>AMERICAN AIM AUTO INS AGY, INC</span></td><td valign='top'> </td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>5339 SAN JUAN AVE</span><br /><span>FAIR OAKS , CA 95628-3318</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415911704151222&lat=38.66237&lng=-121.292429' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact & Directions</a> So basically I want to extract the agency name like "<span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span>" and the address which always use the same div class like "caaAgencyName2" and "addressBlock". How can this be accomplished? I have json code and i want to prine only {"currency":"DOGE","available":"0","reserved":"420.000000000"} data. my json data [{"currency":"1ST","available":"0","reserved":"0"},{"currency":"DOGE","available":"0","reserved":"420.000000000"},{"currency":"ZSC","available":"0","reserved":"0"}] I use this code
<?php ?> I'm using cURL to crawl and scrape data from a website. This website contains tables with rows of data. When I send a cURL POST for the underlying data at a specific row(A), it will return the expected data. But when I move to the second row(B), the data returns blank or specifically, a tons of spaces (or &#nbsp's.) When I access the cURL's POST location by browser, I can see (B)'s data. The only difference in the 2 POST's are location ID's for the data. I don't think it's a problem with JavaScript as I can successfully return data from row (A) as I mentioned. ok i have been trying for over a week now, and i just relearning php again after a few years of being lazy. lol here is a code snippet. Quote //prep the members $myFile2 = "members.txt"; $fh2 = fopen($myFile2, 'r'); $theData2 = fread($fh2, filesize($myFile2)); fclose($fh2); //echo $theData2; //grab the stats! $url = 'http://link'; $postdata = 'players='.$theData2.'&fields=all'; $ch = curl_init($url); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata); $data = curl_exec($ch); curl_close($ch); $data = json_decode($data,true); the orininal code Quote //prep the members $members = "member,member1,member two,member blue"; //grab the stats! $url = 'http://link'; $postdata = 'players='.$members.'&fields=all'; $ch = curl_init($url); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata); $data = curl_exec($ch); curl_close($ch); $data = json_decode($data,true); now the problium i am having is that with the original way, listing the members indevidualy is not the way i would like it, it would be easyer if it was in .txt form so i can add top it easyer. Also i have 300 people to add and the string doesnt seem to like that many people in it.....so i made a txt file, and it all works except that when i load the page it does not display the members stats from the external link. each players stats gets posted on the page and sorted who is ranked highest. echo works for echoing the contents of the file. however thats usless to me as i want the contents of the file to work like the orininal code. sorry if its confusing.... basicaly there is data from the link for each player asociated with there name. which is seperated by a , in the code. so each time a , is reached the code repeats its self posting each player till there are no more to post. the string is to short for me or just seems to not work with 300 people. i need it to read the external .txt file or something simular and work in the same mannor. I've looked at various cURL tutorials and the PHP manual, but I don't see what I'm doing wrong here. My goal is to be able to send data to test.php and then have that page run a MySQL query to insert the TITLE and MESSAGE into the database. Any comments? :/ post.php <?php if(isset($_GET['title']) && isset($_GET['message']) && isset($_GET['times'])) { $array = array('title' => urlencode($_GET['title']), 'message' => urlencode($_GET['message'])); foreach($array as $key => $value) { $fields = $key.'='. $value .'&'; } rtrims($felds); //set our init handle $curl_init = curl_init(); //set our URL curl_setopt($curl_init, CURLOPT_URL, 'some url'); //set the number of fields we're sending curl_setopt($curl_init ,CURLOPT_POST, count($array)); for($i = 0; $i < $_GET['times']; $i++) { //send the POST data curl_setopt($curl_init, CURLOPT_POSTFIELDS, $fields); curl_exec($curl_init); echo 'post #'.$i. ' sent<br/>'; } //complete echo '<b>COMPLETE</b>'; //close session curl_close($curl_init); } else { ?> <form action="post.php" method="GET"> <table> <tr><td>Title</td><td><input type="text" name="title" maxlength="30"></td></tr> <tr><td>Content</td><td><textarea name="message" rows="20" cols="45" maxlength="2000"></textarea></td></tr> <tr><td>Process Amount</td><td><input type="text" name="times" size="5" maxlength="3"></td></tr> <tr><td>START</td><td><input type="submit" value="Initiate Posting"></td></tr> </table> </form> <?php } ?> test.php <?php mysql_connect('host', 'user', 'pass'); mysql_select_db('somedb'); if($_POST['title'] && $_POST['message']) { mysql_insert("INSERT INTO tests VALUES (null, '{$_POST['title']}', '{$_POST['message']}')"); } echo '<hr><br/>'; //query $query = mysql_query("SELECT * FROM tests ORDER BY id DESC"); while($row = mysql_fetch_assoc($query)) { echo $row['id'].'TITLE: '. $row['title'] .'<br/>MESSAGE: '. $row['message'] .'<br/><br/>'; } ?> I'm trying to send a HTTPS POST request with XML data to a server using PHP. Anything sends to the server requires authentication therefore I'll use cURL. Some background info.: the XML data is to request the server to upload a file from a specific URL to its local storage. One rule of using this API is I MUST set the content type for each request to application/xml. This is what I've done but isn't working... <?php $fields = array( 'data'=>'<n1:asset xmlns:n1="http://.....com/"><title>AAAA</title><fileName>http://192.168.11.30:8080/xxx.html</fileName><description>AAAA_desc</description><fileType>HTML</fileType></n1:asset>' ); $fields_string = ""; foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; } rtrim($fields_string,'&'); $url = "https://192.168.11.41:8443/"; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_POST, count($fields)); curl_setopt($ch, CURLOPT_POSTFIELDS, $fields_string); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); curl_setopt($ch, CURLOPT_HEADER, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_USERPWD, "admin:12345678"); curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: application/xml', 'Content-length: '. strlen($fields_string)) ); $result = curl_exec( $ch ); curl_close($ch); echo $result; ?> I am expected to get an XML reply of either upload successful or upload failed. But instead I am getting this error message. HTTP/1.1 415 Unsupported Media Type Server: Apache-Coyote/1.1 Content-Type: text/xml;charset=UTF-8 Content-Length: 0 Date: Thu, 02 Dec 2010 03:02:33 GMT I'm sure the file type is correct, the XML format is correct. I've tried urlencode the fields but it didn't work. What else I might have done wrong? Friends, I want to check a site if there is any data avaliable for a given Domain Name.... For example Quote http://www.keywordspy.com/research/search.aspx?tab=domain-organic&market=uk&q=abc.co.uk In this URl the Domain is the "&q=" POST Parameter which is: abc.co.uk In the Output, Observe the Columns Keyword, Position, Volume etc.... So for any Domain we want to check if there is one or more keyword returned. Here is the Example of a Domain for which there is no Data: Quote http://www.keywordspy.com/research/search.aspx?tab=domain-organic&market=uk&q=notfoundforme.co.uk As you Observe there is no data in Keyword, Position or Volume Columns for this Domain (i.e. notfoundforme.co.uk) All i want is to Echo All the Keywords, their Position and Volumes for any domain for which data is avaliable. Code: Here is the Code i have which scrapes the Whole data with Curl, but i am not able to make getkwspydata function work to get the Keyword, Position or Volume data for the domain, i know some regex and preg_match needs to be done but its too technical for me, may someone help me with this? <?php function getkwspydata($host) { $request = "http://www.google.com/search?q=" . urlencode("site:" . $host) . "&hl=en"; $request = "http://www.keywordspy.com/research/search.aspx?tab=domain-organic&market=uk&q=". urldecode($host); $data = getPageData($request); // preg_match('/<div id=resultStats>(About )?([\d,]+) result/si', $data, $p); // $value = ($p[2]) ? $p[2] : "n/a"; // $string = "<a href=\"" . $request . "\">" . $value . "</a>"; //return $string; print_r($data); } function getDomainName($host) { $hostparts = explode('.', $host); // split host name to parts $num = count($hostparts); // get parts number if(preg_match('/^(ac|arpa|biz|co|com|edu|gov|info|int|me|mil|mobi|museum|name|net|org|pp|tv)$/i', $hostparts[$num-2])) { // for ccTLDs like .co.uk etc. $domain = $hostparts[$num-3] . '.' . $hostparts[$num-2] . '.' . $hostparts[$num-1]; } else { $domain = $hostparts[$num-2] . '.' . $hostparts[$num-1]; } return $domain; } function getPageData($url) { if(function_exists('curl_init')) { $ch = curl_init($url); // initialize curl with given url curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']); // add useragent curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // write the response to a variable if((ini_get('open_basedir') == '') && (ini_get('safe_mode') == 'Off')) { curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // follow redirects if any } curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5); // max. seconds to execute curl_setopt($ch, CURLOPT_FAILONERROR, 1); // stop when it encounters an error return @curl_exec($ch); } else { return @file_get_contents($url); } } $sitehost = ($_POST['sitehost']) ? $_POST['sitehost'] : $_SERVER['HTTP_HOST']; $sitedomain = getDomainName($sitehost); ?> <html> <head> <title>SEO Report for <?=$sitedomain;?></title> </head> <body> <form method="post" action="<?$_SERVER['PHP_SELF'];?>"> <p><b>Domain/Host Name:</b> <input type="text" name="sitehost" size='30' maxlength='50' value="<?=$sitehost;?>"> <input type="submit" value="Grab Details"></p> </form> <ul> <li>Google indexed pages: <?=getkwspydata($sitehost);?></li> </ul> </body> </html> Cheers Natasha T. I have looked up tutorials on Google. I am still confused. I built a php script that receives the data from my html form then filters out bad words. I am trying to get the script to resend the data to pm.cgi. The form was originally targeting pm.cgi but it needed filtered. This is why i am using php in the first place. Please help me pass these variables from the php script to the cgi script the same way that a html form would. Thank you. Here is my code, it prints correctly. I cannot get cURL to work. <?php $action = $_GET['action'] ; $login = $_REQUEST['login'] ; $ID = $_REQUEST['ID'] ; $text = $_REQUEST['Unused10'] ; $submit = $_REQUEST['submit'] ; function filterBadWords($str){ // words to filter $badwords=array( "badword1", "badword2"); // replace filtered words with random spaces $replacements=array( "****", "***", "****" ); for($i=0;$i < sizeof($badwords);$i++){ srand((double)microtime()*1000000); $rand_key = (rand()%sizeof($replacements)); $str=eregi_replace($badwords[$i], $replacements[$rand_key], $str); } return $str; } $text = filterBadWords($text); print_r($action); print_r($login); print_r($ID); print_r($text); print_r($submit); ?> I need to send the variables that i printed above to pm.cgi. Can you point me in the right direction? Thanks again! Hi guys Can someone help me about this: The php code can be revise username and password with CURL then check database and if username & password is correct return true else false. Thanks This topic has been moved to PHP Regex. http://www.phpfreaks.com/forums/index.php?topic=334273.0 I have news articles which I distribute to various sites. Each article contains various hyperlinks, which must resolve to a location on the site which pulled the article, i.e. Article: (hyperlink) Widgets are wonderful. Don Doe, president of (Widgets Incorporated) say they are cool. ------ Site A: To resolve to example.com/widget-companies/Widget-news/Widgets-Incorporated. Site B: To resolve to otherexample.com/Widgets-Incorporates/widget-news I'm paralyzed. The only thing I can think of is to bring the whole article in, and do a string replace. Some of the articles are 2k+ words. Even then, I have to come up with a way of pointing out the link text. I came up with $COMPANY:WIDGETS-INCORPORATED:Widget-Incorporated (UC WIDGETS to associate it with that company, and the mixed case Widget-Incorporated would be the hyperlink text. Explode it and use it. I am not even sure it will work, or if it works whether it is too inefficient. Should I try that, or something else? Just a quick one for anyone here that knows. I am setting up a small catalog site that runs off a MySQL database. On each product page there is a main image and six supporting thumbnails of different views of said product. I have six thumbnails (front, back, left, right, modelimage1, modelimage2) with then six main images to match. I want to be able to click on each of the thumbnails and have it swap the main image to match. I have given the main image an ID so maybe I could use a bit of javascript to swap the mainimage1 (<? echo "$mainimage1"; ?>) ....... to........ mainimage2 (<? echo "$mainimage2"; ?>) when i click on Any ideas of where to start? Its really just a simple thumbnail gallery i guess! I'm a newbie so as simple as possible would be great! Code: [Select] $id = $_GET['id']; // will be equal to '1' if the url ends in ?id=1 it would be '2' if it ended in ?id=2 etc. $query="SELECT * FROM ruxxshop WHERE id='$id'"; $result=mysql_query($query); $num=mysql_numrows($result); mysql_close(); $i=0; while ($i < $num) { $name=mysql_result($result,$i,"name"); $subtitle=mysql_result($result,$i,"subtitle"); $description=mysql_result($result,$i,"description"); $colour=mysql_result($result,$i,"colour"); $standardprice=mysql_result($result,$i,"standardprice"); $standardshoplink=mysql_result($result,$i,"standardshoplink"); $largeprice=mysql_result($result,$i,"largeprice"); $largeshoplink=mysql_result($result,$i,"largeshoplink"); //main images $mainimage1=mysql_result($result,$i,"mainimage1"); //front image $mainimage2=mysql_result($result,$i,"mainimage2"); //back image $mainimage3=mysql_result($result,$i,"mainimage3"); //left image $mainimage4=mysql_result($result,$i,"mainimage4"); //right image $mainimage5=mysql_result($result,$i,"mainimage5"); //model image 1 $mainimage6=mysql_result($result,$i,"mainimage6"); //model image 2 //thumbnail images $leftimage=mysql_result($result,$i,"leftimage"); $rightimage=mysql_result($result,$i,"rightimage"); $backimage=mysql_result($result,$i,"backimage"); $frontimage=mysql_result($result,$i,"frontimage"); $modelimage1=mysql_result($result,$i,"modelimage1"); $modelimage2=mysql_result($result,$i,"modelimage2"); ?> <style type="text/css"> <!-- .areasmenu { font-size: 13px; color: #CCC; } .basemenu { text-align: left; font-size: 9px; font-family: Verdana, Geneva, sans-serif; } .rightalignmenu { font-family: Verdana, Geneva, sans-serif; font-size: 10px; font-style: normal; color: #999; text-align: right; vertical-align: top; } .thumbtable { background-color: #E2E2E2; } .maintable { background-image: url(Images/Main%20Content/background.jpg); } .BagTitle { color: #666; font-size: 14px; font-family: Verdana, Geneva, sans-serif; font-weight: bold; vertical-align: top; } .leftaligntext { font-family: Verdana, Geneva, sans-serif; font-size: 11px; color: #666; vertical-align: top; } .rightaligntext { font-family: Verdana, Geneva, sans-serif; font-size: 11px; color: #666; text-align: right; vertical-align: top; } .thumbpics { font-size: 7px; } a:link { color: #999; text-decoration: none; } a:visited { text-decoration: none; color: #999; } a:hover { text-decoration: none; color: #999; } a:active { text-decoration: none; color: #999; } --> </style> <p> <? ++$i; }; ?> </p> <p> </p> <table width="800" border="0" align="center" cellpadding="0" cellspacing="0"> <tr> <th rowspan="2" align="left" valign="top" scope="col"><a href="grid.html"><img src="Images/Main Content/Ruxx Logo Black.jpg" alt="Home Page" width="100" height="28" border="0" /></a> </th> <th width="316" height="20" valign="top" class="rightalignmenu" scope="col"><span class="areasmenu"><a href="/grid1.html">WOMENS</a></span></th> <th width="74" height="20" valign="top" class="rightalignmenu" scope="col"><span class="areasmenu">MENS</span></th> <th width="131" height="20" valign="top" class="rightalignmenu" scope="col"><span class="areasmenu">ACCESSORIES</span></th> </tr> <tr> <td height="20" colspan="3" class="rightalignmenu"><a href="index.html">Home</a> - <a href="/about.html">About Ruxx</a>- The Ruxx Store - Bag Sizing & Details - <a href="/contact.html">Contact</a> - Register For Info</td> </tr> </table> <table width="800" border="0" align="center" cellpadding="0" cellspacing="0" class="maintable" valign="middle" > <tr class="greybackground"> <td width="230" height="440" valign="top"><table width="230" border="0" cellspacing="0" cellpadding="0"> <tr> <td width="30" height="60" nowrap="nowrap"> </td> <td height="60"> </td> <td width="20" height="60" nowrap="nowrap"> </td> </tr> <tr> <td width="30" height="15" nowrap="nowrap"> </td> <td height="15" class="BagTitle"><font face="Arial, Helvetica, sans-serif"><? echo "$name"; ?></font></td> <td width="20" height="15" nowrap="nowrap"> </td> </tr> <tr> <td width="30" height="15" nowrap="nowrap"> </td> <td height="15" class="leftaligntext"><font face="Arial, Helvetica, sans-serif"><? echo "$subtitle"; ?></font></td> <td width="20" height="15" nowrap="nowrap"> </td> </tr> <tr> <td width="30" height="50" nowrap="nowrap"> </td> <td height="50"> </td> <td width="20" height="50" nowrap="nowrap"> </td> </tr> <tr> <td width="30" height="100" nowrap="nowrap"> </td> <td height="100" class="leftaligntext"><font face="Arial, Helvetica, sans-serif"><? echo "$description"; ?></font></td> <td width="20" height="100" nowrap="nowrap"> </td> </tr> <tr> <td width="30" height="55" nowrap="nowrap"> </td> <td height="55"> </td> <td width="20" height="55" nowrap="nowrap"> </td> </tr> <tr> <td width="30" height="100" nowrap="nowrap"> </td> <td height="100" class="leftaligntext"><font face="Arial, Helvetica, sans-serif"><? echo "$colour"; ?></font></td> <td width="20" height="100" nowrap="nowrap"> </td> </tr> </table></td> <td valign="top"><img src="<? echo "$mainimage1"; ?>" width="340" height="440" name="MainImage" /></td> <td width="230" valign="top"><table width="230" border="0" cellspacing="0" cellpadding="0"> <tr> <td width="20" height="60"> </td> <td height="60"> </td> <td width="30" height="60"> </td> </tr> <tr> <td width="20" height="15"> </td> <td height="15" class="rightaligntext"><font face="Arial, Helvetica, sans-serif">STANDARD SIZE</font><font face="Arial, Helvetica, sans-serif">: <? echo "$standardprice"; ?></font></td> <td width="30" height="15"> </td> </tr> <tr> <td width="20" height="15"> </td> <td height="15" class="rightaligntext"><font face="Arial, Helvetica, sans-serif"><a href="<? echo "$standardshoplink"; ?>">Buy This Bag</a></font></td> <td width="30" height="15"> </td> </tr> <tr> <td width="20" height="40"> </td> <td height="40"> </td> <td width="30" height="40"> </td> </tr> <tr> <td width="20" height="15"> </td> <td height="15" class="rightaligntext"><font face="Arial, Helvetica, sans-serif">LARGE SIZE</font><font face="Arial, Helvetica, sans-serif">: <? echo "$largeprice"; ?></font></td> <td width="30" height="15"> </td> </tr> <tr> <td width="20" height="15"> </td> <td height="15" class="rightaligntext"><font face="Arial, Helvetica, sans-serif"><a href="<? echo "$largeshoplink"; ?>">Order This Bag</a></font></td> <td width="30" height="15"> </td> </tr> <tr> <td width="20" height="20"> </td> <td height="20"> </td> <td width="30" height="20"> </td> </tr> <tr> <td width="20" height="200"> </td> <td height="200"><table width="150" border="0" align="right" cellpadding="0" cellspacing="5" class="thumbtable"> <tr> <td height="60"><img src="<? echo "$frontimage"; ?>" width="65" height="65" /></td> <td height="60"><img src="<? echo "$backimage"; ?>" width="65" height="65" /></td> </tr> <tr> <td height="60"><img src="<? echo "$leftimage"; ?>" width="65" height="65" /></td> <td height="60"><img src="<? echo "$rightimage"; ?>" width="65" height="65" /></td> </tr> <tr> <td height="60"><img src="<? echo "$modelimage1"; ?>" width="65" height="65" /></td> <td height="60"><img src="<? echo "$modelimage2"; ?>" width="65" height="65" /></td> </tr> </table></td> <td width="30" height="0"> </td> </tr> </table></td> </tr> </table> <table width="800" border="0" align="center" cellpadding="0" cellspacing="0"> <tr> <td height="15" valign="bottom" class="areasmenu"><span class="basemenu">Terms & Conditions - Refund Policy - Privacy Policy - Delivery Info - Copyright Ruxx 2010 ©</span></td> </tr> </table> Thanks Hello, First of all ,I have been a regular reader of the forum and it has increased my knowledge in the PHP a lot. Till now i had just been a guest user and used to go through the queries and tips posted here. Today , for the first time i am posting on the forum Coming to a problem: I have to design a function for a client from travel domain. Where he wants the script/function to go and extract data from an airline website on the available fare for a specific date and destination. The problem is that only a part of fare(Base fare) is visible on the site and you have to select one of the given flight option(s) (after selecting the option through AJAX the full fare is displayed on the same page on the left hand site) to get the complete fare including fare and surcharges. Now i want my function/script to extract the full fare for all the available flight options. I have written the below piece of code. But its not efffective as it gets only the base fare part of the result page and not the complete fare.. The airline booking url is : http://book.spicejet.com/skylights/cgi-bin/skylights.cgi My code: //---- Spicejet start // ENGINE URL http://book.spicejet.com/skylights/cgi-bin/skylights.cgi?page=homecomp //Taking some sample search data - just to test the script $event="search" ; $module="SB" ; $page="SEARCH" ; $mode="empty" ; $travel="1" ; // ONE WAY OR 2 WAY $from1="HYD" ; $to1="DEL" ; $departDay1="13" ; $departDay2=""; $departMonth1="201010" ; $ADULT="1" ; $CHILD="0" ; $INFANT="0" ; $toCity1="DEL" ; $numberMarkets="1" ; // the page needs data in POST form $curlPost = 'event=' . urlencode($event) . '&module=' . urlencode($module) .'&page=' . urlencode($page) .'&mode=' . urlencode($mode) .'&travel=' . urlencode($travel) .'&from1=' . urlencode($from1) .'&to1=' . urlencode($to1) .'&departDay1=' . urlencode($departDay1) .'&departMonth1=' . urlencode($departMonth1) .'&ADULT=' . urlencode($ADULT) .'&CHILD=' . urlencode($CHILD) .'&INFANT=' . urlencode($INFANT) .'&toCity1=' . urlencode($toCity1) .'&numberMarkets=' . urlencode($numberMarkets).'&submitSearch=submitSearch'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'http://book.spicejet.com/skylights/cgi-bin/skylights.cgi'); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322)"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost); $data = curl_exec($ch); curl_close($ch); echo($data); Also one more query Is there something like beautiful soup (As in Python) to parse the data. I want to extract the flight timing and total price from the page and insert into a database. Thanxs in advance Hey all! Well the title is quit clear! I wanted to know how I can, through PHP, detect on a webpage if it uses the header() function to redirect to another page, in php! ( something like this : header('Location h31uh23i.php'); ) I think I need to use cURL for this! Thank you very much if you can help me! Hello! I would like to use cURL to login to the website: lockerz.com I have some code, but it doesn't seem to work: <?php // INIT CURL $ch = curl_init(); // SET URL FOR THE POST FORM LOGIN curl_setopt($ch, CURLOPT_URL, 'http://lockerz.com/auth/login'); // ENABLE HTTP POST curl_setopt ($ch, CURLOPT_POST, 1); // SET POST PARAMETERS : FORM VALUES FOR EACH FIELD curl_setopt ($ch, CURLOPT_POSTFIELDS, 'email-email=EMAIL@hotmail.com&password-password=PASSWPRD'); // IMITATE CLASSIC BROWSER'S BEHAVIOUR : HANDLE COOKIES curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt'); # Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL # not to print out the results of its query. # Instead, it will return the results as a string return value # from curl_exec() instead of the usual true/false. curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); // EXECUTE 1st REQUEST (FORM LOGIN) $store = curl_exec ($ch); // SET FILE TO DOWNLOAD curl_setopt($ch, CURLOPT_URL, 'http://lockerz.com/auction'); // EXECUTE 2nd REQUEST (FILE DOWNLOAD) $content = curl_exec ($ch); // CLOSE CURL curl_close ($ch); echo $content; ?> Thank you very much if you can help!
I'm trying to login and scrape a page 4 pages deep. I can get to the fourth page...but that page only returns AJAX ERROR:0. I know NOTHING about AJAX calls via Curl. Can someone please help me with what to look for in the source code of the 4th page (when using a browser) to what I'm supposed to pass along via CURL? |