PHP - How To Get Specific Data Using Curl

Hi there,

I'm writing an application that pulls from two separate web based APIs, collates the data into a single array of data.  Once done, it then enters a loop that cycles through the array and executes a cURL call for each cycle.  This worked perfectly for a controlled set of 10 artificial records and so I've started testing with live data.

On running the code, I eventually the following :
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

I'm not sure how to debug this and so I'm hoping I can find some help here.
I'm running WAMP, and the php.ini contains the following:
max_execution_time = 0   ; Maximum execution time of each script, in seconds
max_input_time = 0   ; Maximum amount of time each script may spend parsing request data

Through testing, I have discovered the following :
The script can do exactly 35 iterations through the loop in under 2 minutes.
Any number greater than this causes the error.
If I skip the first 20 curl submits, I can then do 21 to 55 iterations before failure, so I do not believe it is the 36th value.
If I put a sleep(5) command into the loop, I can still do 35 and no more iterations than that.
If I put a sleep(10) command into the loop, it fails on 35 and needs less.

Code: [Select]
$action = "post";
$url = "http://url.url.url/api/v1.0/udo_Position/create"; // I've replaced the actual URL out of necessity

// set user agent {
print("Setting CURL options, using URL ".$url."<br>");
$cd = curl_init();
curl_setopt($cd, CURLOPT_HEADER, 1);
curl_setopt($cd, CURLOPT_URL, $url);
curl_setopt($cd, CURLOPT_USERPWD, 'username:password');
curl_setopt($cd, CURLOPT_RETURNTRANSFER, 1); // Don't send return value to screen
curl_setopt($cd, CURLOPT_USERAGENT, "Firefox/2.0.0.1"); // spoofing FireFox 2.0
curl_setopt($cd, CURLOPT_VERBOSE, true);
// }

if ($action == 'get') {
;
} elseif ($action == 'post') {
print("POST set;<br>");
print("<pre style='font-family:verdana;font-size:13'>");
print_r($pointval);
print("</pre>");
$pointval["save"] = "Save"; // The AddNew step form requires this.

curl_setopt($cd, CURLOPT_POSTFIELDS, $pointval);
} else {
print("Invalid usage. Please see example<br>");
return;
}

$reply = curl_exec($cd);

$http_status = curl_getinfo($cd, CURLINFO_HTTP_CODE);
print("<pre style='font-family:verdana;font-size:13'>");
print_r($http_status);
print("</pre>");

print("Here is the reply from the AddNew function: <br>");
print("<pre style='font-family:verdana;font-size:13'>");
print_r($reply);
print("</pre>");

if (curl_error($cd)) {
print("Error: ".(curl_error($cd))."<br>");
}

curl_close($cd);
print("Insert Successful.<br>");

I am uncertain whether it is a PHP, Apache or cURL issue at this point.  I have another script that pushes a great deal more cURL calls to the same system but pulls from only one external resource, instead of two.  Any help or suggestions would be appreciated.

I am trying to go to http://lirr42.mta.info/
and get the data from the submitted form. I have rebuilt the form exactly as needed, and done a few adjustments.  The posts that I am sending from MY form, are the same as the one on the site.  This site allows public data mining by the way.  So does someone seem something wrong with my code?

I know the basics are working. I have another page that is working fine. It's just this one ends up returning something..it returns part of the data but it's something wrong with it.

Any advice is appreciated.
Code: [Select]
<?php
define( 'DEBUG', true );

define( 'DEFAULT_STOP', 'Broadway' );

// report all errors during development/debug mode
if ( DEBUG ) 
error_reporting( E_ALL );
else
error_reporting( 0 );

include( 'simplehtmldom/simple_html_dom.php' );

$mta_post_url = 'http://lirr42.mta.info/schedules.php';

// GTFS Specification File Definitions
$gtfs_files = array
(
'agency'          =>  'agency.txt',                  // required
'stops'           =>  'stops.txt',                   // required
'routes'          =>  'routes.txt',                  // required
'trips'           =>  'trips.txt',                   // required
'stop_times'      =>  'stop_times.txt',              // required
'calendar'        =>  'calendar.txt',                // required
'calendar_dates'  =>  'calendar_dates.txt',
'fare_rules'      =>  'fare_rules.txt',
'fare_attributes' =>  'fare_attributes.txt',
'shapes'          =>  'shapes.txt',
'frequencies'     =>  'frequencies.txt',
'transfers'       =>  'transfers.txt'
);

$gtfs_pickup_codes = array
(
    0 => 'Regularly scheduled pickup',
    1 => 'No pickup available',
    2 => 'Must phone agency to arrange pickup',
    3 => 'Must coordinate with driver to arrange pickup'
);

$gtfs_dropoff_codes = array
(
0 => 'Regularly scheduled drop off', 
    1 => 'No drop off available',
    2 => 'Must phone agency to arrange drop off',
    3 => 'Must coordinate with driver to arrange drop off'
);

// load stops file 
$stopsFile = fopen( 'data/lir/' . $gtfs_files[ 'stops' ], "r" );

if ( $stopsFile )
{
// get (and toss) header row
$stopsHeader = fgetcsv( $stopsFile, 1000 );

// print_r( $stopsHeader );

// will hold HTML output for Select dropdown for stops
$stopSelectOptions = '';

// build array from file data
while ( $data = fgetcsv( $stopsFile, 1000 ) )
{
$stopsData[ $data[1] ] = $data[0]; 
}

// get a sorted array of the stop names ( yes this could be done with array_multisort or usort but I didnj't feel like it right now )
$stopNames = array_keys( $stopsData );
sort( $stopNames );

// loop through sorted array and create SELECT options with default SELECTED at Grand Central Terminal
foreach( $stopNames as $stop )
$stopSelectOptions .= sprintf( '<option %s value="%d">%s</option>', ( DEFAULT_STOP == $stop ) ?  'selected="selected"' : '' , $stopsData[ $stop ], $stop ) . "\n\r";
}
fclose( $stopsFile );


// load ads file 
$adsFile = fopen( 'ads.txt', "r" );

if ( $adsFile )
{
// get (and toss) header row
$adsHeader = fgetcsv( $adsFile, 1000 );

// will hold ads available for stops
$ads = array();

// build array from file data
while ( $data = fgetcsv( $adsFile, 1000 ) )
{
$ads[ $data[0] ] = array
(
'filename' => $data[1],
'url'      => $data[2],
'text'     => $data[3]
);
}
fclose( $adsFile );
}


// printout the html header
require_once( 'header.php' );


if ( 'POST' == $_SERVER[ 'REQUEST_METHOD' ] ) 
{
$from_stop = $_POST[ 'FromStation' ];  // @todo Filter! http://www.php.net/manual/en/filter.filters.validate.php
$orig_station = $from_stop;

$to_stop = $_POST[ 'ToStation' ];  // @todo Filter! http://www.php.net/manual/en/filter.filters.validate.php
$dest_station = $to_stop;

if (  $to_stop == $from_stop ) 
{
$error = 'Originating and Destination stops are the same.';
}
else
{

print_ad( $orig_station, $dest_station, $location='top' );

$travel_date = $_POST[ 'RequestDate' ];
$requestTime = $_POST[ 'RequestTime' ];
$am_pm       = $_POST[ 'RequestAMPM' ];
$filter      = $_POST[ 'sortBy' ];

/*
* Lets post this to MTA.info
*/

$mta_curl = curl_init();

$mta_curl_options = array
(
CURLOPT_FAILONERROR    => true,
CURLOPT_FOLLOWLOCATION => false,
// CURLOPT_MUTE           => true,
CURLOPT_POST           => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_CONNECTTIMEOUT => 30,
CURLOPT_TIMEOUT        => 30,
CURLOPT_URL            => $mta_post_url,
CURLOPT_REFERER        => 'http://lirr42.mta.info/',
CURLOPT_USERAGENT      => 'MTA Info Scrape/1.0',
CURLOPT_POSTFIELDS     => http_build_query( $_POST )
);
curl_setopt_array( $mta_curl, $mta_curl_options );
$mta_response = curl_exec( $mta_curl );

if ( false === $mta_response )
{
curl_close( $mta_curl );
die( sprintf( 'Response Code:%s, Curl Error No:%s, Curl Error Message:%s', curl_getinfo( $mta_curl, CURLINFO_HTTP_CODE ),curl_errno( $mta_curl ), curl_error( $mta_curl ) ) );
}
else
{
curl_close( $mta_curl );
echo '<pre>' . htmlentities( $mta_response ) . '</pre>';

$html = new simple_html_dom();
$html->load( $mta_response );
$schedule_table = $html->find( 'table', 0 );  // find second table in the response
$row_count = 0;

// will hold HTML output for table
$stopSchedule = '<table><th>Departs</th><th>Arrives</th><th>Minutes</th><th>Transfer</th><th>Fare</th></tr>';

foreach ( $schedule_table->find('tr') as $schedule_row ) 
{
$row_count++;
// check for and skip header row
if ( 1 == $row_count )
continue;

// check for and skip last row which first TD has rowspan attribute
if ( $schedule_row->children(0)->colspan )
{
continue;
}
// extract table data for this row
$depart_time      = $schedule_row->children( 0 )->innertext;
$arrive_time      = $schedule_row->children( 2 )->innertext;
$minutes_traveled = $schedule_row->children( 4 )->innertext;
$transfer         = $schedule_row->children( 5 )->innertext;
$fare             = $schedule_row->children( 6 )->plaintext; 
// add table row to html output
$stopSchedule .= sprintf( '<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>', $depart_time, $arrive_time, $minutes_traveled, $transfer, $fare ) . "\n\r";
}  // end foreach schedule_row

// add table end tag to html output
$stopSchedule .= '</table>';

// output table to browser
echo $stopSchedule;

print_ad( $orig_station, $dest_station, $location='bottom' );

} // end if mta response

} // end if not same orig and dest 

} // end if POST


if ( 'GET' == $_SERVER[ 'REQUEST_METHOD' ] || ! empty( $error ) ) 
{

print_ad( null, null, $location = 'default' );

?>

<form method="post">

<?php

if ( ! empty ( $error ) )
{
echo sprintf( '<div class="error">%s</div>', $error );
}

?>

From:<br />
<select id="orig_station" name="FromStation" tabindex="2">
<?php echo $stopSelectOptions; ?>
</select>

<br />
<br />

To:<br />
<select id="dest_station" name="ToStation" tabindex="3">
<?php echo $stopSelectOptions; ?>
</select>

<br />
<br />

<input id="date1" name="RequestDate" size="12" maxlength="10" tabindex="4" value="<?php echo date( 'm/d/Y' ); ?>" />

<?php 
$currentHour = trim( date( 'h' ) );
$currentMinutes = trim( date( 'i' ) );
$currentMinutesFloor =  sprintf( '%02d' , $currentMinutes - ( $currentMinutes % 30 ) );
$currentAMPM = trim ( date( 'A' ) );

$start_time = strtotime( '1:00 AM' );
$end_time = strtotime( '1:00 PM' );
?>

<select name="RequestTime" tabindex="5">
<?php
while( $start_time < $end_time )
{
$option_time = date( 'h:i', $start_time );
$option_hour = trim( date( 'h', $start_time ) );

$option_minutes = trim( date( 'i', $start_time ) );
$option_selected = (  $option_hour == $currentHour && $option_minutes  == $currentMinutesFloor );

echo sprintf( '<option %s>%s</option>', ( $option_selected ) ? 'selected="selected"' : '',  $option_time ) . "\r\n";

$start_time = $start_time + 30*60;
}
?>
</select>

<select name="RequestAMPM" tabindex="6">

<option value="AM" <?php echo ( 'AM' == $currentAMPM ) ? 'selected="SELECTED"' : ''; ?>>AM</option>
<option value="PM" <?php echo ( 'PM' == $currentAMPM ) ? 'selected="SELECTED"' : ''; ?>>PM</option>
</select>

<input name="sortBy" type="hidden" value="1" />
<input type="submit" name="submit" value="Check Schedule" />
</form>


<!--  images/base/
ic_menu_back.png
ic_menu_home.png

-->
<?php 

}

require_once( 'footer.php' );

?>

I'm using cURL to crawl and scrape data from a website. This website contains tables with rows of data. When I send a cURL POST for the underlying data at a specific row(A), it will return the expected data. But when I move to the second row(B), the data returns blank or specifically, a tons of spaces (or &#nbsp's.) When I access the cURL's POST location by browser, I can see (B)'s data. The only difference in the 2 POST's are location ID's for the data. I don't think it's a problem with JavaScript as I can successfully return data from row (A) as I mentioned.

OK, I have the initial cURL working but need to figure out how to extract data I want off that webpage to display or store in a database, I tried using dom and xpath, but because of the way the page displays using css, i think its not picking it up. Here is my cURL script:
<?php
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

$target_url = "www.test.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html = curl_exec($ch);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
// parse the html into a DOMDocument
$dom = new DOMDocument();
$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//td");

for ($i = 0; $i < $hrefs->length; $i++) {
$href = $hrefs->item($i);
$url = $href->getAttribute('href');
storeLink($url,$target_url);
echo "<br />Link stored: $url";
}

?>
and here is a snippet of the source of the page I am getting:

<span id="lblTest"><h1 id='surrZipTitle'>Agents in Surrounding Zip Codes</h1><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>4444 MANZANITA AVE STE 6</span><br /><span>CARMICHAEL                    , CA 95608-1488</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415199904150295&lat=38.646142&lng=-121.327623' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a> &nbsp;&nbsp; <a class='faaBlueLink' id='lnkWebSite' style='display: none;' href='http://' target='_blank' onclick="return trackEvent('/External-Link/AgentWebsite/                                                                      ','PROFESSIONAL INS ASSOC, INC.                                ');">Website</a></div></td><td valign='top'>&nbsp;</td></tr></table></td></tr></table><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>AMERICAN AIM AUTO INS AGY, INC</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>5339 SAN JUAN AVE</span><br /><span>FAIR OAKS                     , CA 95628-3318</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415911704151222&lat=38.66237&lng=-121.292429' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a>

So basically I want to extract the agency name like "<span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span>" and the address which always use the same div class like "caaAgencyName2" and "addressBlock".  How can this be accomplished?

Hi there,
OK, there is what I'm trying to accomplish - I want to send POST instantly when the page is loaded without "Submit" button or anything like that.
Here how it looks using JavaScript:
Code: [Select]
<form name="postdata" action="http://www.mywebsite.com/post.php" method="post">
    <input name="data" type="text" value="1"/>
</form>
<script type="text/javascript" language="JavaScript">
document.postdata.submit();
</script>

Works fine with JavaScript, however i can't use it.

Here is reference for CURL: http://php.net/manual/en/book.curl.php
I found this curl_post function at php.net to simplify things up.
<?php

/**
 * Send a POST requst using cURL
 * @param string $url to request
 * @param array $post values to send
 * @param array $options for cURL
 * @return string
 */
function curl_post($url, array $post = NULL, array $options = array())
{
    $defaults = array(
        CURLOPT_POST => 1,
        CURLOPT_HEADER => 0,
        CURLOPT_URL => $url,
        CURLOPT_FRESH_CONNECT => 1,
        CURLOPT_RETURNTRANSFER => 1,
        CURLOPT_FORBID_REUSE => 1,
        CURLOPT_TIMEOUT => 4,
        CURLOPT_POSTFIELDS => http_build_query($post)
    );

    $ch = curl_init();
    curl_setopt_array($ch, ($options + $defaults));
    if( ! $result = curl_exec($ch))
    {
        trigger_error(curl_error($ch));
    }
    curl_close($ch);
    return $result;
}
$test = curl_post("http://www.mywebsite.com/post.php", array("data" => "1"));
print($test);
?>

Looks like it uses GET instead POST (used sniffer to verify this).
I'm trying to send POST data to other host, if that matters.
Any help would be appreciated.
Thanks in advance.