PHP - Curl Crawling Data

Hello everyone,
I developed a php application that crawls a site and generates an xml sitemap with the gathered information.

It works, but as of now I am using brute force tactics.

I have a class that crawls and stores the links in a tree by returning the file_get_contents and using preg match to find the a tags. Is there a quicker method? I've seen people talking about cURL but i don't know if that will make my program any better. My application seems to get results a bit quicker than some others I have seen.

My main concern comes with the sorting. Is there a way to tell if a link on a page is an rss feed or like a downloadable image or zip file or something?
For files, I explode at '/' and check the last array key for a '.' , then I check it against an array of file names I think I want to include.
For feeds I just check the explode array for feed, feeds , rss or ?feed=rss2 is in the array before storage. This works fine for sites I administer and wordpress sites, but it could filter out a cooking site link or something with a feed directory. It also seems like it is one of the most time consuming parts.

I think what I am trying to ask... is there a good way to filter these results? Will cURL or anything else let me check for actual pages and filter out .mp3 files and all the other junk you don't want in a sitemap? Thanks in advance for your time.

Hello,

I had made a website (PHP) for a music company  few years back. They basically sell songs from their site. Now they've faced a problem. Their site has been crawled by abmp3.com and they not only let user download songs from our site but also give a full path of a song to download.

I wonder how this happened and how to stop them from crawling our site. And its likely some other site doing same too. So please anybody help me how to overcome this problem, may be some PHP code can do this??.

Thanks
watsmyname

I am trying to go to http://lirr42.mta.info/
and get the data from the submitted form. I have rebuilt the form exactly as needed, and done a few adjustments.  The posts that I am sending from MY form, are the same as the one on the site.  This site allows public data mining by the way.  So does someone seem something wrong with my code?

I know the basics are working. I have another page that is working fine. It's just this one ends up returning something..it returns part of the data but it's something wrong with it.

Any advice is appreciated.
Code: [Select]
<?php
define( 'DEBUG', true );

define( 'DEFAULT_STOP', 'Broadway' );

// report all errors during development/debug mode
if ( DEBUG ) 
error_reporting( E_ALL );
else
error_reporting( 0 );

include( 'simplehtmldom/simple_html_dom.php' );

$mta_post_url = 'http://lirr42.mta.info/schedules.php';

// GTFS Specification File Definitions
$gtfs_files = array
(
'agency'          =>  'agency.txt',                  // required
'stops'           =>  'stops.txt',                   // required
'routes'          =>  'routes.txt',                  // required
'trips'           =>  'trips.txt',                   // required
'stop_times'      =>  'stop_times.txt',              // required
'calendar'        =>  'calendar.txt',                // required
'calendar_dates'  =>  'calendar_dates.txt',
'fare_rules'      =>  'fare_rules.txt',
'fare_attributes' =>  'fare_attributes.txt',
'shapes'          =>  'shapes.txt',
'frequencies'     =>  'frequencies.txt',
'transfers'       =>  'transfers.txt'
);

$gtfs_pickup_codes = array
(
    0 => 'Regularly scheduled pickup',
    1 => 'No pickup available',
    2 => 'Must phone agency to arrange pickup',
    3 => 'Must coordinate with driver to arrange pickup'
);

$gtfs_dropoff_codes = array
(
0 => 'Regularly scheduled drop off', 
    1 => 'No drop off available',
    2 => 'Must phone agency to arrange drop off',
    3 => 'Must coordinate with driver to arrange drop off'
);

// load stops file 
$stopsFile = fopen( 'data/lir/' . $gtfs_files[ 'stops' ], "r" );

if ( $stopsFile )
{
// get (and toss) header row
$stopsHeader = fgetcsv( $stopsFile, 1000 );

// print_r( $stopsHeader );

// will hold HTML output for Select dropdown for stops
$stopSelectOptions = '';

// build array from file data
while ( $data = fgetcsv( $stopsFile, 1000 ) )
{
$stopsData[ $data[1] ] = $data[0]; 
}

// get a sorted array of the stop names ( yes this could be done with array_multisort or usort but I didnj't feel like it right now )
$stopNames = array_keys( $stopsData );
sort( $stopNames );

// loop through sorted array and create SELECT options with default SELECTED at Grand Central Terminal
foreach( $stopNames as $stop )
$stopSelectOptions .= sprintf( '<option %s value="%d">%s</option>', ( DEFAULT_STOP == $stop ) ?  'selected="selected"' : '' , $stopsData[ $stop ], $stop ) . "\n\r";
}
fclose( $stopsFile );


// load ads file 
$adsFile = fopen( 'ads.txt', "r" );

if ( $adsFile )
{
// get (and toss) header row
$adsHeader = fgetcsv( $adsFile, 1000 );

// will hold ads available for stops
$ads = array();

// build array from file data
while ( $data = fgetcsv( $adsFile, 1000 ) )
{
$ads[ $data[0] ] = array
(
'filename' => $data[1],
'url'      => $data[2],
'text'     => $data[3]
);
}
fclose( $adsFile );
}


// printout the html header
require_once( 'header.php' );


if ( 'POST' == $_SERVER[ 'REQUEST_METHOD' ] ) 
{
$from_stop = $_POST[ 'FromStation' ];  // @todo Filter! http://www.php.net/manual/en/filter.filters.validate.php
$orig_station = $from_stop;

$to_stop = $_POST[ 'ToStation' ];  // @todo Filter! http://www.php.net/manual/en/filter.filters.validate.php
$dest_station = $to_stop;

if (  $to_stop == $from_stop ) 
{
$error = 'Originating and Destination stops are the same.';
}
else
{

print_ad( $orig_station, $dest_station, $location='top' );

$travel_date = $_POST[ 'RequestDate' ];
$requestTime = $_POST[ 'RequestTime' ];
$am_pm       = $_POST[ 'RequestAMPM' ];
$filter      = $_POST[ 'sortBy' ];

/*
* Lets post this to MTA.info
*/

$mta_curl = curl_init();

$mta_curl_options = array
(
CURLOPT_FAILONERROR    => true,
CURLOPT_FOLLOWLOCATION => false,
// CURLOPT_MUTE           => true,
CURLOPT_POST           => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_CONNECTTIMEOUT => 30,
CURLOPT_TIMEOUT        => 30,
CURLOPT_URL            => $mta_post_url,
CURLOPT_REFERER        => 'http://lirr42.mta.info/',
CURLOPT_USERAGENT      => 'MTA Info Scrape/1.0',
CURLOPT_POSTFIELDS     => http_build_query( $_POST )
);
curl_setopt_array( $mta_curl, $mta_curl_options );
$mta_response = curl_exec( $mta_curl );

if ( false === $mta_response )
{
curl_close( $mta_curl );
die( sprintf( 'Response Code:%s, Curl Error No:%s, Curl Error Message:%s', curl_getinfo( $mta_curl, CURLINFO_HTTP_CODE ),curl_errno( $mta_curl ), curl_error( $mta_curl ) ) );
}
else
{
curl_close( $mta_curl );
echo '<pre>' . htmlentities( $mta_response ) . '</pre>';

$html = new simple_html_dom();
$html->load( $mta_response );
$schedule_table = $html->find( 'table', 0 );  // find second table in the response
$row_count = 0;

// will hold HTML output for table
$stopSchedule = '<table><th>Departs</th><th>Arrives</th><th>Minutes</th><th>Transfer</th><th>Fare</th></tr>';

foreach ( $schedule_table->find('tr') as $schedule_row ) 
{
$row_count++;
// check for and skip header row
if ( 1 == $row_count )
continue;

// check for and skip last row which first TD has rowspan attribute
if ( $schedule_row->children(0)->colspan )
{
continue;
}
// extract table data for this row
$depart_time      = $schedule_row->children( 0 )->innertext;
$arrive_time      = $schedule_row->children( 2 )->innertext;
$minutes_traveled = $schedule_row->children( 4 )->innertext;
$transfer         = $schedule_row->children( 5 )->innertext;
$fare             = $schedule_row->children( 6 )->plaintext; 
// add table row to html output
$stopSchedule .= sprintf( '<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>', $depart_time, $arrive_time, $minutes_traveled, $transfer, $fare ) . "\n\r";
}  // end foreach schedule_row

// add table end tag to html output
$stopSchedule .= '</table>';

// output table to browser
echo $stopSchedule;

print_ad( $orig_station, $dest_station, $location='bottom' );

} // end if mta response

} // end if not same orig and dest 

} // end if POST


if ( 'GET' == $_SERVER[ 'REQUEST_METHOD' ] || ! empty( $error ) ) 
{

print_ad( null, null, $location = 'default' );

?>

<form method="post">

<?php

if ( ! empty ( $error ) )
{
echo sprintf( '<div class="error">%s</div>', $error );
}

?>

From:<br />
<select id="orig_station" name="FromStation" tabindex="2">
<?php echo $stopSelectOptions; ?>
</select>

<br />
<br />

To:<br />
<select id="dest_station" name="ToStation" tabindex="3">
<?php echo $stopSelectOptions; ?>
</select>

<br />
<br />

<input id="date1" name="RequestDate" size="12" maxlength="10" tabindex="4" value="<?php echo date( 'm/d/Y' ); ?>" />

<?php 
$currentHour = trim( date( 'h' ) );
$currentMinutes = trim( date( 'i' ) );
$currentMinutesFloor =  sprintf( '%02d' , $currentMinutes - ( $currentMinutes % 30 ) );
$currentAMPM = trim ( date( 'A' ) );

$start_time = strtotime( '1:00 AM' );
$end_time = strtotime( '1:00 PM' );
?>

<select name="RequestTime" tabindex="5">
<?php
while( $start_time < $end_time )
{
$option_time = date( 'h:i', $start_time );
$option_hour = trim( date( 'h', $start_time ) );

$option_minutes = trim( date( 'i', $start_time ) );
$option_selected = (  $option_hour == $currentHour && $option_minutes  == $currentMinutesFloor );

echo sprintf( '<option %s>%s</option>', ( $option_selected ) ? 'selected="selected"' : '',  $option_time ) . "\r\n";

$start_time = $start_time + 30*60;
}
?>
</select>

<select name="RequestAMPM" tabindex="6">

<option value="AM" <?php echo ( 'AM' == $currentAMPM ) ? 'selected="SELECTED"' : ''; ?>>AM</option>
<option value="PM" <?php echo ( 'PM' == $currentAMPM ) ? 'selected="SELECTED"' : ''; ?>>PM</option>
</select>

<input name="sortBy" type="hidden" value="1" />
<input type="submit" name="submit" value="Check Schedule" />
</form>


<!--  images/base/
ic_menu_back.png
ic_menu_home.png

-->
<?php 

}

require_once( 'footer.php' );

?>

OK, I have the initial cURL working but need to figure out how to extract data I want off that webpage to display or store in a database, I tried using dom and xpath, but because of the way the page displays using css, i think its not picking it up. Here is my cURL script:
<?php
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

$target_url = "www.test.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html = curl_exec($ch);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
// parse the html into a DOMDocument
$dom = new DOMDocument();
$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//td");

for ($i = 0; $i < $hrefs->length; $i++) {
$href = $hrefs->item($i);
$url = $href->getAttribute('href');
storeLink($url,$target_url);
echo "<br />Link stored: $url";
}

?>
and here is a snippet of the source of the page I am getting:

<span id="lblTest"><h1 id='surrZipTitle'>Agents in Surrounding Zip Codes</h1><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>4444 MANZANITA AVE STE 6</span><br /><span>CARMICHAEL                    , CA 95608-1488</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415199904150295&lat=38.646142&lng=-121.327623' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a> &nbsp;&nbsp; <a class='faaBlueLink' id='lnkWebSite' style='display: none;' href='http://' target='_blank' onclick="return trackEvent('/External-Link/AgentWebsite/                                                                      ','PROFESSIONAL INS ASSOC, INC.                                ');">Website</a></div></td><td valign='top'>&nbsp;</td></tr></table></td></tr></table><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>AMERICAN AIM AUTO INS AGY, INC</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>5339 SAN JUAN AVE</span><br /><span>FAIR OAKS                     , CA 95628-3318</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415911704151222&lat=38.66237&lng=-121.292429' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a>

So basically I want to extract the agency name like "<span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span>" and the address which always use the same div class like "caaAgencyName2" and "addressBlock".  How can this be accomplished?

First of all... this is a GREAT forum. Been lurking around here for a long time and searched all over but still can't find a solution to a problem I'm facing.

Here's my cURL code - I am basically retrieving data from a web site:
<?php

$curl_handle=curl_init();
curl_setopt($curl_handle,CURLOPT_URL,'http://myprimemobile.com/update.html');
curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2);
curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1);
$buffer = curl_exec($curl_handle);
curl_close($curl_handle);

if (empty($buffer))
{
    print "Connection Timeout<p>";
}
else
{
    print $buffer;
}
?>

If you take a look at the site http://myprimemobile.com/update.html -- it will begin to update new information. I want to basically retrieve the data from that page but in my own way.

I want to loop through the page and search for all the names, addresses, and phone numbers only (not store hours or anything else), and want to put it in a variable so I can create a table with all that information on my page.

Based on the code, all the information is stored in the $buffer variable. I want to break this variable down to extract certain data in a 'div' or a 'table'. Please help me out. Thanks!