PHP - Scrape Website With Curl

I'm just doing a simple scrape on a web page.. works great, except...  the section I scrape is just 10 words inside a table which are updated every few hours. 

Problem is...  my scrape seems to only grab what I found the first time I ran my script, despite the new words being updated a couple hours ago. 

I've tested everything script wise and looked at the html source to be sure everything is showing up... just wondering, is there any thing you can think of that would allow for caching (from source site or the php script) of the words on a page when running my script?  I even opened 2 different browsers and ran the script fresh from these browsers that had never ran it.  When I echo the results, I still get the original result set.

I'm perplexed. 

I'm trying to get a value which changes but is always contained the same way

this is the text Next: Brigadier  (81654 of 220000)

as you can see below this is the value i need to get I just can't figure out the RegEx for it? help please!

Smaller CODE
Code: [Select]
<div class="rankBar">
                        <p class="grade"><span id="ctl00_mainContent_identityBar_currentGlobalRankLabel" class="current">Colonel Grade 3</span><span id="ctl00_mainContent_identityBar_nextGlobalRankLabel" class="future">Next: Brigadier&nbsp;&nbsp;(81654 of 220000)</span></p>
                        <div class="rankMeter">
                            <div id="ctl00_mainContent_identityBar_rankBarPanel" class="bar" style="width:37%;">
<span></span>
</div>
                        </div>

FULL CODE

Code: [Select]
<div id="ctl00_mainContent_identityBar_idbarPanel">

        <div class="glowBox idRankContainer">
            <div class="corner bottomLeft"></div>
            <div class="lattice"></div>
            <div class="content">
                <a href="/Stats/Reach/default.aspx?player=l RaH l" id="ctl00_mainContent_identityBar_avatar" class="avatar"><img id="ctl00_mainContent_identityBar_emblemImg" src="/Stats/emblem.ashx?s=70&amp;0=0&amp;1=0&amp;2=0&amp;3=0&amp;fi=0&amp;bi=0&amp;fl=0&amp;m=3" style="height:70px;width:70px;border-width:0px;"></a>
                <div class="idRank">
                    <div class="userInfo">
<img id="ctl00_mainContent_identityBar_namePlateImg" class="img_plate" src="/images/reachstats/nameplates/512.png" alt="" style="height:30px;width:60px;border-width:0px;">
                       
                        <a href="/projects/reach/article.aspx?ucc=faq&amp;cid=28756"><img id="ctl00_mainContent_identityBar_currentGlobalRankImage" title="Colonel Grade 3" class="img_rankIcon" src="/images/reachstats/grades/med/A4BF62C6-1E3F-468A-9A73-8237600B2AD3.png" alt="Colonel Grade 3" style="height:40px;width:40px;border-width:0px;"></a> 
                        <h2><a href="/Stats/Reach/default.aspx?player=l RaH l" id="ctl00_mainContent_identityBar_gamerTag">l RaH l</a> - FLIR</h2>
                       
                    </div>
                    <div class="rankBar">
                        <p class="grade"><span id="ctl00_mainContent_identityBar_currentGlobalRankLabel" class="current">Colonel Grade 3</span><span id="ctl00_mainContent_identityBar_nextGlobalRankLabel" class="future">Next: Brigadier&nbsp;&nbsp;(81654 of 220000)</span></p>
                        <div class="rankMeter">
                            <div id="ctl00_mainContent_identityBar_rankBarPanel" class="bar" style="width:37%;">
<span></span>
</div>
                        </div>
                    </div>
                </div>
            </div>
        </div>
        <div class="glowBox shields">
            <div class="corner topRight"></div>
            <div class="lattice"></div>
            <div class="content">
                <ul>
                    <li class="campaign">
                        <p>Campaign</p>
                        <img id="ctl00_mainContent_identityBar_spCampaignImage" title="Completed Legendary Difficulty" alt="Campaign Progress" src="/images/reachstats/campaign_progress/legendary.png" style="height:59px;width:54px;border-width:0px;">
                    </li>
                    <li class="coop">
                        <p>Co-op</p>
                        <img id="ctl00_mainContent_identityBar_coopCampaignImage" title="Completed Normal Difficulty" alt="Coop Campaign Progress" src="/images/reachstats/campaign_progress/normal.png" style="height:59px;width:54px;border-width:0px;">
                    </li>
                    <li class="arena_display">
                        <p>Arena</p>
                        <a id="ctl00_mainContent_identityBar_arenaLink" href="/stats/reach/careerstats/playlists.aspx?player=l RaH l&amp;vc=2"><img id="ctl00_mainContent_identityBar_currentBestArenaDivisionImg" alt="" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;"></a>
                        <div class="glowBox popOut po_arenaSeason2">
                            <div class="corner topRight"></div>
                            <div class="corner bottomLeft"></div>
                            <div class="content">       
                                <h4>Arena Season 6</h4>

<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>Doubles Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>
<div class="rule_transparent"></div>
<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>FFA Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>
<div class="rule_transparent"></div>
<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>Team Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>

   
                            </div>
                        </div>     
                    </li>
                </ul>
            </div>
        </div>     
</div>

I need to scrape a Chinese website. (I guess there is no difference between scraping a Chinese website and a normal one?)

It's my suppliers weblink. They have have told me to download images and text from their website profile link off 1688.com.

There is an API - but from what I've read, it's pants + my virus checked doesn't allow me to visit the API doc page.

What tool should I use?

I've got lots of experience coding - but master of none. Maybe I still fall into the newbie category. LOL.

I saw a link from an article... they gave these names:

Goutte
Simple HTML DOM
htmlSQL
cURL
Requests
HTTPful
Buzz
Guzzle

Which should I consider?

IMPORTANT: I need to download and then upload into my Woocommerce website.

I need images + variation details. I'll have a little file with translations, so if XYZ is found in Chinese, then it is replaced with this.

I thought I would mention the extra detail incase it was relevant to considering which scraping tool to pick.

Thanks!

 

Edited July 11, 2019 by acebase

OK, I have the initial cURL working but need to figure out how to extract data I want off that webpage to display or store in a database, I tried using dom and xpath, but because of the way the page displays using css, i think its not picking it up. Here is my cURL script:
<?php
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

$target_url = "www.test.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html = curl_exec($ch);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
// parse the html into a DOMDocument
$dom = new DOMDocument();
$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//td");

for ($i = 0; $i < $hrefs->length; $i++) {
$href = $hrefs->item($i);
$url = $href->getAttribute('href');
storeLink($url,$target_url);
echo "<br />Link stored: $url";
}

?>
and here is a snippet of the source of the page I am getting:

<span id="lblTest"><h1 id='surrZipTitle'>Agents in Surrounding Zip Codes</h1><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>4444 MANZANITA AVE STE 6</span><br /><span>CARMICHAEL                    , CA 95608-1488</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415199904150295&lat=38.646142&lng=-121.327623' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a> &nbsp;&nbsp; <a class='faaBlueLink' id='lnkWebSite' style='display: none;' href='http://' target='_blank' onclick="return trackEvent('/External-Link/AgentWebsite/                                                                      ','PROFESSIONAL INS ASSOC, INC.                                ');">Website</a></div></td><td valign='top'>&nbsp;</td></tr></table></td></tr></table><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>AMERICAN AIM AUTO INS AGY, INC</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>5339 SAN JUAN AVE</span><br /><span>FAIR OAKS                     , CA 95628-3318</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415911704151222&lat=38.66237&lng=-121.292429' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a>

So basically I want to extract the agency name like "<span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span>" and the address which always use the same div class like "caaAgencyName2" and "addressBlock".  How can this be accomplished?

Hello,

I am trying to use cURL to login to a website, but I can't seem to get it working.

Website I'm trying to login to:
http://www.uniquearticlewizard.com/amember/member.php

Here is what their form code looks like:
Code: [Select]
<form name="login" method="post" action="/amember/member.php">
 
<table class="vedit" >
    <tr>
        <th>Username</th>
        <td><input type="text" name="amember_login" size="15" value="" /></td>
    </tr>
    <tr>
        <th>Password</th>
        <td><input type="password" name="amember_pass" size="15" /></td>
    </tr>
        <tr>
        <td colspan="2" style='padding:0px; padding-bottom: 2px;'>
<input type="checkbox" name="remember_login" value="1">
<span class="small">Remember my password?</span>
        </td>
    </tr>
    </table>
<input type="hidden" name="login_attempt_id" value="1291657877" />
<br />
 
<span class='button'><input type="submit" value="&nbsp;&nbsp;&nbsp;Login&nbsp;&nbsp;&nbsp;" /></span>&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<span class='button'><input type="button" value="&nbsp;&nbsp;&nbsp;Back&nbsp;&nbsp;&nbsp;" onclick="history.back(-1)" /></span>
</form>

As you can see they are using a javascript button to submit the form, which doesn't have a name attribute. So I'm not sure how to get around this and tell cURL to submit the form. When I Googled I found something that said just submit the other information and it will submit itself, but I'm not sure if that's right.

Here is my attempt, but I just get a blank screen. I think the script is working, but something on there end is exiting out due to me not supplying a required piece of information. I'm not sure what that is though.

Code: [Select]
<?php

set_time_limit(0);

$options = array(
  CURLOPT_RETURNTRANSFER => true,     // return web page
  CURLOPT_HEADER         => false,    // don't return headers
  CURLOPT_FOLLOWLOCATION => true,     // follow redirects
  CURLOPT_ENCODING       => "",       // handle all encodings
  CURLOPT_USERAGENT      => "spider", // who am i
  CURLOPT_AUTOREFERER    => true,     // set referer on redirect
  CURLOPT_CONNECTTIMEOUT => 120,      // timeout on connect
  CURLOPT_TIMEOUT        => 120,      // timeout on response
  CURLOPT_MAXREDIRS      => 10,       // stop after 10 redirects
);

$ch      = curl_init( "http://www.uniquearticlewizard.com/amember/member.php" );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err     = curl_errno( $ch );
$errmsg  = curl_error( $ch );
$header  = curl_getinfo( $ch );
curl_close( $ch );

$header['content'] = $content;

preg_match('/name="login_attempt_id" value="(.*)" \/>/', $header['content'], $form_id);

$value = $form_id[1];

$ch = curl_init();

// SET URL FOR THE POST FORM LOGIN
curl_setopt($ch, CURLOPT_URL, 'http://www.uniquearticlewizard.com/amember/member.php');

// ENABLE HTTP POST
curl_setopt ($ch, CURLOPT_POST, 1);

$data = array('amember_login' => '*****', 'amember_pass' => '*****', 'login_attempt_id' => $value, 'remember_login' => '1');

// SET POST PARAMETERS : FORM VALUES FOR EACH FIELD
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

// IMITATE CLASSIC BROWSER'S BEHAVIOUR : HANDLE COOKIES
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

# Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
# not to print out the results of its query.
# Instead, it will return the results as a string return value
# from curl_exec() instead of the usual true/false.
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

// EXECUTE 1st REQUEST (FORM LOGIN)
$store = curl_exec($ch);

echo $store;

curl_close ($ch); 

?>

They do have a form value that changes on every page refresh, it just tracks the login attempt (which is a long number). I was able to scrape that and put it in the form with the correct value. I thought adding that would successfully log me in, but apparently there is something else going on.

Any help would be greatly appreciated!