PHP - How To Scrape?

I am using file_get_contents with a url (http) to screen scrape certain web pages that are publicly accessible.
However there are certain web pages (https) that require me to use an x509 certificate stored in my browser to make them visible.
I would like to scrape them too.

How would I scrape them using a php script?

Thanks in Advance.

I'm trying to get a value which changes but is always contained the same way

this is the text Next: Brigadier  (81654 of 220000)

as you can see below this is the value i need to get I just can't figure out the RegEx for it? help please!

Smaller CODE
Code: [Select]
<div class="rankBar">
                        <p class="grade"><span id="ctl00_mainContent_identityBar_currentGlobalRankLabel" class="current">Colonel Grade 3</span><span id="ctl00_mainContent_identityBar_nextGlobalRankLabel" class="future">Next: Brigadier&nbsp;&nbsp;(81654 of 220000)</span></p>
                        <div class="rankMeter">
                            <div id="ctl00_mainContent_identityBar_rankBarPanel" class="bar" style="width:37%;">
<span></span>
</div>
                        </div>

FULL CODE

Code: [Select]
<div id="ctl00_mainContent_identityBar_idbarPanel">

        <div class="glowBox idRankContainer">
            <div class="corner bottomLeft"></div>
            <div class="lattice"></div>
            <div class="content">
                <a href="/Stats/Reach/default.aspx?player=l RaH l" id="ctl00_mainContent_identityBar_avatar" class="avatar"><img id="ctl00_mainContent_identityBar_emblemImg" src="/Stats/emblem.ashx?s=70&amp;0=0&amp;1=0&amp;2=0&amp;3=0&amp;fi=0&amp;bi=0&amp;fl=0&amp;m=3" style="height:70px;width:70px;border-width:0px;"></a>
                <div class="idRank">
                    <div class="userInfo">
<img id="ctl00_mainContent_identityBar_namePlateImg" class="img_plate" src="/images/reachstats/nameplates/512.png" alt="" style="height:30px;width:60px;border-width:0px;">
                       
                        <a href="/projects/reach/article.aspx?ucc=faq&amp;cid=28756"><img id="ctl00_mainContent_identityBar_currentGlobalRankImage" title="Colonel Grade 3" class="img_rankIcon" src="/images/reachstats/grades/med/A4BF62C6-1E3F-468A-9A73-8237600B2AD3.png" alt="Colonel Grade 3" style="height:40px;width:40px;border-width:0px;"></a> 
                        <h2><a href="/Stats/Reach/default.aspx?player=l RaH l" id="ctl00_mainContent_identityBar_gamerTag">l RaH l</a> - FLIR</h2>
                       
                    </div>
                    <div class="rankBar">
                        <p class="grade"><span id="ctl00_mainContent_identityBar_currentGlobalRankLabel" class="current">Colonel Grade 3</span><span id="ctl00_mainContent_identityBar_nextGlobalRankLabel" class="future">Next: Brigadier&nbsp;&nbsp;(81654 of 220000)</span></p>
                        <div class="rankMeter">
                            <div id="ctl00_mainContent_identityBar_rankBarPanel" class="bar" style="width:37%;">
<span></span>
</div>
                        </div>
                    </div>
                </div>
            </div>
        </div>
        <div class="glowBox shields">
            <div class="corner topRight"></div>
            <div class="lattice"></div>
            <div class="content">
                <ul>
                    <li class="campaign">
                        <p>Campaign</p>
                        <img id="ctl00_mainContent_identityBar_spCampaignImage" title="Completed Legendary Difficulty" alt="Campaign Progress" src="/images/reachstats/campaign_progress/legendary.png" style="height:59px;width:54px;border-width:0px;">
                    </li>
                    <li class="coop">
                        <p>Co-op</p>
                        <img id="ctl00_mainContent_identityBar_coopCampaignImage" title="Completed Normal Difficulty" alt="Coop Campaign Progress" src="/images/reachstats/campaign_progress/normal.png" style="height:59px;width:54px;border-width:0px;">
                    </li>
                    <li class="arena_display">
                        <p>Arena</p>
                        <a id="ctl00_mainContent_identityBar_arenaLink" href="/stats/reach/careerstats/playlists.aspx?player=l RaH l&amp;vc=2"><img id="ctl00_mainContent_identityBar_currentBestArenaDivisionImg" alt="" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;"></a>
                        <div class="glowBox popOut po_arenaSeason2">
                            <div class="corner topRight"></div>
                            <div class="corner bottomLeft"></div>
                            <div class="content">       
                                <h4>Arena Season 6</h4>

<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>Doubles Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>
<div class="rule_transparent"></div>
<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>FFA Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>
<div class="rule_transparent"></div>
<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>Team Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>

   
                            </div>
                        </div>     
                    </li>
                </ul>
            </div>
        </div>     
</div>

I usually run scrape on a website and get information and the new ones are usually added automatically but its not adding a new information. I am wondering whethere I have an issue with my code.

I usually run scrape through the sitemap.

Code: [Select]
$qry = "CREATE TABLE sitemap ( id varchar(30), price decimal(6,2), url varchar( 1024 ) )";

// Create the table
mysql_query ( $qry, $con );

$numSitemapPages = 350;
$html = new simple_html_dom();

if($_ECHO) echo "START: Fetching site map...<br />";

for( $i = 0; $i < $numSitemapPages; $i++ )
{
if($_ECHO) echo "Page $i<br />";

$fileContents = file_get_contents( "http://www.website.co.uk/SiteMap-S" . $i . ".aspx" );
$html->load( $fileContents );

$hrefs = $html->find( "a[style=color: Blue; text-decoration: underline;]" );

if ( isset( $hrefs[ 0 ] ) )
{
foreach( $hrefs as $href )
{
$url = "http://www.website.co.uk/" . $href->href;
$qry = "INSERT INTO sitemap(url) VALUES( '$url' )";

mysql_query( $qry, $con );

if($_ECHO) echo "MYSQL: Added $href->href to DB<br />";
}
}
else
if($_ECHO) echo "NO URLS FOUND ON THIS PAGE!<br />";
}

echo "END: Fetching site map...<br />";

exit(0);

Hello guys, i'm trying to screen scrape the original content from every RSS feed. The RSS feeds works fine however when i try to screen scrape every content using the library simple html dom. At first it works fine but when it tries to extract the second  feed's original content then i get this error:

Fatal error: Cannot redeclare file_get_html() (previously declared in C:\wamp\www\mashup\protected\views\articles\simple_html_dom.php:37) in C:\wamp\www\mashup\protected\views\articles\simple_html_dom.php on line 41
part of my code is as follows:

Code: [Select]
foreach($RSS_DOC->channel->item as $RSSitem)
{

$item_id = md5($RSSitem->title);
$item_title = $RSSitem->title;
$item_date  = date("Y-m-j G:i:s", strtotime($RSSitem->pubDate));
$item_url = $RSSitem->link;

echo "Processing item '" , $item_id , "<br/>";
echo $item_title, " - ";
echo $item_date, "<br/>";
echo $item_url, "<br/>";

//screen scrape original article
include('simple_html_dom.php');
$html = file_get_dom($item_url); 
foreach($html->find('td[class=rel_headline_cmt]') as $element)
{
echo $element;
}
}
Any help with this?

For my site I need to screenscrape a page on a site.
The problem is, to access the page that contains the data I need, I have to login to my account first.
I know there are ways to simulate a form submission with ASP, but my server is Linux and can't use ASP.

I'm wondering if any of you know how I would be able to simulate a POST with something like cURL?  And possibly write an example script?

Thanks in advance. 

(This may be in the wrong section, please move it if it is. Thanks)

I'm just doing a simple scrape on a web page.. works great, except...  the section I scrape is just 10 words inside a table which are updated every few hours. 

Problem is...  my scrape seems to only grab what I found the first time I ran my script, despite the new words being updated a couple hours ago. 

I've tested everything script wise and looked at the html source to be sure everything is showing up... just wondering, is there any thing you can think of that would allow for caching (from source site or the php script) of the words on a page when running my script?  I even opened 2 different browsers and ran the script fresh from these browsers that had never ran it.  When I echo the results, I still get the original result set.

I'm perplexed. 

I need to scrape a Chinese website. (I guess there is no difference between scraping a Chinese website and a normal one?)

It's my suppliers weblink. They have have told me to download images and text from their website profile link off 1688.com.

There is an API - but from what I've read, it's pants + my virus checked doesn't allow me to visit the API doc page.

What tool should I use?

I've got lots of experience coding - but master of none. Maybe I still fall into the newbie category. LOL.

I saw a link from an article... they gave these names:

Goutte
Simple HTML DOM
htmlSQL
cURL
Requests
HTTPful
Buzz
Guzzle

Which should I consider?

IMPORTANT: I need to download and then upload into my Woocommerce website.

I need images + variation details. I'll have a little file with translations, so if XYZ is found in Chinese, then it is replaced with this.

I thought I would mention the extra detail incase it was relevant to considering which scraping tool to pick.

Thanks!

 

Edited July 11, 2019 by acebase

Hi all, I'm trying to scrape the contents of a page that is behind a login screen; namely: http://my.mail.ru/apps. Here's my code. It almost works, but doesn't appear to be properly logging in -- I just get a login screen on the url download. Any ideas? Thanks much.

Here's my code



<?php
 
$ch=login();
$html=downloadUrl('http://my.mail.ru/apps', $ch);
echo $html;
  
function downloadUrl($Url, $ch){
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_REFERER, "http://my.mail.ru/cgi-bin/login?noclear=1&page=http%3a%2f%2fmy.mail.ru%2fapps%2f");
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$output = curl_exec($ch);
return $output;
}
 
 
 
function login(){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'http://my.mail.ru/cgi-bin/login?noclear=1&page=http%3a%2f%2fmy.mail.ru%2fapps%2f'); //login URL
    curl_setopt ($ch, CURLOPT_POST, 1);
$postData='
                page=http%3A%2F%2Fmy.mail.ru%2Fapps%2F
                &Login=username
                &Domain=mail.ru
    &Password=password';
    curl_setopt ($ch, CURLOPT_POSTFIELDS, $postData);
    curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
                curl_setopt ($ch, CURLOPT_FOLLOWLOCATION,1);
                curl_setopt ($ch, CURLOPT_MAXREDIRS, 10);  
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    $store = curl_exec ($ch);
    return $ch;
}


?>

Hi everyone,

I am making a screen scraper in php which scrapes the usernames from forum posts and stores them in an SQL database.  I need some help with part of the preg_match code if possible please? 

The code and also the pseudo code I have so far is:(the pseudocode I am having trouble with but will try to solve my self if possible).
Edit: sorry, editted the page as was confusing to read.  Please ask for clarificaiton if there is anything I have failed to explain properly. thank you.

//I will be placing the following php in the confirmation page people see after making a new post, so for this example lets say the referrer header says: http://www.mysite.com/showthread.php?tid=1'

Code: [Select]
$threadurl=$_SERVER['HTTP_REFERER'];
// scrape the page
Code: [Select]
$content = file_get_contents($threadurl);

// find the pattern in source which makes it easy to find the username- the only things that change are the uid and the color
Code: [Select]
if (preg_match("/\b uid=792"><span style="color:#ffcc00">fapafap</span></a>\b /i", $content)) {
//extract username from this string search
Don't know!

//copy the username (in this case 'fapafap') to the database along with the referral ID

Code: [Select]
$query_insert="INSERT INTO newpostersdatabase(username,referrerurl) VALUES('$username','$threadurl')" ;
$result=mysql_query ( $query_insert);
if(!$result){
die(mysql_error());
}

Thank you so much for any guidance, I know that I have totally messed up with the string search also but my brain is too small it seems!

Alright, so I play a browser game called Politics and War. I run an alliance that has 74 members. In that alliance we offer a bank service for all our members, but I - being the leader - am the only one who can access the bank. I have been building a site that works with the game API to gather data for members and create a dashboard. One of the features I am trying to build is allowing them to withdraw from their account instantly.

 

So, what I need:

To be able to submit a POST request to login to the site (specifically on this page --> https://politicsandwar.com/login) with my username and password, but then I need to keep the session active and navigate to a different page (the alliance bank page). On that page I first need to scrape a value from a hidden input (token) and then I need to submit a POST request to this same page while still being logged in. 

 

I am not asking someone to do it for me, but rather someone to help me know how to go about this. I have never submitted post requests with PHP, but I have used PHP cURL in the past. I also have made POST requests with JS, but never PHP.

 

Thank you so much for anyone that is able to help!

require_once 'phpSimpleHtmlDomClass.php'; $html = '<div> <div class="man">Name: madac</div> <div class="man">Age: 18 <div class="man">Class: 12</div> </div>' $name=$html->find('div[class="man"]', 0)->innertext; $age=$html->find('div[class="man"]', 1)->innertext; $cls=$html->find('div[class="man"]', 2)->innertext;

wanna get a text from each div class="man" but it didn't work because there is a missing closing div tag on 2nd line of html code. please help me to fix this.

thanks in advance.