PHP - Screen Scrape Original Content From Rss

Dear all,
I write this code to extract the widget from this page:http://www.widgetbox.com/widget/accuwidget The widget information is hidden under the tag <iframe> and is inside the src. I try using this code and it always show me error of: Fatal error: Call to undefined method DOMNodeList::getAttribute() 

Code: [Select]
<?php
get();

function get(){
        $url = "http://www.widgetbox.com/widget/accuwidget";

        $tidy = new tidy();
        $repaired = $tidy->repairfile($url);    //The code is dirty, so it need to be tidy
        
        $xml = new DOMDocument();
        $xml->loadHTML($repaired);

        $xpath = new DOMXpath($xml);
        $cloud = $xpath->query("//div[@id='preview-div']/div/iframe");
        $widget = $cloud->getAttribute("src");
        
        echo $widget;
}
?>

Sorry that I didn't input the code of the page i want to scrape the information. It's just that the code is so long.

Thank you all in advance

Hi everyone, I need help with scraping. I have script for scraping IMDB, but I need a few more thing. I dont know how to scrape more Budget, Opening Weekend, and Gross. Rest information that I need I scrape on this way:

Code: [Select]
   //code removed to discourage people from scraping IMDB

Can somebody help me to scrape Budget, Opening Weekend and Gross also?

I'm trying to get a value which changes but is always contained the same way

this is the text Next: Brigadier&nbsp;&nbsp;(81654 of 220000)

as you can see below this is the value i need to get I just can't figure out the RegEx for it? help please!

Smaller CODE
Code: [Select]
<div class="rankBar">
                        <p class="grade"><span id="ctl00_mainContent_identityBar_currentGlobalRankLabel" class="current">Colonel Grade 3</span><span id="ctl00_mainContent_identityBar_nextGlobalRankLabel" class="future">Next: Brigadier&nbsp;&nbsp;(81654 of 220000)</span></p>
                        <div class="rankMeter">
                            <div id="ctl00_mainContent_identityBar_rankBarPanel" class="bar" style="width:37%;">
<span></span>
</div>
                        </div>

FULL CODE

Code: [Select]
<div id="ctl00_mainContent_identityBar_idbarPanel">

        <div class="glowBox idRankContainer">
            <div class="corner bottomLeft"></div>
            <div class="lattice"></div>
            <div class="content">
                <a href="/Stats/Reach/default.aspx?player=l RaH l" id="ctl00_mainContent_identityBar_avatar" class="avatar"><img id="ctl00_mainContent_identityBar_emblemImg" src="/Stats/emblem.ashx?s=70&amp;0=0&amp;1=0&amp;2=0&amp;3=0&amp;fi=0&amp;bi=0&amp;fl=0&amp;m=3" style="height:70px;width:70px;border-width:0px;"></a>
                <div class="idRank">
                    <div class="userInfo">
<img id="ctl00_mainContent_identityBar_namePlateImg" class="img_plate" src="/images/reachstats/nameplates/512.png" alt="" style="height:30px;width:60px;border-width:0px;">
                       
                        <a href="/projects/reach/article.aspx?ucc=faq&amp;cid=28756"><img id="ctl00_mainContent_identityBar_currentGlobalRankImage" title="Colonel Grade 3" class="img_rankIcon" src="/images/reachstats/grades/med/A4BF62C6-1E3F-468A-9A73-8237600B2AD3.png" alt="Colonel Grade 3" style="height:40px;width:40px;border-width:0px;"></a> 
                        <h2><a href="/Stats/Reach/default.aspx?player=l RaH l" id="ctl00_mainContent_identityBar_gamerTag">l RaH l</a> - FLIR</h2>
                       
                    </div>
                    <div class="rankBar">
                        <p class="grade"><span id="ctl00_mainContent_identityBar_currentGlobalRankLabel" class="current">Colonel Grade 3</span><span id="ctl00_mainContent_identityBar_nextGlobalRankLabel" class="future">Next: Brigadier&nbsp;&nbsp;(81654 of 220000)</span></p>
                        <div class="rankMeter">
                            <div id="ctl00_mainContent_identityBar_rankBarPanel" class="bar" style="width:37%;">
<span></span>
</div>
                        </div>
                    </div>
                </div>
            </div>
        </div>
        <div class="glowBox shields">
            <div class="corner topRight"></div>
            <div class="lattice"></div>
            <div class="content">
                <ul>
                    <li class="campaign">
                        <p>Campaign</p>
                        <img id="ctl00_mainContent_identityBar_spCampaignImage" title="Completed Legendary Difficulty" alt="Campaign Progress" src="/images/reachstats/campaign_progress/legendary.png" style="height:59px;width:54px;border-width:0px;">
                    </li>
                    <li class="coop">
                        <p>Co-op</p>
                        <img id="ctl00_mainContent_identityBar_coopCampaignImage" title="Completed Normal Difficulty" alt="Coop Campaign Progress" src="/images/reachstats/campaign_progress/normal.png" style="height:59px;width:54px;border-width:0px;">
                    </li>
                    <li class="arena_display">
                        <p>Arena</p>
                        <a id="ctl00_mainContent_identityBar_arenaLink" href="/stats/reach/careerstats/playlists.aspx?player=l RaH l&amp;vc=2"><img id="ctl00_mainContent_identityBar_currentBestArenaDivisionImg" alt="" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;"></a>
                        <div class="glowBox popOut po_arenaSeason2">
                            <div class="corner topRight"></div>
                            <div class="corner bottomLeft"></div>
                            <div class="content">       
                                <h4>Arena Season 6</h4>

<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>Doubles Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl00_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>
<div class="rule_transparent"></div>
<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>FFA Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl02_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>
<div class="rule_transparent"></div>
<div class="twoColumn">
<img id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_arenaImage" src="/images/reachstats/arena_div/0.png" style="height:47px;width:48px;border-width:0px;">
<h5>Team Arena</h5>
<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_divisionName">Not Qualified Yet</p>

<p id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_lastSeasonControl">Last Season: <img id="ctl00_mainContent_identityBar_arenaRepeater_ctl04_lastSeasonImage" src="/images/reachstats/arena_div/0.png" style="height:23px;width:23px;border-width:0px;"> </p>
</div>

   
                            </div>
                        </div>     
                    </li>
                </ul>
            </div>
        </div>     
</div>

Hi!

I'm into a little project where I want to retrieve data from the swedish news website DN's SOS Live page (http://www.dn.se/nyheter/soslive).
On the page there is an iFrame and there is the data I want to retrieve. The iFrame address is: http://div.dn.se/dn/sos/soslive.php?id= ... er/soslive

Here is the code I have and it stoped working yesterday.

Code: [Select]
function curl_download($Url){
 
    // is cURL installed yet?
    if (!function_exists('curl_init')){
        die('Sorry cURL is not installed!');
    }
 
    // OK cool - then let's create a new cURL resource handle
    $ch = curl_init();
 
    // Now set some options (most are optional)
 
    // Set URL to download
    curl_setopt($ch, CURLOPT_URL, $Url);
 
    // Set a referer
    curl_setopt($ch, CURLOPT_REFERER, "http://www.dn.se");
 
    // User agent
    curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
 
    // Include header in result? (0 = yes, 1 = no)
    curl_setopt($ch, CURLOPT_HEADER, 0);
 
    // Should cURL return or print out the data? (true = return, false = print)
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 
    // Timeout in seconds
    curl_setopt($ch, CURLOPT_TIMEOUT, 10);
 
    // Download the given URL, and return output
    $output = curl_exec($ch);
 
    // Close the cURL resource, and free system resources
    curl_close($ch);
 
    return $output;
}

$sosURL = 'http://div.dn.se/dn/sos/soslive.php?id=p://www.dn.se/nyheter/soslive';
$data = curl_download($sosURL);
The data variable is empty.

I notice now that when I enter the webaddress "http://div.dn.se/dn/sos/soslive.php?id=p://www.dn.se/nyheter/soslive" there is no content, although this is the web address in the iFrame.

How has DN solved this and how could I get around it?

Best regards
Stefan

Hi Guys quite new to php, but getting along roses. I am a big football fan and on my localhost I want to replicate the football fixtures for the week. Its really an learning exercise, but I need projects to learn - struggle to learn from books.
I want to copy this list into a table (eventually):
http://news.bbc.co.uk/sport1/hi/foot...es/default.stm

I have managed to understand how to grab the data, however I think my problem is in the for loops that display the data.
It displays the dates, and the tournament titles, but then it lists all the games - and doesnt get some quite right anyway (look at Stevenage). Then if you scroll down it then just displays the Tournaments.
I think its pretty close, but please can someone point out my mistakes.
Thanks guys:
Code: [Select]
<?Php
$file_string = file_get_contents('http://news.bbc.co.uk/sport1/hi/football/fixtures/default.stm');
preg_match_all('/<div class="mvb"><b>(.*)<\/b><\/div>/i', $file_string, $links);
preg_match_all('/<div class="pvtb"><b>(.*)<\/b><\/div>/i', $file_string, $games);
preg_match_all('/class="stats">(.*)<\/a>/i', $file_string, $teams);
echo '<ol>';
$l = 0;

for($i = 0; $i < count($links[1]); $i++) {
echo '<div>'  . $links[1][$i] . '</div><BR>';

for($j = 0; $j < count($games[1]); $j++) {
echo '<BR><B><U><div>'  . $games[1][$j] . '</div></U></B><BR>';

for($k = $l; $k < count($teams[1]); $k++) {
echo  strip_tags($teams[1][$k]) . '</a><BR>';
$l=$k;
}
}
}
echo '</ol>';
?>
I think the problem is that it doesnt know what order the stuff is supposed to be in. But Im not sure how to write the code to tell it the order. Should each type of preg_match_all be an array or something?

I need to scrape a Chinese website. (I guess there is no difference between scraping a Chinese website and a normal one?)

It's my suppliers weblink. They have have told me to download images and text from their website profile link off 1688.com.

There is an API - but from what I've read, it's pants + my virus checked doesn't allow me to visit the API doc page.

What tool should I use?

I've got lots of experience coding - but master of none. Maybe I still fall into the newbie category. LOL.

I saw a link from an article... they gave these names:

Goutte
Simple HTML DOM
htmlSQL
cURL
Requests
HTTPful
Buzz
Guzzle

Which should I consider?

IMPORTANT: I need to download and then upload into my Woocommerce website.

I need images + variation details. I'll have a little file with translations, so if XYZ is found in Chinese, then it is replaced with this.

I thought I would mention the extra detail incase it was relevant to considering which scraping tool to pick.

Thanks!

 

Edited July 11, 2019 by acebase

I'm just doing a simple scrape on a web page.. works great, except...  the section I scrape is just 10 words inside a table which are updated every few hours. 

Problem is...  my scrape seems to only grab what I found the first time I ran my script, despite the new words being updated a couple hours ago. 

I've tested everything script wise and looked at the html source to be sure everything is showing up... just wondering, is there any thing you can think of that would allow for caching (from source site or the php script) of the words on a page when running my script?  I even opened 2 different browsers and ran the script fresh from these browsers that had never ran it.  When I echo the results, I still get the original result set.

I'm perplexed. 

I usually run scrape on a website and get information and the new ones are usually added automatically but its not adding a new information. I am wondering whethere I have an issue with my code.

I usually run scrape through the sitemap.

Code: [Select]
$qry = "CREATE TABLE sitemap ( id varchar(30), price decimal(6,2), url varchar( 1024 ) )";

// Create the table
mysql_query ( $qry, $con );

$numSitemapPages = 350;
$html = new simple_html_dom();

if($_ECHO) echo "START: Fetching site map...<br />";

for( $i = 0; $i < $numSitemapPages; $i++ )
{
if($_ECHO) echo "Page $i<br />";

$fileContents = file_get_contents( "http://www.website.co.uk/SiteMap-S" . $i . ".aspx" );
$html->load( $fileContents );

$hrefs = $html->find( "a[style=color: Blue; text-decoration: underline;]" );

if ( isset( $hrefs[ 0 ] ) )
{
foreach( $hrefs as $href )
{
$url = "http://www.website.co.uk/" . $href->href;
$qry = "INSERT INTO sitemap(url) VALUES( '$url' )";

mysql_query( $qry, $con );

if($_ECHO) echo "MYSQL: Added $href->href to DB<br />";
}
}
else
if($_ECHO) echo "NO URLS FOUND ON THIS PAGE!<br />";
}

echo "END: Fetching site map...<br />";

exit(0);

Hi everyone,

I am making a screen scraper in php which scrapes the usernames from forum posts and stores them in an SQL database.  I need some help with part of the preg_match code if possible please? 

The code and also the pseudo code I have so far is:(the pseudocode I am having trouble with but will try to solve my self if possible).
Edit: sorry, editted the page as was confusing to read.  Please ask for clarificaiton if there is anything I have failed to explain properly. thank you.

//I will be placing the following php in the confirmation page people see after making a new post, so for this example lets say the referrer header says: http://www.mysite.com/showthread.php?tid=1'

Code: [Select]
$threadurl=$_SERVER['HTTP_REFERER'];
// scrape the page
Code: [Select]
$content = file_get_contents($threadurl);

// find the pattern in source which makes it easy to find the username- the only things that change are the uid and the color
Code: [Select]
if (preg_match("/\b uid=792"><span style="color:#ffcc00">fapafap</span></a>\b /i", $content)) {
//extract username from this string search
Don't know!

//copy the username (in this case 'fapafap') to the database along with the referral ID

Code: [Select]
$query_insert="INSERT INTO newpostersdatabase(username,referrerurl) VALUES('$username','$threadurl')" ;
$result=mysql_query ( $query_insert);
if(!$result){
die(mysql_error());
}

Thank you so much for any guidance, I know that I have totally messed up with the string search also but my brain is too small it seems!

Hi all, I'm trying to scrape the contents of a page that is behind a login screen; namely: http://my.mail.ru/apps. Here's my code. It almost works, but doesn't appear to be properly logging in -- I just get a login screen on the url download. Any ideas? Thanks much.

Here's my code



<?php
 
$ch=login();
$html=downloadUrl('http://my.mail.ru/apps', $ch);
echo $html;
  
function downloadUrl($Url, $ch){
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_REFERER, "http://my.mail.ru/cgi-bin/login?noclear=1&page=http%3a%2f%2fmy.mail.ru%2fapps%2f");
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$output = curl_exec($ch);
return $output;
}
 
 
 
function login(){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'http://my.mail.ru/cgi-bin/login?noclear=1&page=http%3a%2f%2fmy.mail.ru%2fapps%2f'); //login URL
    curl_setopt ($ch, CURLOPT_POST, 1);
$postData='
                page=http%3A%2F%2Fmy.mail.ru%2Fapps%2F
                &Login=username
                &Domain=mail.ru
    &Password=password';
    curl_setopt ($ch, CURLOPT_POSTFIELDS, $postData);
    curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
                curl_setopt ($ch, CURLOPT_FOLLOWLOCATION,1);
                curl_setopt ($ch, CURLOPT_MAXREDIRS, 10);  
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    $store = curl_exec ($ch);
    return $ch;
}


?>

Alright, so I play a browser game called Politics and War. I run an alliance that has 74 members. In that alliance we offer a bank service for all our members, but I - being the leader - am the only one who can access the bank. I have been building a site that works with the game API to gather data for members and create a dashboard. One of the features I am trying to build is allowing them to withdraw from their account instantly.

 

So, what I need:

To be able to submit a POST request to login to the site (specifically on this page --> https://politicsandwar.com/login) with my username and password, but then I need to keep the session active and navigate to a different page (the alliance bank page). On that page I first need to scrape a value from a hidden input (token) and then I need to submit a POST request to this same page while still being logged in. 

 

I am not asking someone to do it for me, but rather someone to help me know how to go about this. I have never submitted post requests with PHP, but I have used PHP cURL in the past. I also have made POST requests with JS, but never PHP.

 

Thank you so much for anyone that is able to help!

by any means is there a way to view a email.php form source? not just the html part but the entire script including the php?

I want people to login and then be redirected back to the page from where they came from.  I was thinking that maybe I would have to use $_GET for 3 pages but I don't know if there is any easier way.

Hello everyone,
I'm trying to extrapolate the original file from a digitally signed.
I know that I have to use the function openssl_pkcs7_verify, but I can't manage to make it works, and I have already made several researches on the reason.

What I discovered is that the problem is because the digital signature that I'm trying to manage is in DER format, while PHP handles only PEM signatures.

Does anybody know if there is a way to convert a DER signature in a PEM signature? Keep in mind that the signature is attached to the file, then the encoding base64 I think is not possible as it would codify also the content of the file, making it unusable.

Alternatively, someone knows if there is a way to manage signatures DER in PHP?

Thanks for your help!