PHP - Collecting Data From Another Website

Hey,


Im busy making a social network style wall feed for my site where people make a status and people can comment.


Now when some one comments the users active in the comment wall + original status maker gets notified but now the issue is theres so many while loops going on im thinking there must be a better way?


My current method is as follows:

1) Get the userid of the person who made the status
2) Collect all unique userid's of all the people who made a comment
3) Create the array of userids
4) Loop the array and insert row after row of each userid to notify them a comment was made on a feed they are active on.


Thing is - im thinking when i scale it up with alot of users this looping is going to increase the load alot so is there really any more efficient method i can use ?

My website needs to collect credit card information WITHOUT 3rd party!   I asked godaddy what i is i need for that and they told me if i get a merchandising account with my bank and call them to tell them what i want to do they will send me a script and i use that script on my site.     Does anyone have experience with this?   my server is PCI compliant

Hi there, hope one of you can help me with a problem im just having.

Ok, lets start with the explanation what i want to to:
I'd like to collect headlines from a html site, and get the test results in an array, so that the array structure represents the dom-levels of the site.

Small Example:
<h1>test1.1<h1/>
<h2>test2.1</h2>
<h2>test2.2</h2>
<h3>test3</h3>
<h1>test1.2<h1/>
Shoul end in a array structure like:

Code: [Select]
array('level 1' => array (
            'sibble1' => array (
                'headline' => 'test1.1',
                'level2' => array(
                    'sibble1' => array (
                        'headline' => 'test2.1',
                        'level3' => array(), // empty data, needs to be processed anyway to find gaps, to maybe a h4 headline would be existing
                    ),
                    'sibble2' => array (
                        'headline' => 'test2.2',
                        'level3' => array (
                            'sibble1' => array (
                                'headline' => 'test3.1',
                                'level4' => array(), // empty data, needs to be processed anyway to find gaps, to maybe a h4 headline would be existing
                            ),
                        ),
                    ),
                ),
            ),
            'sibble2' => array (
                'headline' => 'test1.1',
                'level2' => array(),
             ),
        ),
);

So i hope out of this example you can see what i want to do. level represents the healdine level 1-9, sibblin is as name for the childs on the headline level.

Ok, so to extract the herefore needed data out of the html, i build a class with a recursive function, that filters the html by a regex from one headline to the next, first iteratin all childs, if there are no more childs i go tho ne nextsibbling element.

as an running example code look he

Code: [Select]
<?php

class Application_Model_DomParser {

    const PREG_MATCH_ONE_ELEMENT_TO_MUCH_LENGTH = 4;

    private $slicedFields = array();

    public function __construct() {
    }

    public function sliceFirstSectionToDataFields($level = 1, $haystack) {

        preg_match_all("#(.*?)<h$level>(.*?)</h$level>(.*?)(<h$level>|$)#s", $haystack, $data);

        // prepare the chunkData
        $pageText = '';
        if (isset($data[1][0])) {
            $pageText = $data[1][0];
        }
        $headline = '';
        if (isset($data[2][0])) {
            $headline = $data[2][0];
        }
        $dataToProcessNextLevel ='';
        if (isset($data[3][0])) {
            $dataToProcessNextLevel = $data[3][0];
        }
        // @todo dirty warnings compression, search why warning occures
        @$posOfNextChild = strlen($data[0][0]) - self::PREG_MATCH_ONE_ELEMENT_TO_MUCH_LENGTH;


// from here goes the debug...
echo $headline ."::" .strlen($dataToProcessNextLevel). "<br>";


        if (strlen($dataToProcessNextLevel) <= self::PREG_MATCH_ONE_ELEMENT_TO_MUCH_LENGTH) {
            if ($level < 9) {
                $dataToProcessNextLevel = $haystack;
            }
            else {
                return;
            }
        }
        //recursive check for next level in $dataToProcessNextLevel
        $nextLevel = $level + 1;
        $this->sliceFirstSectionToDataFields($nextLevel, $dataToProcessNextLevel);


        $haystack = substr($haystack, $posOfNextChild);
        // slized To The End
        if (self::PREG_MATCH_ONE_ELEMENT_TO_MUCH_LENGTH == strlen($haystack)) {
//            die("slized To The End");
            return;
        }

        // recursive check for other childs in actuall level...
        $this->sliceFirstSectionToDataFields($level, $haystack);
    }
}


$htmlString = 'test<h1>headkline1.1</h1>
<p>test test</p>
<h2>headline 2.1</h2>test test
<h2>headline 2.2</h2>
<p>tes test test</p>
<h3>headline 3.1</h3>test
<h3>headline 3.2</h3>test
<h2>headline 2.3</h2>
<p>&nbsp;</p>
<p>test</p>
<h1>headline 1.2</h1>
<h2>headline 2.4</h2>
<p>11111111111112222222222222222222222</p>
<p>&nbsp;ewfwrefg upowmdg w3q09umq09wrt n3q089ty 3q0898943ty -98 41</p>
<h3>headline 3.3</h3>
<p>test</p>
<p>test</p>
<p>test</p>
<h1>1.3 testtest</h1>
<p>test</p>
<h3>head 3.4</h3>
<p>sadfsadfsadf asdfsda f sdaas saf saddas</p>
<h3>head 3.4</h3>
<h3>test 1.3</h3>
<p>test;</p>
';

$model = new Application_Model_DomParser();
$result = $model->sliceFirstSectionToDataFields(1, $this->htmlString);


So letting this piece of code run, you can se via the debug echo, every headline is found, even in the right order of its occurence.

My problem now is dont get it how to return the extracted values, and collect them to get a result as my above shown structure shows.

So i spent ours to solve this problem but didnt come to a result.

I know its a hard problem, and to help me takes time, cause its a complex situation.

Allthough i hope someone knows a anser. I need this problem solved, to get my mind rested!

Thanks, and greetings
Selma

I am wondering if it's possible to get some data from another website via PHP? I would like to get data from website http://www.gamersfirst.com/warrock/?q=Player&nickname=soldier, and the exact data which I need is "Level" which is in this case 2. Can this value be grabbed and if it can, where can I get some info about how to do it?

It's very possible, and I've seen a website that's done it (http://www.zybez.net/radio/) - They are able to include the recent songs played on there website. The information can be obtained from right he http://68.168.100.60:7942/played.html

But I'm not sure if thats where they get it from. If I login to my Admin section, there will be an option that says get XML stats. But I'm not sure if that shows the recent songs played like it would at http://68.168.100.60:7942/played.html

(Everytime I refresh the page, the stats/data/info in the XML change)

Code: [Select]
1250001522Rockhttp://srbuckey.listen2myradio.comMusicPEAK - RockKings of Leon - Radioactive N/AN/AN/A226153140audio/mpeg1.9.86201901701119000000004013207.144.125.60NSPlayer/11.0.6001.7006 WMFSDK/11.0026101521287587284Kings of Leon - Radioactive 1287586919Three Days Grace - Riot 1287586542Alice In Chains - Here Comes The Rooster 1287586281Green Day - Boulevard of Broken Dreams 1287586277down 1287586075Green Day - American Idiot 1287586044Metallica - Enter Sandman 1287585836ACDC Highway To Hell.mp3 1287585449Metallica - 04 The Unforgiven I.MP3 1287585448[HTTP/1.1 200 OK] http://www.ventsi.com/Music/MetallicA/1991 - Black Album/Metallica - 04 The Unforgiven I.MP3

Hello!

I would like to grab data from a website. What I want is to take the name and time and output it like following:

 

Ayarith 59 seconds

Indirarc 54 seconds

 

The data is taken from an online game with the following link: 

https://medivia.online/community/online/legacy

 

Exemple data from this link:

<li><div class="med-width-25">59 seconds ago</div><div class="med-width-35"><a href="/community/character/Ayarith">Ayarith</a></div><div class="med-width-15">Druid</div><div class="med-width-25 med-text-right med-pr-40">32</div></li>

<li><div class="med-width-25">54 seconds ago</div><div class="med-width-35"><a href="/community/character/Indirarc">Indirarc</a></div><div class="med-width-15">Druid</div><div class="med-width-25 med-text-right med-pr-40">20</div></li>

 

I hope you can help me out!

OK, I have the initial cURL working but need to figure out how to extract data I want off that webpage to display or store in a database, I tried using dom and xpath, but because of the way the page displays using css, i think its not picking it up. Here is my cURL script:
<?php
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

$target_url = "www.test.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html = curl_exec($ch);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
// parse the html into a DOMDocument
$dom = new DOMDocument();
$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//td");

for ($i = 0; $i < $hrefs->length; $i++) {
$href = $hrefs->item($i);
$url = $href->getAttribute('href');
storeLink($url,$target_url);
echo "<br />Link stored: $url";
}

?>
and here is a snippet of the source of the page I am getting:

<span id="lblTest"><h1 id='surrZipTitle'>Agents in Surrounding Zip Codes</h1><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>4444 MANZANITA AVE STE 6</span><br /><span>CARMICHAEL                    , CA 95608-1488</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415199904150295&lat=38.646142&lng=-121.327623' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a> &nbsp;&nbsp; <a class='faaBlueLink' id='lnkWebSite' style='display: none;' href='http://' target='_blank' onclick="return trackEvent('/External-Link/AgentWebsite/                                                                      ','PROFESSIONAL INS ASSOC, INC.                                ');">Website</a></div></td><td valign='top'>&nbsp;</td></tr></table></td></tr></table><table cellpadding='0' cellspacing='0' border='0' class='tblDent'><tr><td class='tdEliteTitle'><span class='caaSubHead3 addwidth'>H.K. Dent Elite</span></td></tr><tr><td class='tdEliteContent'><table cellpadding='0' cellspacing='0' border='0'><tr><td valign='top'><span class='caaAgencyName2 addwidth'>AMERICAN AIM AUTO INS AGY, INC</span></td><td valign='top'>&nbsp;</td></tr></table><table cellpadding='0' cellspacing='0' border='0'><tr><td width='360px' valign='top'><div class='addressBlock'><span>5339 SAN JUAN AVE</span><br /><span>FAIR OAKS                     , CA 95628-3318</span><br /><a class='faaBlueLink' id='lnkContact' href='http://www.safeco.com/portal/server.pt/gateway/PTARGS_0_20656_395_362_0_43/http%3B/por-portlets-prd.int.apps.safeco.com%3B13425/dotcom/FindAnAgent/find-an-agent/contactanagent.aspx?RequestType=agency&level=elite&Id=0415911704151222&lat=38.66237&lng=-121.292429' onclick='oOobj4.Preferences.Plugins.Events.poX=0;'>Contact &amp; Directions</a>

So basically I want to extract the agency name like "<span class='caaAgencyName2 addwidth'>PROFESSIONAL INS ASSOC, INC.</span>" and the address which always use the same div class like "caaAgencyName2" and "addressBlock".  How can this be accomplished?

I have a page (input.php) that will allow a user to upload a CSV file. This file has 5 columns (SKU, Product, Quantity, Retail Price, and Total Retail Price). The CSV upload will only have the SKU number and Quantity filled in.

When the user hit upload the (import.php) page is suppose to go to the site and pull the product up by searching the SKU number and pulling the price and product (brand and title).

I paid a freelancer to create this code. I watched it work on his machine. I cant seem to get it to work on mine (wont pull price or product) and he is non-responsive now. Any help would be greatly appreciated!!

I added some note in the code as I was troubleshooting.

 

<?php
ini_set('max_execution_time', 0);
error_reporting(0);
move_uploaded_file($_FILES["file"]["tmp_name"], "upload/". $_FILES["file"]["name"]);
$handle = fopen("upload/". $_FILES["file"]["name"], "r");
$file = '';
$line .= "SKU,Product,Quantity,Retail Price,Total Retail Price";
$file .=  $line . PHP_EOL;
for ($i = 0; $row = fgetcsv($handle ); ++$i) {
    // Do something will $row array
	if($row[0]!="" AND $i>0)
	{
		$line="";
		#echo "<pre>";
		#print_r($row);
		$SKU=$row[0];
		$quantity=$row[2];
		$loop=1;
		do{
			$url = "https://www.homedepot.com/s/".$SKU;
			$ch = curl_init();
			curl_setopt($ch, CURLOPT_URL, $url);
			curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
			curl_setopt($ch, CURLOPT_HEADER, 1);
			$response = curl_exec($ch);
			
			$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
			$headers = substr($response, 0, $header_size);
			$body = substr($response, $header_size);
			curl_close($ch);
			
			header("Content-Type:text/plain; charset=UTF-8");
			$headers_arr = explode("\r\n", $headers);
			$str=$headers_arr[5];
			$arr=explode(":",$str);
			$check=trim($arr[0]);
			#echo $check; ### remove troubleshooting
			if($check=="location")  # made lowercase so it would get inside the If statement
			{
				#echo "Dustin"; ## remove troubleshooting
				$productPageLink=$headers_arr[5];
				$productPageLink=str_replace("Location:","",$productPageLink);
				#echo $productPageLink; ## troubleshooting -- seems to be getting the links
				$productPageLink=trim($productPageLink);
				$productPageLink=str_replace("http:","https:",$productPageLink);
				#echo $productPageLink; ## troubleshooting -- still seems to have links
				
				$ch = curl_init();	
				#echo $ch; ##troubleshooting -- prints out "resouce id"
				curl_setopt($ch, CURLOPT_URL, $productPageLink);
				#echo $ch; ##troubleshooting -- prints out "resouce id"
				curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
				curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');		
				curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');		
				$headers = array(); 
				$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0';
				$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
				$headers[] = 'Accept-Language: en-US,en;q=0.5';
				$headers[] = 'Upgrade-Insecure-Requests: 1';
				$headers[] = 'Connection: keep-alive';
				$headers[] = 'Te: Trailers';
				#echo $headers; ##troubleshooting -- ## Troubleshooting -- prints out "Array"
				curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
				
				$result = curl_exec($ch);
				#echo $result; ### troubleshooting - doesnt have any data
				if (curl_errno($ch)) {
					echo 'Error:' . curl_error($ch);  # it end inside this if statement, however no error is printed
				}
				curl_close($ch);
				preg_match_all('/<h2 class="product-title__brand" itemprop="brand" data-component="clickable brand link">(.*?)<\/h2>/s', $result, $output_array_brand);
				#echo "<pre>"; ###
				#print_r($output_array_brand);####
				
				$brand=trim(strip_tags($output_array_brand[1][0]));
				
				preg_match_all('/<h1 class="product-title__title">(.*)<\/h1>/', $result, $output_array);
				$productTitle=$output_array[1][0];
				$productTitle=$brand." ".$productTitle;
				
				preg_match_all('/<span class="price__dollars">(.*?)<\/span>/s', $result, $output_array_price);
				preg_match_all('/<span class="price__cents">(.*)<\/span>/', $result, $output_array_cent);
				#echo "<pre>";
				#print_r($output_array_price);
				$price=trim(strip_tags($output_array_price[1][0]));
				$cent=trim(strip_tags($output_array_cent[1][0]));
				if($cent!="" OR $cent!=0)
				{
					$price=$price.".".$cent;
				}
				$line.=$row[0].",";
				$line.='"'.$productTitle.'",';
				$line.=$row[2].",";
				$line.=$price.",";
				$totalPrice=$row[2]*$price;
				$line.=$totalPrice;
				$file .=  $line . PHP_EOL;   
			}
				
			# echo "<br>";
		 $loop=$loop+1;
		 #echo "<br>";
		 if($loop>4)
		 {
			 if($check!="Location")
			 {
				$line.=$row[0].",";
				$line.=',';
				$line.=$row[2].",";
				$line.=",";
				$line.="";
				$file .=  $line . PHP_EOL;   
				break;
			 }
		 }
		}
		while($check!="Location");
		
				
	}
}
fclose($handle);
header('Content-Type: application/csv'); 
$output=$_REQUEST['output'];
		header('Content-disposition: attachment; filename='.$output.'.csv');
		
		echo $file;
		#header('Content-disposition: attachment; filename='.$output.'1.csv');
		
		#echo $file1;
		exit;
?>

 

There is a school locator script at https://www.ocps.net/parents/pages/FindaSchool.aspx
When we input any address it returns schools located in that locality.

For example:

Use these details and submit the form

Street Number : 3902
Street Name   : Bobolink
Street Type   : Lane
City         : Orlando


It returns a row having three schools Elementary,Middle and High school
On clicking the more button it takes to the respective school details
So for each school I need the respective school names

Audobon Elementary
Glenridge Middle
Winter Park High

Please suggest some ways to achieve this functionality


Thanks



_________________________________________________ ____________________
http://travelinfo.techserveglobal.com/
http://blog.techserveglobal.com/

Hi,

I am quite new to PHP and to learn more about it I decided I would make a little project to easy my life a little bit

I want to make a site that fetches lunch menus from the restaurants nearby my work. I would just like some help getting pointed in the right direction on how to make this happen

Thanks!

Br,
Niklas

I would like to display weather conditions on my website and store them in mysql database.
I wonder if it is possible to load web page into php for parsing so the required info could be found and used?
Is it doable with php and what functions could you recommend for this functionality?
How can I have program running in the background on the server which would be triggered every so often to perform this task?
Another question is if I want to trigger some action once a week like Tuesday at noon, is there function in php which could be used to check what day of the week and time it is ?
I am new to website design so any help will be greatly appreciated.

Hi,   I've got a "data.txt" file with the following content:

One1:One2:One3:One4:One5:One6
Two1:Two2:Two3:Two4:Two5:Two6
Three1:Three2:Three3:Three4:Three5:Three6

Now I want to be able to take each data and put on a specific location of a htlm code. Each line for its own. So for the first line, it should look something like this:

<html>
<head>
<title></title>
<head>
<body>
<h1>One2</h1>
<h2>One4 some other Text One5</h2>
<img src="One6.jpg">
</body>
</html>

Unforturtunately I don't have a clue how to do that. Can anybody help me out or does someone know a good and easy tutorial?   Thanks a lot

When I use PHP to gain data from another webpage I usually use
file_get_contents

Is there a faster way to retrieve data from webpages faster since it only seems to do around 200 website line retrievals per minute?

Is cURL faster?
Thank you

Hi, I'm trying to retrieve/scrape some information from a website using the class name and the tag name. Below is the example in VB: 

        Dim htmL_cat As HTMLDocument
        Dim objTableL_cat As Object, objDatL_cat As Object, objItemL_cat As Object, objKeyL_cat As Object
        Dim intRowL_cat As Long

        Set htmL_cat = New HTMLDocument

        With CreateObject("MSXML2.XMLHTTP")
            .Open "GET", "http://www.lelong.com.my/Auc/List/BrowseAll.asp", False
            .send
            htmL_cat.body.innerHTML = .responseText
        End With

        With htmL_cat
            Set objTableL_cat = .getElementsByClassName("CatLevel1")   'Find elements with class name first
            For Each objDatL_cat In objTableL_cat
                Set objKeyL_cat = objDatL_cat.getElementsByTagName("a")    'Next, find elements with tag name
                For Each objItemL_cat In objKeyL_cat
                    Sheets("Analytics").Range("E6").Offset(intRowL_cat, 0) = objItemL_cat.innerText
                    intRowL_cat = intRowL_cat + 1
                Next
            Next
        End With

        Set htmL_cat = Nothing
        Set objTableL_cat = Nothing
        Set objKeyL_cat = Nothing

How do I do the same using PHP? Thanks.    

I'm a beginner to PHP and am stuck as to how I can fetch data from a MYSQL database and output that information in html to the website. Currently I get the information from the database using normal php:

$result = mysql_query("SELECT * FROM truffleProducts");

Then I assign each field from the database to a php variable in a while loop:
 

while ($row = mysql_fetch_array($result)) {
$id = $row{'id'};
$name = $row{'name'};
$price = $row{'Price_per_kg'};
$season = $row{'Season'};
$country = $row{'Country'};
$image = $row{'image'};
$review = $row{'review'};
}

Then I just print each variable in a table using simple html and php. My problem is that it only outputs the latest element or row from my database. I want to output the database within a HTML template so that it is professional and tidy, but i cant figure out how to do that. Any help?
Edited by louiscb, 22 October 2014 - 07:58 AM.