|
PHP - Curl To Get The Contents Of A Remote Web Page Into A Php Variable
Hi dear community i want to run a Curl to get the contents of a remote web page into a PHP variable <?php // // The PHP curl module supports the received page to be returned in a variable // if told. // $ch = curl_init(); curl_setopt($ch, CURLOPT_URL,"http://www.myurl.com/"); curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); $result=curl_exec ($ch); curl_close ($ch); ?> I declare this variable:($ch) Well - can i work with this variable - eg. to parse this with a parser!? look forward to hear from you $result=curl_exec ($ch); gretings Similar TutorialsHi all, I am new here and I really thank God that I found this forum! I have read the materials here on working with cURL on javascript-form-submission pages. However, I can't to get my script to work. Can anybody here please help me out or drop me a hint on where to correct my script? ==Situation== My company utilizes http://www.myfax.com/free/ to send our company faxes. My task is to write a code that would submit files for faxes electronically. Note: The site also requires e-mail confirmation but I haven't get to that stage yet. I have run tests on submitting fax requests both by code and manually through the site, and have confirmed that the code doesn't work on the submission level because I managed to receive confirmation e-mails for manual submissions. Also, I tried my script with different staff email addresses because I figured out that it blocks only the same e-mail address from sending more than 2 faxes a day. Code: (php) [Select] <?php //target page url $strPage_url = 'www.myfax.com/free/'; //create array of data to be posted $arrPost_data = array ( 'ctl00$MainSection$tbRecipientName' => 'I am recipient', //max length = 50 'ctl00$MainSection$tbRecipientCompany' => 'I am recipient company', //max length = 50 'ctl00$MainSection$tbRecipientFaxNumber' => '+1 (206) 202-8273', //recipient fax 'ctl00$MainSection$ddlRecipientCountry' => html_entity_decode ('{"c":{"i":"2","n":"United States","t":"1","s":"US"},"m":{"i":"1","v":"+1 (###) ###-####","d":"","f":"","c":"","r":""}}'), 'ctl00$MainSection$tbSenderName' => 'I am sender', //max length = 50 'ctl00$MainSection$tbSenderCompany' => 'I am sender company', //max length = 50 'ctl00$MainSection$tbSenderEmailAddress' => 'abc@example.com', //email 'ctl00$MainSection$nbAntiSpam$nbAntiSpam_NoBotExtender_ClientState' => '-150', //number drawn from inspecting the packages sent by manual form submission 'ctl00$MainSection$fileUpload' => '@/files/file.pdf', //file 'ctl00$MainSection$tbMessage' => 'hello world', //message '__EVENTTARGET' => '', '__EVENTARGUMENT' => '', '__VIEWSTATEENCRYPTED' => '' ); //visit the page and get cookies $curl_connection = curl_init ($strPage_url); curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 0); curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"); curl_setopt($curl_connection, CURLOPT_REFERER, "http://www.myfax.com/free/"); curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true); curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1); curl_setopt ($curl_connection, CURLOPT_COOKIEJAR, 'CURLCOOKIE'); $strGet_page_contents = curl_exec ($curl_connection); curl_close ($curl_connection); //get page to retrieve view state and event validation if ( preg_match ( '/"__VIEWSTATE"[\s]+?value="([\s\S]+?)"/' , $strGet_page_contents , $arrView_state ) ) { $strView_state = $arrView_state[1]; $arrPost_data['__VIEWSTATE'] = $strView_state; } if ( preg_match ( '/"__EVENTVALIDATION"[\s]+?value="([\s\S]+?)"/' , $strGet_page_contents , $arrEvent_validation ) ) { $strEvent_validation = $arrEvent_validation[1]; $arrPost_data['__EVENTVALIDATION'] = $strEvent_validation; } if ( preg_match ( '/id="ctl00_MainSection_nbAntiSpam_nbAntiSpam_NoBotExtender_ClientState" value="([\s\S]+?)"/' , $strGet_page_contents , $arrAnti_spam ) ) { $strAnti_spam = $arrAnti_spam[1]; $arrPost_data['ctl00$MainSection$nbAntiSpam$nbAntiSpam_NoBotExtender_ClientState'] = $strAnti_spam; } //traverse array and prepare data for posting (key1=value1) foreach ( $arrPost_data as $key => $value) { $arrPost_items[] = $key . '=' . $value; } //create the final string to be posted using implode() $strPost_string = implode ('&', $arrPost_items); //create cURL connection $curl_connection = curl_init($strPage_url); //set options curl_setopt ($curl_connection, CURLOPT_POST, 1); curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 0); curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"); curl_setopt($curl_connection, CURLOPT_REFERER, "http://www.myfax.com/free/"); curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true); curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1); //set cookie curl_setopt ($curl_connection, CURLOPT_COOKIEFILE, 'CURLCOOKIE'); unlink ( 'CURLCOOKIE' ); curl_setopt($curl_connection, CURLOPT_COOKIE, session_name() . '=' . session_id()); //set header $arrHeaders = array ( 'Content-Type' => 'application/x-www-form-urlencoded; charset=utf-8' ); curl_setopt($curl_connection, CURLOPT_HTTPHEADER, $arrHeaders ); //set data to be posted curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $strPost_string); //perform our request $strResult = curl_exec($curl_connection); //show information regarding the request - for debugging echo "<pre>"; print_r(curl_getinfo($curl_connection)); echo curl_errno($curl_connection) . '-' . curl_error($curl_connection); echo "<hr>"; var_dump ($arrPost_items); echo "</pre>"; //close the connection curl_close($curl_connection); ?> Hey guys i'm using the following code to POST data to the HTTPS url given below. But I m unable to perform a remote login and access the rest of the website. Can u find out the flaws in the code snippet and rectify it ? Or suggest a new code snippet. Also can anyone suggest how to open/read the contents of the login restricted pages on a successful login . $fullurl = "https://premium.rpnet.biz/login.php/"; $postFields="username=&password=&cookieval=on&login="; $username=""; $password=""; $ch = curl_init(); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch, CURLOPT_VERBOSE, 1); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); curl_setopt($ch, CURLOPT_FAILONERROR, 0); curl_setopt($ch, CURLOPT_USERPWD, "$username:$password"); curl_setopt($ch, CURLOPT_POSTFIELDS,$postFields); curl_setopt($ch, CURLOPT_URL, $fullurl); curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt"); # The name of the file containing the cookie data. curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt"); $returned = curl_exec($ch); curl_close ($ch); var_dump($returned); --------------------------------------------------------------- Output: HTTP/1.1 200 OK Date: Fri, 05 Nov 2010 16:27:58 GMT Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.3.3 Set-Cookie: PHPSESSID=nthfd25gn6vpm89k18fpgamgc6; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 6336 Connection: close Content-Type: text/html; charset=UTF-8 ================================================ If I change the url to https://premium.rpnet.biz/usercp.php/ then the output is HTTP/1.1 302 Found Date: Fri, 05 Nov 2010 16:28:00 GMT Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.3.3 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Set-Cookie: lo[uname]=deleted; expires=Thu, 05-Nov-2009 16:27:59 GMT; path=/; domain=premium.rpnet.biz Set-Cookie: lo[pass]=deleted; expires=Thu, 05-Nov-2009 16:27:59 GMT; path=/; domain=premium.rpnet.biz Location: https://premium.rpnet.biz/login.php Content-Length: 0 Connection: close Content-Type: text/html; charset=UTF-8 bool(true) I am setting up an API for my users, and the user sends post data via cURL. Is there a way I can see what site that the post data is coming from? would $_SERVER['REMOTE_HOST'] work? I am not using that to validate information. Hi, I'm trying to auto login to a website(created by me ;-) ) using the curl. as I am new to this I don't know how to make this possible. following is the code I tried but this is not submitting the data in the other site. the 'usr_name' and 'password' the field names in the page "http://localhost/myproject/users/login". and I have given a print_r in that site and it is displaying Array ( [loginType] => L [step] => confirmation [usr_name] => dasd@hotmail.com [password] => test123 ) but not submitting the form. please help me.... this is the code i've tried..I got this from web... $login = "http://localhost/myproject/users/login"; $param="loginType=L&step=confirmation&usr_name=dasd@hotmail.com&password=test123"; $c = curl_init(); curl_setopt($c, CURLOPT_URL, $login); curl_setopt($c, CURLOPT_COOKIEJAR, "cookies.txt"); curl_setopt($c, CURLOPT_COOKIEFILE, "cookies.txt"); curl_setopt($c, CURLOPT_POST, 1); curl_setopt($c, CURLOPT_POSTFIELDS, $param); curl_setopt($c, CURLOPT_RETURNTRANSFER, 1); curl_setopt($c, CURLOPT_FOLLOWLOCATION, 1); echo curl_exec($c); thanks in advance.... Hey All, I will be marketing a simple php script shortly and would like to Obfuscate my code and have it check to see if the person is running the script on the proper domain, much like other Purchased Scripts, you have to buy licenses and some are limited to a Per Domain basis. Could I use cURL to run a script on another server (mine) that takes the URL it came from, and a variable "the auth key" and check if the person is using the script on the proper domain? If the domain and auth key don't match in the DB i would return a message saying invalid key or something, else continue to execute the script. <?php //authkey would be defined in config file $authkey = "769870afhljkzxf90436"; $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,"http://example.com/page.php?authkey=$authkey"); curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2); curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1); $buffer = curl_exec($curl_handle); curl_close($curl_handle); if (empty($buffer)) { print "Sorry, AuthServ is performing maintenance.....<p>"; } else { print $buffer; } ?> Is there a smarter way to do this? I don't have an extra 300 bucks to use something like ioncube, just looking for a general proper direction on going about this as I am totally clueless. thanks Hi guys, I am making a bot which only scrapes the source code of the site AFTER logging into the site.The script to login is : Code: [Select] <?php $username="xxx"; $password="iwonttellyou"; $url="http://internet.com/login.php"; $cookie="cookie.txt"; $postdata = "name=".$username."&password=".$password; $ch = curl_init(); curl_setopt ($ch, CURLOPT_URL, $url); curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); curl_setopt ($ch, CURLOPT_TIMEOUT, 60); curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 0); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie); curl_setopt ($ch, CURLOPT_REFERER, $url); curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata); curl_setopt ($ch, CURLOPT_POST, 1); $result = curl_exec ($ch); echo $result; ?> I can see different SESSION ID's in cookie.txt everytime i compile this code, which makes me believe its working.However what next? How should i go to that site again, already logged in and scrape the data ? Some suggestions would be nice. I have seen a couple of other threads relatively similar but I still find my problem unique so I hope you guys give me a chance.
What I want to do: use proxy to bypass same origin policy and to save these values (that changes and need to be checked by timer) into variables that can be modified.
A server contains data.asp which looks something like this and updates from time to time (although with a bit more data):
{ "Title":"Tile of song" ,"Artist":"The artist" ,"Album":"The album" ,"CDCover":"/covers/randomcover.jpg" }
The method I have tried is like the one on http://qnimate.com/s...Policy_for_AJAX. In case you guys don't want to follow the link I've posted the data here as well.
<script>
function loadXMLDoc()
{
var xmlhttp;
if (window.XMLHttpRequest)
{
xmlhttp=new XMLHttpRequest();
}
else
{
xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
}
xmlhttp.onreadystatechange=function()
{
if (xmlhttp.readyState==4 && xmlhttp.status==200)
{
document.getElementById("myDiv").innerHTML=xmlhttp.responseText;
}
}
xmlhttp.open("GET","http://www.qnimate.com/ajaxdata.php",true);
xmlhttp.send();
}
</script>
</head>
<body>
<button type="button" onclick="loadXMLDoc()">Request data</button>
<div id="myDiv"></div>
And the proxy file:
<?php
echo file_get_contents('http://www.qscutter.com/ajaxdata.txt');
?>
So what I want to do is store the data into variables that I can alter, not show it inside a div using innerHTML. Preferably check this ever so often for changes as well.
Suggestions?
I can add that I have the permission from the server owner to retrieve the information and to use it but no possibility to add script to their server, hence the proxy bypass attempt.
Hi i dont know whether i've overlooked something but im finding it hard to answer this myself.. Background of the problem: Im keeping the contents of a page in a database, so each page would be a template, and the body text will be stored in the database. So if i want to call up the contents for my 'About' page, i would use the php command: "<?php getAboutPosts(); ?>" The code behind that will be: function getAboutPosts() { $query = mysql_query("SELECT * FROM posts WHERE id = '20' ") or die (mysql_error()); while($subm = mysql_fetch_assoc($query)) { echo "<h2>" . $subm['Title'] . " by iVisual Media" . "</h2>"; echo $subm['Content']; } } My Actual Problem The text does echo fine, but it is completely unstructured, I.E. it isn't paragraphed, i cant include any type of tag like <b> because it doesn't seem to echo them, although when i check PHPMyAdmin it has all the tags saved in with the content? Example: I have this saved in MySQL Database: <h4>We are a professional website and graphic design business in Basildon, Essex, providing online marketing strategies for companies throughout the UK.</h4> <p><br /> Whether you require a brochure website, blog or a full e-commerce solution, Imagine Design Studio can help. Maybe you have a new business and want to create an online presence or an established business looking to pull your company into the 21st century.</p> <p> But when it echos, i get this: We are a professional website and graphic design business in Basildon, Essex, providing online marketing strategies for companies throughout the UK. Whether you require a brochure website, blog or a full e-commerce solution, Imagine Design Studio can help. Maybe you have a new business and want to create an online presence or an established business looking to pull your company into the 21st century. Things i've tried: Adding a class to the DIV which the echo will go into - This works for colouring etc, but i still need to paragraph the text and use italics occasionally. Please help guys! Thanks in advance Hi All, I'm having a major mind blank, and can't find anything in the previous posts resolving what I'm after. I'm setting $searchtext = $_POST['searchtext']; I want to check $seachtext is not null. I've seen isset($searchtext) but it doesn't solve my problem. Basiclaly; I want an if statement to say if(isset($searchtext)) {......} Thoughts? I am attempting to us glob to display contents of a users folder using a session variable. Example: I have a session variable called department Code: [Select] $row_fullname['department']; In department I have the name of the department the user belongs to such as: office, plant, maintenance, and groundskeeping I created a folder called docs inside of docs there are 4 subfolders called office, plant, maintenance, and groundskeeping I found this code which will display the contents of the folder: Code: [Select] <?php $files = glob( './docs/office/*.*' ); foreach ( $files as $file ) { echo '<a href="./docs/office/' . basename( $file ) . '"target="_blank">' . basename( $file ) . '</a><br />'; } ?> The above code works fine, but I would like it to only display the contents of a departments folder only if the user is part on that department. Here is an example that I know is completely wrong but it may help explain what I am trying to do. Code: [Select] <?php ]<?php $files = glob( './docs/echo $row_fullname['department'];/*.*' ); foreach ( $files as $file ) { echo '<a href="./docs/echo $row_fullname['department'];/' . basename( $file ) . '"target="_blank">' . basename( $file ) . '</a><br />'; } ?>Thanks for your time Hi I can get this to work can you point me in the right direction curl_setopt( $ch, CURLOPT_URL, "http://website.com/action=%22getcontent%22%20shipid=%22\"$content\"%22%20/%3E%3C/request%3E"); I want to include the variable $content in the url thanks hi guys I just could not find the answer on forum. I have a url like this http://www.example.com/deal.php?id=367 and my question is : how can i use curl function to get the html and the id value which is 367 thanks I have been searching for hours to try and figure out how I could print a remote page using PHP. I know in javascript, you can print the current page with: Code: [Select] <a href="javascript:window.print()">Print Page</a> But what I want to do is "javascript:window.print(http://some-link.com)", but would be nice if I could do it using PHP, if it's even possible. I'm a new PHP users needing some help. I've got two code snippets that run a CURL post to a syslog server grabbing specific records that I then parse with regex. Both are similar differing in the specific CURL URL and Regex. One works does not. The failing program seems to fail the regex parse creating a null array. What I've done thus far. Tested the regex on regex101 (works fine, using PCRE). Dumped and echoed the CURL created variable and then rechecked against regex101. ( again all good) Created a static variable in the code of the CURL return string (regex works fine). So the CURL is working, the variable string is working appearing as expected and the regex works. But the variable feels like's it null when passed directly from the CURL to the regex. No error logs, but this is where I'm stuck I don't know what to pick at next. Here's a version of the code. Appreciate any thoughts. <?php
$str = curl_exec($ch); // curl steps ends
//$str = '_bkt _cd _indextime _kv _raw _serial _si _sourcetype _time dhcp_type host index linecount source sourcetype splunk_server Configuration initialization for /opt/splunk/etc took 17ms when dispatching a search (search ID: 1591198882.902292) Your search was restricted by ( ( index=nat OR index=network OR index=radius ) OR ( source=/data/syslog/security/ipblocker ) ) base lispy: [ AND 66 82 8a 9c dhcpack e4 f8 sourcetype::infoblox:dhcp [ OR index::nat index::network index::radius source::/data/syslog/security/ipblocker ] ] search context: user="testuser", app="search", bs-pathname="/opt/splunk/etc" Your timerange was substituted based on your search string network~978~07708B5A-37B1-4315-8EFA-70B96D12856C 978:118251396 1591198398 1 Jun 3 11:33:15 128.230.100.36 dhcpd[21353]: DHCPACK on 10.1.0.19 to e4:f8:9c:82:66:8a (ITS-NDD-BOA-T01) via bond0 relay bond0 lease-duration 7200 0 its-splunk-idx2.syr.edu network infoblox:dhcp 2020-06-03 11:33:15.000 EDT DHCPACK 128.230.100.36 network 1 /data/syslog/network/dhcp infoblox:dhcp its-splunk-idx2.syr.edu'; when I send a post variable to another page for processing using cURL and then I echo the variable I am getting extra slashes and periods this is where I set the fields to send. Code: [Select] curl_setopt($ch, CURLOPT_POSTFIELDS,"name='. $name .'&email='. $email .'&serial='. $serialnumber .'"); this is what happens when I echo $serial on the processing page. Code: [Select] \'. SLNM7IV6ZM98YJRXO8D5 .\' it should look like Code: [Select] SLNM7IV6ZM98YJRXO8D5 This is source of a page: http://vn-ace.com/forum/test.txt I want to get contents betweent <blockquote class="postcontent restore">() and </blockquote>() but i dont know how to do ? How can i do ? Please help me. I am newbie. Thanks Hey guys, I'm curious if there's any way to retrieve the contents of a certain div on a page, say I want to get the title of the movie on this page: click (meaning movie title, "Transformers" in this case) Is it possible? Thank you in advance. Hey guys, I'm making an Ajax call to my database and retrieving a newsletters HTML code as the result. I then prepend it into a div area called code_content. I use some php str_replace to add some links also. When the content loads into the div, the user can then add further content to it via Ajax calls, appending articles and that. What I'm trying to figure out is how can I then retrieve all the newly edited contents of this code_content div soni can send it to my Ajax function to write to te db? Hey guys I usually code in C and I started looking into PHP which I think is pretty good. I'm wondering if there's a function that can be used to execute a command from a remote server on my local machine??? As example if I'd like to have a PHP page/script running on my server which when logged onto would display me a page with buttons like "PING, NETSTAT, IPCONFIG, TRACERT" and when clicked on it would exec those commands but not on the server but from my machine??? So that I would get the PING from my local machine to google.com instead of from the server to google.com ??? making any sense ??? thanks Hello, I need to execute a remote web page (distant server) every X seconds. How is it possible and is there any way to make it look exactly like a normal user request through a standard browser like Firefox, Chrome, etc. ? I've heard of cURL, but it seems that servers may block it, so there must be somehow a difference between a usual user request and a cURL execution. Thank you for any help ! :-) Matthew |
|||||||