PHP - Spider For Business Directory Data Mining
Hi all,
I need to get all businesses including details from http://www.nswbusinesschamber.com.au/Business-directory.aspx?name=&location= . What is the best way to approach this? Are there any scripts out there? Thanks... Similar TutorialsI am in the process of teaching myself PHP/MySQL. I have taught myself other programming languages in the past so its not that i can't learn. but as i'm new and trying to learn both at the same time, i'm needing some help with php / mysql to datamine either of the following sites. www.findchips.com www.eciaauthorized.com WHat i want to do is after putting in the part number, for each distributor that returns data i want to extract the following data for each distributor Quantity Break Resale at quantity break Current Stock I want to write the data into a field in the database that holds the p/n's and my company's resale. I'm wanting to do this as an exercise so that i can dynamically benchmark my resales against that of my competitors. To get a feel of the data that is returned, please use the following p/n. 1n4001-e3/54 any and all help would be greatly appreciated i am making a search engine and i need help because i need it so you submit your url then it automatically scrolls it for the description and keywords then puts it in the database. my html code: Code: [Select] <form action="submit_url.php" method="get"> <input type="text" name="url" value="url" /> <input type="submit" name="submit" value="submit" /> </form> php code so far: <html> <head> <title>submitting url: <?php $url = $_GET['url']; echo $url; ?></title> <link href="style.css" rel="stylesheet" type="text/css" /> </head> <body> <?php echo "Submitting <b>$url</b>"; $url = fopen("http://knexideas.co.cc", "r") or exit("Unable to open"); while (!feof($url)) { fgetc($url); } fclose($url); ?> </body> </html> all it does is reads the website if you put 'echo $url' at the bottom it just reads and prints the web page. Hi , i've spend some time looking how its possible to spider a phpbb forum with a php script. I'd like to -for example- do a search with the CURL functions , and read out some of the links in the searchresults(topics..). Finally save the links that i want into a mysql database. Somebody got an idea? Hi all ,Its my first post here and I'm still very new to PHP . Im trying to wright a Web crawler script except i want this script to just crawl the 1 target website I enter. Basically i want my script to go to ultimateguitar.com or 911tabs.com or any other guitar tabs website and crawl the site and index any guitar tabs they have in there database. This will provid my website with a "phonebook" of guitar tabs. Its not illeagle or in breach of any copyrights im only making a database of links. Any help would be greatly appreciated! Hey guys, I am making a site where certain content will be limited to "members only", where membership is free. Now I want the google bot or whatever bot to be able to see and index this content, but when a user visits it, I want to hide it from them unless they are a member (I already do that). So basically I want to have a function that I can call that will return true or false if the current page is being requested by a search engine spider. I know it's possible because I regularly see forums doing that; posts are hidden unless you register but if you look through the google cached version, the posts are visible. How can I do that? so far all I have is the following, so that the rest of my code works. function is_spider(){ return true; } I read a http://iarematt.com/how-to-detect-a-search-engine-spidercrawler-with-php/ which talks about this, but I don't really trust it... What do you guys think, How can this be done? Hello, everyone. I'm still kind of new at this, so please bear with me. I've customized a website (hosted by Yahoo Small Business) which is now fully functional aside from the "Contact Us" page. I have a PHP file named "send_email.php" uploaded to my root directory which has the following code: <?php $contact_name = $_POST['name']; $contact_email = $_POST['email']; $contact_subject = $_POST['subject']; $contact_message = $_POST['message']; if( $contact_name == true ) { $sender = $contact_email; $receiver = "my@email.com"; $client_ip = $_SERVER['REMOTE_ADDR']; $email_body = "Name: $contact_name \nEmail: $sender \n\nSubject: $contact_subject \n\nMessage: \n\n$contact_message \n\nIP: $client_ip \n\nFlash Contact Form provided by http://www.flashmo.com"; $extra = "From: $sender\r\n" . "Reply-To: $sender \r\n" . "X-Mailer: PHP/" . phpversion(); if( mail( $receiver, "Flash Contact Form - $contact_subject", $email_body, $extra ) ) { echo "success=yes"; } else { echo "success=no"; } } ?> I went ahead and changed "my@email.com" to my actual email address. The problem I'm having, though, is that the form fails every time...except if I enter my own email address in the "Email" field which should be filled in by the customer. I can't figure this out. It's probably something simple that I'm doing wrong, or perhaps it has something to do with Yahoo Web Hosting. I'm also thinking I have to fill something in the "Client IP" section of the code...but I'm not sure what should go there. I'd really appreciate your time and responses. Thanks so much. I am looking at a website which will have a number of items on it advertised,such as:
businesses
jobs
property
etc
what I want to know is when designing the database would you have these in one table or a table for each?
Edited by brown2005, 14 October 2014 - 08:43 AM. I have a web app that sends out notifications. I set it up to send through Google business, but they have an outbound limit. Does anyone know of a mail service that lets you send an unlimited amount of outbound and actually works good? is there an easy way to add weekdays to a stored date ... so far i have echo date ( 'Y-m-j' , strtotime ( '5 weekdays' ) ); this adds 5 weekdays to the current date , can i have it add 5 weekdays to say $TableDate 1; Thanks in advance... Hi guys, I've been working on a script for a while now, and I'm sure it doesn't look great and all, and it's probably really messed up.. But right now I've finally got it working! There's only 1 thing I'd really like to add.. Searching through & listing of remote directories! The directories I'm trying to list have directory listings enabled, and I think it *should* be possible. I just have no clue how. Here's my current code in a beautiful mix of HTML and PHP: <? $border_size = "0"; function returner($what) { $what=explode("/",$what); $tps=count($what); $what=$what[$tps-1]; return $what; } $page_url= ""; $home_url=returner(__FILE__); if(isset($_GET['q'])) { $qtext=$_GET['q']; } else { $qtext=""; } function getdirsize($directory, $format=FALSE) { $size = 0; if(substr($directory,-1) == '/') { $directory = substr($directory,0,-1); } if(!file_exists($directory) || !is_dir($directory) || !is_readable($directory)) { return -1; } if($handle = opendir($directory)) { while(($file = readdir($handle)) !== false) { $path = $directory.'/'.$file; if($file != '.' && $file != '..') { if(is_file($path)) { $size += filesize($path); } elseif(is_dir($path)) { $handlesize = getdirsize($path); if($handlesize >= 0) { $size += $handlesize; } else { return -1; } } } } closedir($handle); } if($format == TRUE) { if($size / 1048576 > 1) { return round($size / 1048576, 1).' MB'; } elseif($size / 1024 > 1) { return round($size / 1024, 1).' KB'; } else { return round($size, 1).' bytes'; } } else { return $size; } } if(isset($_GET['type'])){ $type=$_GET['type']; } else { $type="new"; } $textures=0; $models=0; $avatars=0; $seqs=0; $sounds=0; foreach (glob("textures/*.jpg") as $texture){ $textures++; } foreach (glob("models/*.zip") as $model){ $models++; } foreach (glob("avatars/*.zip") as $avatar){ $avatars++; } foreach (glob("seqs/*.zip") as $seq){ $seqs++; } foreach (glob("sounds/*.zip") as $sound){ $sounds++; } ?> <!DOCTYPE html> <html> <head> <title>ObjectPath Search</title> <style type="text/css"> #wrapper { width: 850px; margin: 30px auto 30px auto; padding: 10px; } body { color:#C6C6C6; background:#1E1E1E; /* margin:0; padding:0; */ overflow-x:hidden; } #tabs { font: 85% "Trebuchet MS", sans-serif; } .left { float: left; } .right { float: right; } a:link, a:visited, a:active { color: #3DB015; text-decoration: none; } a:hover { color: #00E0FF; } h2 { color: #3DB015; padding-bottom: 0.2em; font-size: 110%; } ul#icon {margin: 0; padding: 0;} ul#icon li {margin: 1px; position: relative; padding: 1px 0; cursor: pointer; float: left; list-style: none;} ul#icon span.ui-icon {float: left; margin: 0 1px;} </style> <link type="text/css" href="http://objects.jk-hosting.com/search/css/black-tie/jquery-ui-1.8.2.custom.css" rel="stylesheet" /> <script type="text/javascript" src="http://objects.jk-hosting.com/search/js/jquery-1.4.2.min.js"></script> <script type="text/javascript" src="http://objects.jk-hosting.com/search/js/jquery-ui-1.8.2.custom.min.js"></script> <script type="text/javascript"> function formHandler(form){ var URL = document.form.site.options[document.form.site.selectedIndex].value; window.location.href = URL; }; $(function(){ // Tabs $('#tabs').tabs(); }); </script> </head> <body> <div id="wrapper"> <div id="tabs"> <!-- Tabs start --> <ul> <li><a href="#tab-search">Search</a></li> <li><a href="#tab-list">List Objects</a></li> <li><a href="#tab-info">OP info</a></li> </ul> <div id="tab-search"><!-- Searchtab start --> Please enter a string to search for, and choose a folder to search in. <br /><br /> <form name="Search"> <input type='hidden' value='search' name='type'> <input value='<? print $qtext; ?>' type='text' name='q'> <select name='map'> <option selected='selected' value='models'>Models</option> <option value='avatars'>Avatars</option> <option value='textures'>Textures</option> <option value='seqs'>Seqs</option> <option value='sounds'>Sounds</option></select> <input type='submit' value='Search'> </form> </div> <!-- Searchtab end --> <div id="tab-list"><!-- Listtab start --> Please pick a folder to browse. <br /><br /> <form name="form"> <select name="site" onChange="javascript:formHandler()"> <option value="#">Look in folder...</option> <option value="<? print $page_url; ?>?type=list&map=models">Models</option> <option value="<? print $page_url; ?>?type=list&map=avatars">Avatars</option> <option value="<? print $page_url; ?>?type=list&map=textures">Textures</option> <option value="<? print $page_url; ?>?type=list&map=seqs">Seqs</option> <option value="<? print $page_url; ?>?type=list&map=sounds">Sounds</option> </select> </form> </div> <!-- Listtab end --> <div id="tab-info"><!-- Info tab start --> The OP currently contains: <br /><br /> <table> <tr><td><b><? echo $models; ?></b></td> <td>Models</td></tr> <tr><td><b><? echo $avatars; ?></b></td> <td>Avatars</td></tr> <tr><td><b><? echo $textures; ?></b></td> <td>Textures</td></tr> <tr><td><b><? echo $seqs; ?></b></td> <td>Seqs</td></tr> <tr><td><b><? echo $sounds; ?></b></td> <td>Sounds</td></tr> </table> </div> <!-- Info tab end --> </div> <!-- Tabs end --> </div> <!-- Start PHP generated content --> <? if($type=="search" || $type=="list") { $M=$_GET['map']; if($type=="search") { $Q=$_GET['q']; $empty="Nothing found with <b>\"" . $Q . "\"</b> in it's name.<br />\nPlease make a more general search query, or try a different folder.\n\n"; } else { $Q=""; $empty='This folder is empty'; } if($M=="textures") { $ext="jpg"; } else { $ext="zip"; } $i=0; print "<hr>\n"; $endfile=array(); $endsize=array(); $endsize2=array(); foreach (glob($M."/*".$Q."*.".$ext) as $filename) { $filename = explode(".", $filename); $filename=$filename[0]; $filename = explode("/", $filename); $filename=$filename[1]; $i++; $endfile[$i]=$filename; if($ext=="jpg") { $endfile[$i]="<a name='".$endfile[$i]."' href='".$pageurl."?type=view&name=".$endfile[$i]."&folder=".$M."&from=".$type."&addon=".$Q."'>".$endfile[$i]."</a>"; } $endsize[$i]=$size; $endsize2[$i]=$size2; } if($i != 1) { print "<b>".$i."</b> items were found.\n<hr>\n"; } else { print "<b>".$i."</b> item was found.\n<hr>\n"; } echo("<table width='100%' border='" . $border_size . "' cellspacing='0' cellpadding='0' >\n"); if($i!=0) { for ($t = 1; $t < $i; $t++) { $thumbfile = $M."/".$endfile[$t].'.jpg'; if(file_exists($thumbfile)) { $thumbnail = "<a name='".$endfile[$t]."' href='".$page_url."?type=view&name=".$endfile[$t]."&folder=".$M."&from=".$type."&addon=".$Q."'><ul id='icon'><li class='ui-state-default ui-corner-all' title='".$endfile[$t]."'><span class='ui-icon ui-icon-image'></span></li></ul></a>"; } else { $thumbnail = ""; } if($t=="1") { echo("<tr><td width='10%'>Number</td><td width='3%'><ul id='icon'><li class='ui-state-default ui-corner-all' title='".$endfile[$t]."'><span class='ui-icon ui-icon-image'></span></li></ul></td><td width='60%'>Name</td></tr>\n"); } echo("<tr><td>" . $t . "</td><td>".$thumbnail."</td><td>" . $endfile[$t] . "</td></tr>\n"); flush(); } $thumbfile = $M."/".$endfile[$t].'.jpg'; if(file_exists($thumbfile)) { $thumbnail = "<a name='".$endfile[$t]."' href='".$page_url."?type=view&name=".$endfile[$t]."&folder=".$M."&from=".$type."&addon=".$Q."'><ul id='icon'><li class='ui-state-default ui-corner-all' title='".$endfile[$t]."'><span class='ui-icon ui-icon-image'></span></li></ul></a>"; } else { $thumbnail = ""; } echo("<tr><td>" . $t . "</td><td>".$thumbnail."</td><td>" . $endfile[$t] . "</td></tr>\n"); } print "</table>\n"; if($i=="0") { print $empty; } } elseif($type=="view") { $filename=$_GET['name']; $folder=$_GET['folder']; if($_GET['from']=="list"){ $addon="?type=list&map=".$folder."#".$filename; } if($_GET['from']=="search"){ $addon="?type=search&q=".$filename."&map=".$folder."#".$filename; } print"<center><a href='".$home_url."'>Home</a></center>"; print "<hr>\n<center><img src='".$folder."/".$filename.".jpg'></img></center>\n<hr>\n<br />\n<a href='".$page_url."".$addon."'>Previous Page</a>\n"; } $htmlshow=""; if($_GET['type']=="returnOPfile") { if(isset($_GET['split'])) { $splitter=$_GET['split']; } else { $splitter=" | "; } if(isset($_GET['html'])) { $htmlshow="<br />"; } foreach (glob("textures/*.jpg") as $texture){ if(isset($_GET['size'])) { $size=$splitter.filesize($texture); } $texture = explode("/", $texture); $texture=$texture[1]; print "textures".$splitter.$texture.$size."\n".$htmlshow; } foreach (glob("models/*.zip") as $model){ if(isset($_GET['size'])) { $size=$splitter.filesize($model); } $model = explode("/", $model); $model=$model[1]; print "models".$splitter.$model.$size."\n".$htmlshow; } foreach (glob("avatars/*.zip") as $avatar){ if(isset($_GET['size'])) { $size=$splitter.filesize($avatar); } $avatar = explode("/", $avatar); $avatar=$avatar[1]; print "avatars".$splitter.$avatar.$size."\n".$htmlshow; } foreach (glob("seqs/*.zip") as $seq){ if(isset($_GET['size'])) { $size=$splitter.filesize($seq); } $seq = explode("/", $seq); $seq=$seq[1]; print "seqs".$splitter.$seq.$size."\n".$htmlshow; } foreach (glob("sounds/*.zip") as $sound){ if(isset($_GET['size'])) { $size=$splitter.filesize($sound); } $sound = explode("/", $sound); $sound=$sound[1]; print "sounds".$splitter.$sound.$size."\n".$htmlshow; } } ?> <!-- End PHP generated content --> </body> </html> So right now my question to you PHP freaks is, can you please help me edit my script so I can search through a remote directory? *This* is one of the directories I wish to be able to search through & list.. Thanks in advance. Edit; It might help if you know what the site currently looks like. *Click* Hi, I have a small business and I'm looking to bring on a full stack PHP developer.
I'd like for the applicants to build a small full stack example application before I decided. Can anyone help me with cool -- samll -- example projects that might be appropriate for interviews? All the examples online are NOT full stack examples, and very basic -- like writing a file, etc. I need a small full stack application example that's not too big to scare off but not so small it won't matter.
Thanks! Hi,
Can anyone recommend some Business Intelligence tools that can be integrated into a PHP (wordpress/joomla) based website? I've had a good look around the net but they all seem to be written in Java/J2EE and appear to be software based - not sure if im after the possible or impossible here...
Thanks
i have made an delete files script which works for only one directory but not sub directory so i want to delete files of same extention from directory and subdirectory. My current code is Code: [Select] <? $dir = 'hmm/'; function scanr($dir){ $arr = glob($dir.'/*.jpg'); foreach($arr as $vv){ //check if $vv is a file if(is_file($vv)){ //if file, get the filename $vx=explode('/',$vv); $file=$vx[count($vx)-1]; // if no extension delete the file unlink($vv); // print the deletion message echo $vv." deleted!<br>";}else{ // if $vv is a dir then scan it again for files scanr($vv); }} } scanr($dir); ?> I'm trying to echo the directory and sub directory only. I am not looking to show the files contained - only folders. This script reads a directory, and prints out the name of all directories in the directory that I've chosen This works, but it's only displaying the last 2. Right now, I have 6 directories in the the folder I specified. Can anybody tell me why it does not display all of them? Code: [Select] <?php $path = $_SERVER['DOCUMENT_ROOT']."/panel/api/connectors/sample/projects/"; foreach(glob($path . '*', GLOB_ONLYDIR) as $dir) { $dir = basename($dir); echo $dir."<br />"; } ?> Thanks in advance Hey so i need help searching for contacts in my telephone directory...how do i do that in my php? is it a form method or if statements? Hello, can someone please help me make this. I want a simple php directory script that lists files and folders. If a folder is clicked on then open that directory and if a file is clicked on it will download or if its a .txt or picture it will open in the browser. BUT I NEED IT TO LOOK EXACTLY LIKE THIS: http://test512.x10.mx/ which would be the default cpanel indexing but i use godaddy and i dont like there indexing page. Please help and whoever make it i will provide a subdomain on my website for you to host your stuff. Dear friends, I know Zip a file in PHP but I can't Zip a directory.How to Zip a directory in PHP?Can you give an example? I am using php's zip class, and I am able to zip files up individually, but can I zip an entire directory with its file and sub-folders? I am trying to figure out a way to echo the directory name of the script location. |