PHP - Extract Text From Page

Hey folks, I am trying to create a small script that will retrieve content from a site, strip it of everything but human readable words, then remove numbers, single letters, and words that I specify. I have the following code which is live on
http://salesleadhq.com/tools/crawler/meta.php?url=http://www.cooking.com.

My problem is that it is not removing all of the the words I specify, only some... ??
I think i would rather an external word list as well... if anyone can assist me with that.

Thank you!

Code: [Select]
<?php
$url = (isset($_GET['url']) ?$_GET['url'] : 0);
$str = file_get_contents($url);
####################################################################3
function get_url_contents($url){
        $crl = curl_init();
        $timeout = 5;
        curl_setopt ($crl, CURLOPT_URL,$url);
        curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, $timeout);
        $ret = curl_exec($crl);
        curl_close($crl);
        return $ret;
}
#--------------------------------------Strip html tag----------------------------------------------------
function StripHtmlTags( $text )
{
  // PHP's strip_tags() function will remove tags, but it
  // doesn't remove scripts, styles, and other unwanted
  // invisible text between tags.  Also, as a prelude to
  // tokenizing the text, we need to insure that when
  // block-level tags (such as <p> or <div>) are removed,
  // neighboring words aren't joined.
  $text = preg_replace(
    array(
      // Remove invisible content
      '@<head[^>]*?>.*?</head>@siu',
      '@<style[^>]*?>.*?</style>@siu',
      '@<script[^>]*?.*?</script>@siu',
      '@<object[^>]*?.*?</object>@siu',
      '@<embed[^>]*?.*?</embed>@siu',
      '@<applet[^>]*?.*?</applet>@siu',
      '@<noframes[^>]*?.*?</noframes>@siu',
      '@<noscript[^>]*?.*?</noscript>@siu',
      '@<noembed[^>]*?.*?</noembed>@siu',

      // Add line breaks before & after blocks
      '@<((br)|(hr))@iu',
      '@</?((address)|(blockquote)|(center)|(del))@iu',
      '@</?((div)|(h[1-9])|(ins)|(isindex)|(p)|(pre))@iu',
      '@</?((dir)|(dl)|(dt)|(dd)|(li)|(menu)|(ol)|(ul))@iu',
      '@</?((table)|(th)|(td)|(caption))@iu',
      '@</?((form)|(button)|(fieldset)|(legend)|(input))@iu',
      '@</?((label)|(select)|(optgroup)|(option)|(textarea))@iu',
      '@</?((frameset)|(frame)|(iframe))@iu',
    ),
    array(' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0",),$text );

  // Remove all remaining tags and comments and return.
  return strtolower( $text );
}

function RemoveComments( & $string )
{
  $string = preg_replace("%(#|;|(//)).*%","",$string);
  $string = preg_replace("%/\*(?:(?!\*/).)*\*/%s","",$string); // google for negative lookahead
  return $string;
}


$html = StripHtmlTags($str);

###Remove number in html################
$html  = preg_replace("/[0-9]/", " ", $html);

#replace &nbsp; by ' '
$html = str_replace("&nbsp;", " ", $html);

######remove any words################
$remove_word = array("amp","carry","serious","for","re","looking","accessories","you","used","wright","none","selection","come","second","you","new","a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your");
foreach($remove_word as $word) {
$html = preg_replace("/\s". $word ."\s/", " ", $html);
}

######remove space
$html =  preg_replace ('/<[^>]*>/', '', $html);

$html =  preg_replace('/\s\s+/', ', ', $html);
$html =  preg_replace('/[\s\W]+/',', ',$html);   // Strip off spaces and non-alpha-numeric

#remove white space, Keep : . ( ) : &
//$html = preg_replace('/\s+/', ', ', $html);


###process#########################################################################
$array_loop = explode(",", $html);
$array_loop1 = $array_loop;
$arr_tem = array();

foreach($array_loop as $key=>$val) {
if(in_array($val, $array_loop1)) {
if(!$arr_tem[$val]) $arr_tem[$val] = 0;
$arr_tem[$val] += 1;

if ( ($k = array_search($val, $array_loop1) ) !== false )
unset($array_loop1[$k]);
}
}

arsort($arr_tem);

###echo top 20 words############################################################
echo "<h3>Top 20 words used most</h3>";
$i = 1;
foreach($arr_tem as $key=>$val) {
if($i<=20) {
echo $i.":&nbsp;&nbsp;".$key." (".$val." words)<br />";
$i++;
}else break;
}
echo "<hr />";
###print array#####################################################################
echo (implode(", ", array_keys($arr_tem)));

?>

For example:

I am using this code:

Code: [Select]
$myFile = "newuser.txt";
$fh = fopen($myFile, 'r');
$theData = fread($fh, 5);
fclose($fh);
echo $theData;
and it displays:

Code: [Select]
Bob 2 Which I am reading from my newuser.txt file!

Which corresponds to the username bob, and he has the ID of 2.

Now I want to make that linkable like this:

Code: [Select]
<a href=.?act=Profile&id=$IDFROMTEXTFILE(2)>$NAMEFROMTEXTFILE(BOB)</a>

this is possible? If so, Thanks!

Hello. I have one programming problem. I have this log, from witch i have to read specific area of text:
webtopay.log

OK 123.456.7.89 [2012-03-15 09:09:59 -0400] v1.5: MIKRO to:"1398", from:"865458961", id:"13525948", sms:"MCLADM thing"

So i need the script to extract word "thing" from that log. Also that script has to check if there is new entries in the log, and extract text from the last one.
(Explaining in other words, that script should extract word AFTER MCLADM. Every time its a different word)

p.s. I need that script to be integrated here (this has to send command to server "/manuadd (text from log)" :
Code: [Select]
<?php
try{
     $HOST = "178.16.35.196"; //the ip of the bukkit server
     $password = "MCLietuva";
     //Can't touch this:
     $sock = socket_create(AF_INET, SOCK_STREAM, 0)
     or die("error: could not create socket\n");
     $succ = socket_connect($sock, $HOST, 4445) 
     or die("error: could not connect to host\n");
     //Authentification
     socket_write($sock, $command = md5($password)."<Password>", strlen($command) + 1)
     or die("error: failed to write to socket\n");
     //Begin custom code here.
     socket_write($sock, $command = "/Command/ExecuteConsoleCommandAndReturn-SimpleBroadCast:broadcast lol;", strlen($command) + 1) //Writing text/command we want to send to the server
     or die("error: failed to write to socket\n");
     sleep(2); // This is example code and here has to be that script i want to make.
     //while(($returnedString = socket_read($sock,50000))!= ""){
        $returnedString = socket_read($sock,50000,PHP_NORMAL_READ);
        print($returnedString)
     //}
     print("End of script");
     socket_close($sock);
}catch(Exception $e){
    echo $e->getMessage();
}
?>

I hope i made things clear and you will help me
Thanks

Is it possible to write a php script that would extract data from an external web page. I've been working with php for a few years as a hobby, but I've never seen this done or needed a reason to until now. Thanks in advance for any responses.

This topic has been moved to Third Party PHP Scripts.

http://www.phpfreaks.com/forums/index.php?topic=321546.0