PHP - Php Simple Html Dom Parser And Database Insertion

hello dear community,

i am currently wroking on a approach to parse some sites that contain datas on Foundations in Switzerland
with some details like goals, contact-E-Mail and the like,,,

See http://www.foundationfinder.ch/  which has a dataset of 790 foundations. All the data are free to use - with no limitations copyrights on it.

I have tried it with PHP Simple HTML DOM Parser - but , i have seen that it is difficult to get all necessary data -that is needed to get it up and running.

Who is wanting to jump in and help in creating this scraper/parser.   I love to hear from you.

Please help me - to get up to speed with this approach?


regards
Dilbertone

good day dear community,


this is a big issue. I have to decide: between native PHP DOM Extension or of simple DOM html parser

well i want to parse the site he http://buergerstiftungen.de/cps/rde/xchg/SID-A7DCD0D1-702CE0FA/buergerstiftungen/hs.xsl/db.htm

http://buergerstiftungen.de/cps/rde/xchg/SID-A7DCD0D1-702CE0FA/buergerstiftungen/hs.xsl/db.htm

I will suggest to use the native PHP "DOM" Extension instead of "simple html parser", since it will be much faster and easier

What do you think about this one here...:

Code: [Select]
$doc = new DOMDocument
@$doc->loadHTMLFile('...URL....'); // Using the @ operator to hide parse errors
$contents = $doc->getElementById('content')->nodeValue; // Text contents of #content


look  forward to hear from you

best regards
db1

Hello dear friends,

first of all : merry merry Xmas!!!


i want to parse with the simple Simple HTML DOM Parser,


well i am pretty new to php and to the Simple HTML DOM Parser.

My example: http://schulen.bildung-rp.de/gehezu/startseite/einzelanzeige.html?tx_wfqbe_pi1[uid]=60119

I want to collect the data in the block:

I have investigated the sourcecode - and found out that the attribute of interest should be this one: class="content"div class="content"><!-- TYPO3SEARCH_begin -->


here the code is: - my trails.

 

// inculde the Simple HTML DOM Parser 
include_once('simple_html_dom.php');

// get the file we want to parse right now,create a DOM 
$html = file_get_html('');

// simple_html_dom::find() creates a new 
// simple_html_dom-Objekt, that consists out of
// corresponding childelements 

foreach($html->find('class: content ') as $h3) {

  // simple_html_dom::get the text in a tag
  // den Text innerhalb eines Tags
  if($h3->innertext == 'Text of a H3 Tag') {
    break;
  }
}

// simple_html_dom::next_sibling() gives the
// next   Element
$table = $h3->next_sibling();



but believe me - it  gives me not back what is aimed.

what  have id done wrong...? 

dbone

I'm using PHP 5.2 Server and Simple HTML DOM 1.5. This script scrape or extract data from a football site, its fully working on PHP 5.9 Server but I need to know how I can fix it for PHP 5.2 server. Can someone give me a hint on how can I fix the error? Thanks in advance.

My PHP 5.2 Server script output shows:
++++++++++++++++
Object id #599 Object id #604 Object id #609 Object id #614 Object id #619
Object id #627 Object id #632 Object id #637 Object id #642 Object id #647
Object id #655 Object id #660 Object id #665 Object id #670 Object id #675
Object id #683 Object id #688 Object id #693 Object id #698 Object id #703
Object id #711 Object id #716 Object id #721 Object id #726 Object id #731
++++++++++++++++

while PHP 5.9 Server says
++++++++++++++++
Rk Player Team POS OPPONENT
1 Aaron Rodgers GB QB at CAR
2 Tom Brady NE QB vs. SD
3 Matt Schaub HOU QB at MIA
4 Michael Vick PHI QB at ATL
++++++++++++++++

I did applied the bug solution listed on https://sourceforge.net/tracker/index.php?func=detail&aid=3107230&group_id=218559&atid=1044037 but it is still not working. It says:
++++++++++++++++
Details:

I get compiler errors in PHP 5.2 when using this as an object.

The offending lines are 609 and 940, which both contain this construct:

if ($this->size>0) $this->char = $this->doc[0];

This tries to get the first character of $this->doc, but PHP 5.2 sees it as trying to access it as an array. It's easily fixed by this:

if ($this->size>0) $this->char = substr($this->doc, 0, 1);

Or you could probably use chr(ord($this->doc)) as well. Either way solves the compile error without changing functionality.
++++++++++++++++

Here are my codes:

Code: [Select]
<?php
# don't forget the library
include('simple_html_dom.php');

# this is the global array we fill with article information
$articles = array();
$source = 'http://www.athlonsports.com/columns/winning-game-plan/fantasy-football-qb-rankings';
# passing in the first page to parse, it will crawl to the end
# on its own
getArticles($source);      


function getArticles($page) {
global $articles, $descriptions;

$html = new simple_html_dom();
$html->load_file($page);

//$items = $html->find('div[class=preview]');
$items = $html->find('tbody tr');

foreach($items as $post) {
    # remember comments count as nodes
   /*$articles[] = array($post->children(3)->outertext,
                        $post->children(6)->first_child()->outertext);*/
    $articles[] = array($post->children(0), $post->children(1), $post->children(2), $post->children(3), $post->children(4));
}

# lets see if there's a next page
if($next = $html->find('a[class=nextpostslink]', 0)) {
    $URL = $next->href;
    echo "going on to $URL <<<\n";
    # memory leak clean up
   $html->clear();
    unset($html);

    getArticles($URL);
}
}

?>


<html>
<head>
</head>
<body>
<?
echo "Source: " . $source;
?>
<table cellpadding="5" cellspacing="0" border="0">
<?php
    foreach($articles as $item) {
        echo "<tr>";
        echo "<td>" . $item[0] . "</td><td>" . $item[1] . "</td><td>" . $item[2] . "</td>";
        echo "<td>" . $item[3] . "</td><td>" . $item[4] . "</td>";
        echo "<tr>";
    }
?>
</table>


</body>
</html>

I'm trying to learn writing php scripts with html-simple-dom. I am scrapping my website with html-simple-dom and the below code, but I am running into an issue with inserting the scrapped data into my database.

This code scraps the website and grabs the html data I want.

Code: [Select]
$ret =  $html->find('table[class=data] tr');
foreach($ret as $visitor){
$visitor =  $visitor->find('td','1') . "<br>";
$insert="INSERT INTO $dbtable (visitor) VALUES ('$visitor')";
mysql_query($insert) OR die(mysql_error());
}
But this is being inserted into the database "visitor" field.

Code: [Select]
<td colspan="1" rowspan="1" style="text-align: left;"><a style="border-bottom:1px dotted;" onclick="loadTeamSpotlight(jQuery(this));" rel="HEN" href="javascript:void(0);">HENRY</a></td><br>

I only want "HENRY" to be extracted out of the html code above and inserted into the database table.

Any help would be greatly appreciated, as I have been pulling my hair out trying to figure this out. Do I need to use explode or something?

Thanks,

Code: [Select]
//enter into btree ----------------------------------------------
$sql_plevel='select nlevel from btree where uniqueid = '.$formjoinedunder;
$abc = mysql_query($sql_plevel , $db)
$result_plevel = mysql_fetch_assoc($abc) or die(mysql_error());
$plevel = $result_plevel['nlevel'] + 1;
$insert_btree = 'INSERT INTO btree (uniqueid,nlevel,pside) VALUES
('.$reguniqueid.','.$plevel.','.$formposition.')';
mysql_query($insert_btree , $db) or die(mysql_error());
Can any one please tell me what is the problem in the above code it does not insert into the table btree.

<?php

if(isset($_POST['submit'])){

$uname = $_POST['username'];
$pword = $_POST['password'];

/*** mysql hostname ***/
$hostname = 'localhost';

/*** mysql username ***/
$username = 'root';

/*** mysql password ***/
$password = 'anty90';

try {
    $link = new PDO("mysql:host=$hostname;dbname=gambling", $username, $password);
    /*** echo a message saying we have connected ***/
    echo 'Connected to database<br />';

    /*** INSERT data ***/
    $stmt = $link->prepare("INSERT INTO gamb(username, password) VALUES (?, ?)");
	
	try{
	$stmt->execute(array("$uname", "$pword"));
	} catch(PDOException $e){
	
		echo "Exception caught: $e";
	
	}

    /*** echo the number of affected rows ***/
    //echo $count;

    /*** close the database connection ***/
    $link = null;
    }
catch(PDOException $e)
    {
    echo $e->getMessage();
    }
  
}
?>

<html>

<form action='home.php' method='post'>

<input type="text" name="username" >
<input type="password" name="password" >
<input type="submit" name="submit" value="submit">

</form>

</html>

I'm new to databse programming so I was just wondering if this was vulnerable to sql injection or not.

Im using some software called php html dom parser i wont to be able to keep the souce tidy

i.e before dom parser

<?php

//////////////////////SEO TOOL///////////////////////////

$title = 'Green Deal Nationwide - PB Energy Solutions Ltd';
$description = 'Delivering all your environmental needs to \'green\' up your business, improve reputation, increase profitability and give a competitive advantage.';

///////////////////////////////////////////////////////

?>
<?php include('includes/settings.php'); ?>
<?php include('includes/header.php'); ?>

    <div class="container">

        <div id="large-page-img">
        <img src="<?php echo URL(); ?>images/home-page-slide.jpg" width="911" height="230" />
        <img src="<?php echo URL(); ?>images/home-page-slide-1.jpg" width="911" height="230" />
        <img src="<?php echo URL(); ?>images/home-page-slide-2.jpg" width="911" height="230" />
        </div>
        <div id="content-home">
     
            <div class="iedit">

after dom parser saved to file

<?php    //////////////////////SEO TOOL///////////////////////////    $title = 'Green Deal Nationwide - PB Energy Solutions Ltd';  $description = 'Delivering all your environmental needs to \'green\' up your business, improve reputation, increase profitability and give a competitive advantage.';    ///////////////////////////////////////////////////////    ?>  <?php include('includes/settings.php'); ?>  <?php include('includes/header.php'); ?>        <div class="container">            <div id="large-page-img">          <img src="<?php echo URL(); ?>images/home-page-slide.jpg" width="911" height="230" />          <img src="<?php echo URL(); ?>images/home-page-slide-1.jpg" width="911" height="230" />          <img src="<?php echo URL(); ?>images/home-page-slide-2.jpg" width="911" height="230" />          </div>          <div id="content-home">                      <div class="iedit"><div class="iedit">

is there anyway i can keep it like the original fil after dom?

how should i approach the following:
a page with a products list+link to product page

i want to build a crawler that loops through all the products in the list and goes to the product page and
and parses the product page.

need help with the loop

hello dear Freaks 

 

i am currently  musing bout the portover of a python bs4 parser to php -  working with the simplehtmldom-parser / pr the DOM-selectors... (see below).

The project: for a list of meta-data of wordpress-plugins: - approx 50 plugins are of interest! but the challenge is: i want to fetch meta-data of all the existing plugins. What i subsequently want to filter out after the fetch is - those plugins that have the newest timestamp - that are updated (most) recently. It is all aobut acutality...

https://wordpress.org/plugins/participants-database ....and so on and so forth.  

https://wordpress.org/plugins/wp-job-manager
https://wordpress.org/plugins/ninja-forms
https://wordpress.org/plugins/participants-database ....and so on and so forth.
 

we have the following set of meta-data for each wordpress-plugin:

Version: 1.9.5.12 
installations: 10,000+    
WordPress Version: 5.0 or higher 
Tested up to: 5.4 PHP  
Version: 5.6 or higher    
Tags 3 Tags:databasemembersign-up formvolunteer
Last updated: 19 hours ago

 

the project consits of two parts:   the looping-part: (which seems to be pretty straightforward). the parser-part: where i have some issues - see below. I'm trying to loop through an array of URLs and scrape the data below from a list of wordpress-plugins. See my loop below-

as a base i think it is good starting point to work from the following target-url: 

 plugins wordpress.org/plugins/browse/popular with 99 pages of content: cf ...
 wordpress.org/plugins/browse/popular/page/1 
wordpress.org/plugins/browse/popular/page/2
wordpress.org/plugins/browse/popular/page/99

 

the Output of text_nodes:

['Version: 1.9.5.12', 'Active installations: 10,000+', 'Tested up to: 5.6 ']  

but if we want to fetch the data of all the wordpress-plugins and subesquently sort them to show the -let us say - latest 50 updated plugins. This would be a interesting task:

 

first of all we need to fetch the urls

then we fetch the information and have to sort out the newest- the newest timestamp. Ie the plugin that updated most recently

List the 50 newest items - that are the 50 plugins that are updated recently ..

 

we have the following set
 

see here the Soup_ 

 soup = BeautifulSoup(r.content, 'html.parser')
        target = [item.get_text(strip=True, separator=" ") for item in soup.find(
            "h3", class_="screen-reader-text").find_next("ul").findAll("li")[:8]]
        head = [soup.find("h1", class_="plugin-title").text]
        new = [x for x in target if x.startswith(
            ("V", "Las", "Ac", "W", "T", "P"))]
        return head + new


with ThreadPoolExecutor(max_workers=50) as executor1:
    futures1 = [executor1.submit(parser, url) for url in allin]

for future in futures1:
    print(future.result())

 

see the formal output

 

Quote

[lorem ipsum dolor sit amet', 'Version: 2.34.1', 'Last updated: 5 months ago', 'Tags: magna aliquyam erat, sed diam voluptua. At vero eos et accusam']
[consetetur sadipscing elitr', 'Version: 6.54.1', 'Last updated: 5 months ago', 'Tags: lorem ipsum dolor sit amet']
[sed diam nonumy eirmod tempor invidunt ut labore', 'Version: 7.16.1', 'Last updated: 5 months ago', 'Tags: tarifa, sevilla lisabin invidunt ut labore et dolore magna aliquyam erat']
[tempor invidunt ut taria malaga jerusalem labore', 'Version: 9.58.1', 'Last updated: 5 months ago', 'Tags: ilabore et lissabon dolore magna aliquyam erat']

 

background: https://stackoverflow.com/questions/61106309/fetching-multiple-urls-with-beautifulsoup-gathering-meta-data-in-wp-plugins

Well - i guess that we c an do this with the simple DOM Parser - here the seclector reference. 

https://stackoverflow.com/questions/1390568/how-can-i-match-on-an-attribute-that-contains-a-certain-string

 

look forward to any hint and help.

 

have a great day 

Edited May 3, 2020 by dil_bert

Hello Guys,
With this code I am able to match two strings, where one of them is standard and the other is changeable. This doesn't matter. What it does matter that I want to insert the values into a database consisted of a number of columns. As you can see in the code every iteration it gives one value of the whole insertion.
I hope I made my question clear.
Code: [Select]
<?php
preg_match_all('/\b(' . implode("|", array_map("preg_quote", $dictionary)) . ')/i', $file, $foundwords);
$sim = array_count_values($foundwords[0]);
$max = max ($sim);
foreach ($sim as $key=> $value) {
$norm = ($value/$max);
echo $key. " = $norm </br>";
}
$foundwords[0] = array_flip ($foundwords[0]);
ksort($foundwords[0]);
foreach ($foundwords[0] as $key=> $value) {
echo "<br/>$key "; // [color=red]after here how can I insert all the values in one row[/color]
 } 
?>

I have 2 tables and am trying to conduct an SQL query. I simply need to know how to use a JOIN when two fields in each table are needed to match the records up. I would rather not create another field in each table.

I have put my query below which would work other than for what is obviously not possible ("ON xyz=xyz && abc=abc" etc.)

Table 1 "Employees"

employeeDept / deptEmployeeRef / name / wage

A / 2 / Dave Smith / 2000
B / 2 / Robert Brown / 2500


Table 2 "Deductions"

monthNo / employeeDept / deptEmployeeRef / deduction

32 / B / 2 / 300
32 / C / 2 / 300
32 / A / 3 / 300
33 / B / 2 / 500


Each department has employees starting from number 1 so two fields are needed in each table to match the records up.

SELECT t1.name, (t1.wage - t2.deduction) FROM $tableName1 AS t1 LEFT JOIN $tableName2 AS t2 ON t1.employeeDept=t2.employeeDept && t1.deptEmployeeRef=t2.deptEmployeeRef WHERE t2.monthNo = '32'

So this should return:
Dave Smith 2000
Robert Brown 2200

Hi guys,

I want to create a simple database and use PHP to output the database to certain positions on a HTML page.

I am basically trying to recreate this: http://bit.ly/dX4byI

I want to have a form where the event info is selected from drop down box such as: Venue and Date then the the rest of the form data (event name, artists, opening times) will go into the right box depending on this.

Does anyone have any pointers? Am I even going about this the right way?!

i have enters some data using form which submit data in sql table1 like this,

Fields name in table -->>         id   First name     Lastname    Date   
data saved           -->>              1  user                1               2011-05-10
                                               2  user                2               2011-05-11
                                               3  user                3               2011-05-12

now i dont want to duplicate date in database let say while inserting data using form,

Firstname :  ____________
Lasename :  ____________
Date         :  ____________ (i dont want this date to b save if the date alredy exist in database,
                                          it should prompt user "Date already exist and to edit click here
                                         (this will be the link to that date which is already exist so that we can edit) " )

                                SAVE


i user  files ,       form.php ,   insert.php(so insert values in database) ,
so tell me what function should i use to solve my problem....