monkeyphp/php-spider

A basic web spider written in PHP.

0.0.2 2016-09-24 18:34 UTC

Requires

• php: >=5.3.3
• zendframework/zend-dom: ^2.4
• zendframework/zend-eventmanager: ^2.4
• zendframework/zend-http: ^2.4
• zendframework/zend-uri: ^2.4

Requires (Dev)

• phpunit/phpunit: ~3.7

BSD-3-Clause

• David White

README

I needed a really basic spider to crawl a site to warm the cache; this is the result.

Examples

Simple

The simplest way to use PhpSpider is to construct an instance and pass the root url of the site that you wish to crawl to the Spider::crawl method.

By design, PhpSpider will only crawl pages that

• Return a content-type header of text\html
• That are on the same domain as the supplied root

use PhpSpider\Spider\Spider;

$spider = new Spider();
$spider->crawl('https://www.example.com');

Advanced

If you need to override how PhpSpider works you can add event listeners to be notified, that can then affect how PhpSpider operates.

There are 5 events triggered by PhpSpider

• Spider::SPIDER_CRAWL_PRE - Triggered before PhpSpider starts to crawl a site.
• Spider::SPIDER_CRAWL_POST - Triggered one PhpSpider has finished it's crawl.
• Spider::SPIDER_CRAWL_PAGE_PRE - Triggered just before a page is crawled
• Spider::SPIDER_CRAWL_PAGE_POST - Triggered once a page is crawled
• Spider::SPIDER_CRAWL_PAGE_ERROR - Trigged if an error occurs whilst crawling a page

You can find examples in the examples directory included in this repository.

    $ php ./examples/example_0.php

use PhpSpider\Spider\Spider;
use Zend\EventManager\Event;

$listener function ($event) {
    $uri = $event->getParam('uri');
    echo $uri;
};

$spider = new Spider();
$spider->getEventManager()->attach(Spider::SPIDER_CRAWL_PAGE_PRE, $listener, 1000);

$spider->crawl('https://www.example.com');

To Run Tests

$ vendor/bin/phpunit -c tests/phpunit.xml