repat/spatie-crawler-redis

Redis CrawlQueue for spatie/crawler

0.2.1 2022-01-06 20:50 UTC

Requires

• php: >=7.1
• predis/predis: ^1.1
• spatie/crawler: >=4.6

Requires (Dev)

• larapack/dd: ^1.1
• phpunit/phpunit: ^7.0

MIT

• repat

• redis
• predis
• crawler
• spatie
• crawlqueue

README

spatie-crawler-redis is an alternative CrawlerQueue implementing the Spatie\Crawler\CrawlQueue\CrawlQueue interface using Redis Hashes.

Installation

$ composer require repat/spatie-crawler-redis

Example

Create a Predis\Client beforehand if you need options, such as selecting a database. If you don't pass a client, a new one without options will be used. Predis assumes 127.0.0.1, 6379 and 0 as default host, port and database. You can also pass a custom prefix, otherwise uniqid() will be used.

use Repat\CrawlQueue\RedisCrawlQueue;

// see https://github.com/nrk/predis for options
$options = [
    'database' => 7,
];

$prefix = uniqid() . ':'; // same as passing no prefix

$redisClient = new \Predis\Client($options);

// ...
->setCrawlQueue(new RedisCrawlQueue($redisClient, $prefix))

// uses new \Predis\Client without options
->setCrawlQueue(new RedisCrawlQueue())

TODO

• phpredis support

Testing

Thanks spatie for the tests. These are the instructions:

To run the tests you'll have to start the included node based server first in a separate terminal window.

cd tests/server
npm install
./start_server.sh

With the server running, you can start testing.

vendor/bin/phpunit

License

• MIT, see LICENSE

Version

• Version 0.2.1

Contact

repat

• Homepage: repat.de
• e-mail: [e-mail]
• Twitter: @repat123