siteorigin/laravel-kernel-crawler

Crawls a Laravel site using internal Kernel.

0.1.0 2021-03-05 09:43 UTC

Requires

• php: ^7.4
• icamys/php-sitemap-generator: ^2.0
• laravel/framework: ^8.0
• symfony/dom-crawler: ^5.2

Requires (Dev)

• mockery/mockery: ^1.4
• orchestra/testbench: ^6.7
• phpunit/phpunit: ^9.5

MIT

• Greg Priday

README

This package gives you a local site crawler based on a Laravel LazyCollection. Each request goes directly through the local HTTPKernel, so it doesn't generate web server requests.

2 examples of you might need this is for warming your site's cache, and generating a sitemap. This package offers commands for both of these.

It's currently compatible with Laravel 8.0+

Installation

Installation instructions coming once this package is on Packagist.

To publish the configuration files, use:

php artisan vendor:publish --provider="SiteOrigin\KernelCrawler\CrawlerServiceProvider" --tag="config"

Usage

Here's a very basic use example:

use SiteOrigin\KernelCrawler\Crawler;
use SiteOrigin\KernelCrawler\Exchange;


$crawler = new Crawler();
$crawler->each(function(Exchange $ex){
	// $ex->request is an Illuminate\Http\Request object
	$ex->request->url();
	// $ex->response is an Illuminate\Http\Response object
	$ex->response->getContent()
});

After creating a Crawler, you can use all of the Laravel LazyCollection functions. This offers a collection of Exchange objects.

Warming Cache

If all you need to do is generate dummy requests to each public URL on your site, then you can use php arisan crawler:start. This is useful if you want to warm the cache for all public URLs on your site.

Generating a Sitemap

This package offers the command php artisan crawler:sitemap to generate the sitemap. You can configure your sitemap using the crawler.php configuration file. See the Installation section for instructions on publishing this config file.