commently / crawler

v1.0.0 2026-08-11 16:15 UTC

Requires

• php: ^8.3
• illuminate/cache: ^12.0|^13.0
• illuminate/contracts: ^12.0|^13.0
• illuminate/http: ^12.0|^13.0
• illuminate/support: ^12.0|^13.0

MIT

Published on Packagist.org on 2026-08-11 16:21 UTC

README

A minimal, polite URL crawler for Laravel. Give it a URL and it brings you the raw document — respecting robots.txt, per-host rate limits and conditional requests. It does not know (or care) what it crawls. Parsing and consuming the content is entirely your job, so the package plugs into any project without dictating a schema.

use Commently\Crawler\CrawlRequest;
use Commently\Crawler\Crawler;
use Commently\Crawler\CrawlOutcome;

$response = app(Crawler::class)->fetch(new CrawlRequest('https://example.com/feed.xml'));

if ($response->outcome === CrawlOutcome::Success) {
    file_put_contents('feed.xml', $response->body);
}

Why this package

Most "crawler" packages are opinionated about what you crawl: they discover links, follow HTML, render JavaScript, or ship their own database schema. laravel-polite-crawler deliberately does none of that. It is just the polite transport layer:

• what it does — fetch a URL (or a batch of URLs) over HTTP, politely and concurrently, and hand you the raw body plus transport metadata;
• what it doesn't do — parse feeds, scrape HTML, store rows, decide when to fetch again.

The collection pipeline stays in your application:

your scheduler  →  CrawlUrl job / Crawler::fetchMany  →  your parser  →  your storage
   (when)                 (polite fetch)                     (what it means)

Features

• Polite by default — a single honest User-Agent, no browser impersonation.
• robots.txt — consulted (and cached) per host; a failure to fetch it never blocks.
• Per-host rate limiting — minimum delay between two requests to the same host and at most N concurrent requests per host, enforced across processes via the distributed cache.
• Global concurrency cap — simultaneous requests across all hosts, batched in waves.
• Conditional requests — sends If-None-Match / If-Modified-Since and reports 304 Not Modified instead of re-downloading.
• Outcome classification — every attempt ends in one of five explicit outcomes (success, not_modified, error, throttled, blocked), with Retry-After parsed for 429 responses.
• Batch fetching — fetchMany() crawls a list of URLs concurrently while respecting the same politeness rules, with a permit hook for app-level idempotency.
• Queued entry point — a ready CrawlUrl job that hands fetched documents to a Sink implementation you bind in the container.
• Zero domain coupling — no Eloquent, no models, no schema.

Requirements

• PHP 8.3+
• Laravel 12 or 13

Installation

composer require commently/crawler

If you are developing against a local checkout (the recommended workflow for this package), use a path repository in your composer.json:

{
    "repositories": [
        { "type": "path", "url": "packages/*" }
    ],
    "require": {
        "commently/crawler": "*"
    }
}

The service provider is auto-discovered. Publish the configuration when you want to tune the defaults:

php artisan vendor:publish --tag=crawler-config

Usage

Fetch a single URL

use Commently\Crawler\CrawlRequest;
use Commently\Crawler\Crawler;
use Commently\Crawler\CrawlOutcome;

$response = app(Crawler::class)->fetch(
    new CrawlRequest(
        url: 'https://example.com/feed.xml',
        etag: $previousEtag,            // sent as If-None-Match
        lastModified: $previousDate,    // sent as If-Modified-Since
    )
);

match ($response->outcome) {
    CrawlOutcome::Success => $this->store($response->body, $response->etag, $response->lastModified),
    CrawlOutcome::NotModified => $this->keepCachedCopy(),          // status 304
    CrawlOutcome::Error => $this->retryLater($response->retryAfterSeconds, $response->error),
    CrawlOutcome::Throttled => $this->retryLater(30),               // host politeness said "wait"
    CrawlOutcome::Blocked => $this->skip(),                          // robots.txt said "no"
};

CrawlRequest is a plain DTO:

Property	Description
url	The URL to fetch.
key	Stable identifier used to key batch results; defaults to the URL.
etag	Sent as If-None-Match.
lastModified	Sent as If-Modified-Since.
headers	Extra request headers merged on top of the defaults.

CrawlResponse carries the raw body and transport metadata:

Property	Description
outcome	One of CrawlOutcome::Success/NotModified/Error/Throttled/Blocked.
status	HTTP status code (0 for transport errors / gated attempts).
body	Raw response body (empty for not_modified).
headers	Response headers (header(string $name) for case-insensitive lookup).
effectiveUri	Final URL after redirects.
duration	Request time in seconds, when available.
etag / lastModified	Validators returned by the server.
retryAfterSeconds	Parsed Retry-After for 429 responses.
error	Reason message for error outcomes.

Fetch many URLs concurrently

fetchMany() crawls a list of requests in waves, applying the same politeness rules, and returns only the attempts that were actually fired:

use Commently\Crawler\CrawlRequest;
use Commently\Crawler\Crawler;

$responses = app(Crawler::class)->fetchMany([
    new CrawlRequest('https://a.example.com/feed', key: 'a'),
    new CrawlRequest('https://b.example.com/feed', key: 'b'),
    new CrawlRequest('https://c.example.com/feed', key: 'c'),
]);

foreach ($responses as $key => $response) {
    // $responses contains a result only for requests that were actually sent.
    // Requests skipped by politeness (host not eligible, lock busy, robots.txt)
    // are absent — retry them on a later run.
}

The permit hook

Pass a permit callable to refuse individual requests right before they are fired — useful for app-level idempotency (e.g. "has this feed already been fetched by another worker?"):

$responses = $crawler->fetchMany($requests, permit: fn (string $key, CrawlRequest $request) => $this->isNotAlreadyFetched($key));

Sink + queued job

The package ships a CrawlUrl queued job and a Sink contract. The job fetches the URL (enforcing politeness) and hands the document to whatever Sink is bound in the container:

use Commently\Crawler\Contracts\Sink;
use Commently\Crawler\CrawlResponse;

class StoreFeedSink implements Sink
{
    public function handle(CrawlResponse $response): void
    {
        // parse $response->body and store it somewhere
    }
}

Bind it in a service provider:

use App\Sinks\StoreFeedSink;
use Commently\Crawler\Contracts\Sink;

$this->app->bind(Sink::class, StoreFeedSink::class);

Then dispatch crawls anywhere:

use Commently\Crawler\CrawlRequest;
use Commently\Crawler\Jobs\CrawlUrl;

CrawlUrl::dispatch(new CrawlRequest('https://example.com/feed.xml'));

Note: the job only calls the sink for success / not_modified outcomes. error, throttled and blocked attempts are dropped (log or requeue them yourself if you care).

Configuration

All settings are optional; sensible defaults apply.

// config/crawler.php
return [

    // Single honest crawler User-Agent. Never rotated, never browser-masqueraded.
    'user_agent' => env('RSS_CRAWLER_USER_AGENT', 'CommentlyBot/1.0 (+https://example.com/bot)'),

    'http' => [
        'connect_timeout' => 5,    // seconds for the TCP/TLS handshake
        'timeout'         => 20,   // seconds for the whole request
        'max_concurrency' => 15,   // max simultaneous requests across all hosts
        'max_redirects'   => 5,
        'http_version'    => '1.1', // '1.1' avoids HTTP/2 resets on some CDNs
    ],

    'rate_limit' => [
        'per_host_delay'          => 30,  // min pause between two requests to the same host (s)
        'max_concurrent_per_host' => 1,   // max concurrent requests to the same host
        'lock_ttl'                => 120, // TTL of the distributed host locks (s)
    ],

    'robots' => [
        'enabled'   => true,
        'cache_ttl' => 86400, // robots.txt is fetched at most once per host per TTL
    ],

];

How politeness works

• Per-host delay. After a request to a host, a timestamp is written to the cache; further requests to that host are skipped until the delay has passed. A redirect to another host applies the same spacing to the target host.
• Per-host concurrency. A distributed lock per host guarantees at most max_concurrent_per_host in-flight requests, even across multiple queue workers or processes.
• Global concurrency. fetchMany() builds waves of at most max_concurrency requests at a time.
• robots.txt. Fetched once per host (cached for cache_ttl) and only the matching User-agent group (the crawler's own UA or *) is honoured. Unreachable robots.txt never blocks fetching.

Honest headers

The crawler sends a fixed, honest User-Agent, an XML-capable Accept list and Accept-Encoding: gzip. It never impersonates a browser, rotates user agents, or sends Sec-Fetch-* headers.

Testing

composer test

The package ships no tests of its own; it is covered by the consuming application's test suite (see the HTTP-fake based tests for rate limiting, robots.txt and conditional-request behaviour).

License

MIT