wizardcompass / robots-txt-parser

v2.0.0 2026-09-05 17:24 UTC

Requires

• php: ^8.2
• leopoletto/robots-txt-parser: ^3.0

Requires (Dev)

• friendsofphp/php-cs-fixer: ^3.87
• phpstan/phpstan: ^2.1
• phpunit/phpunit: ^10.0

MIT

• Leonardo Poletto

Published on Packagist.org on 2026-09-05 17:25 UTC

README

This package is retired. Use leopoletto/robots-txt-parser instead.

Version 2.0.0 is a compatibility layer. It contains no parsing of its own — every call is delegated to leopoletto/robots-txt-parser, and the result is reshaped into the arrays this package has always returned.

Nothing breaks if you do nothing. Existing code keeps working, and the original test suite is retained unchanged as the proof of that. But this package will not gain features, so new work should use the replacement directly.

Why it was retired

The replacement is the same author's actively maintained parser, and does considerably more:

	wizardcompass/robots-txt-parser	leopoletto/robots-txt-parser
Directive parsing	counts by type	typed records with line numbers
User-agent groups	flat counts	real groups; shared rules resolve correctly
Access checking	none	RFC 9309 precedence, with the deciding rule
Meta robots / X-Robots-Tag	none	parsed, validated, reconciled
Known crawlers	none	~1,800, with categories
SEO audit	none	findings with impact and fixes
Static analysis	level 8	level 8

Migrating

// before
$parser = new WizardCompass\RobotsTxtParser\RobotsTxtParser;
$result = $parser->parse($content);
$result['record_counts']['by_type']['disallow'];
$result['sitemaps'];

// after
$response = (new Leopoletto\RobotsTxtParser\RobotsTxtParser)->parseText($content);
$document = $response->records();

count($document->disallowed());
$document->sitemaps();                          // typed, with line numbers and validity
$document->isAllowed('GPTBot', '/some/path');   // this package could not answer this

Fetching a URL now requires identifying your bot, which is what site owners expect:

// before
$parser->parseFromUrl('https://example.com');

// after
(new Leopoletto\RobotsTxtParser\RobotsTxtParser)
    ->withBotSignature('MyBot', '1.0', 'https://example.com/bot')
    ->parseUrl('https://example.com/some/page');

validate() returned a pass/fail verdict. The replacement explains what is wrong and what it costs:

$report = (new Leopoletto\RobotsTxtParser\Audit\Auditor)->audit($response);

foreach ($report->actionable() as $finding) {
    $finding->title;    // "1 of 6 search engines are blocked"
    $finding->impact;   // why that matters
    $finding->fix;      // what to do about it
}

Behaviour that changed in 2.0.0

Two deliberate divergences, both because the old behaviour reported something untrue:

• Crawldelay: without the hyphen is no longer counted as a crawl delay. No crawler recognises that spelling, so counting it told you a rule was in force when it was not. It is now counted under other.
• validate() messages are reworded. The is_valid verdict and the error/warning split are unchanged.

License

MIT.