README

A PHP library for parsing and auditing robots.txt — from a URL, a file, or a string — including the robots <meta> tags and X-Robots-Tag headers that govern indexing alongside it.

Built for analysis rather than crawling. It reports what a document says, why a given rule applies to a given URL, and — through the audit — what any of it means for a site's visibility.

use Leopoletto\RobotsTxtParser\RobotsTxtParser;

$response = (new RobotsTxtParser())
    ->withBotSignature('MyBot', '1.0', 'https://example.com/bot')
    ->parseUrl('https://example.com/products/widget');

$response->isAllowed('GPTBot', '/products/widget');   // false
$response->records()->userAgents();                   // list<UserAgent>
$response->records()->groups();                       // list<Group>

Then ask what it means:

use Leopoletto\RobotsTxtParser\Audit\Auditor;

foreach ((new Auditor())->audit($response)->actionable() as $finding) {
    $finding->title;    // "3 of 3 user-triggered AI fetches are blocked"
    $finding->impact;   // why that costs something
    $finding->fix;      // what to do about it
}

Installation

composer require leopoletto/robots-txt-parser

Requires PHP 8.2+. The only runtime dependencies are Guzzle and PSR-7 — no framework.

Parsing

The three entry points behave identically. The same document produces the same records, the same line numbers and the same issues whether it was fetched, uploaded or pasted.

$parser = new RobotsTxtParser();

$parser->parseText($contents);
$parser->parseFile('/path/to/robots.txt');

$parser->withBotSignature('MyBot', '1.0', 'https://example.com/bot')
       ->parseUrl('https://example.com/some/page');

Identifying your bot

parseUrl() makes real HTTP requests, so it requires an identity:

$parser->withBotSignature('MyBot', '1.0', 'https://example.com/bot');
// Mozilla/5.0 (compatible; MyBot/1.0; https://example.com/bot)

$parser->withUserAgent('MyBot/1.0 (+https://example.com/bot)');

The product token (MyBot) is also what the parser checks its own access against — see Fetching behaviour.

Keeping the original document

$response = $parser->keepContent()->parseText($contents);
$response->content(); // the document, with line endings normalised to \n

Reading a document

$response->records() returns a Document. Every method on it is pure — nothing carries state between calls.

$document = $response->records();

$document->userAgents();        // list<UserAgent>
$document->groups();            // list<Group>
$document->sitemaps();          // list<Sitemap>
$document->comments();          // list<Comment>
$document->issues();            // list<Issue>
$document->headerDirectives();  // list<HeaderDirective>
$document->metaDirectives();    // list<MetaDirective>

$document->allowed();           // every Allow rule
$document->disallowed('GPTBot');// Disallow rules governing GPTBot
$document->crawlDelay('GPTBot');

$document->toArray();           // JSON-ready summary of everything

Every record exposes line() and toArray(), so rendering a document is uniform.

Groups

A group is one or more consecutive User-agent: lines plus the directives that follow, per RFC 9309 §2.2.1.

User-agent: GPTBot
User-agent: ChatGPT-User
Disallow: /admin

Both agents share the one rule, and the parser models that directly:

$group = $document->groupFor('ChatGPT-User');
$group->tokens();            // ['GPTBot', 'ChatGPT-User']
$group->directives();        // the rules governing both

Comments and sitemaps between User-agent: lines do not break the run. A directive does.

Checking access

$document->isAllowed('GPTBot', '/admin/reports');   // false

$decision = $document->decide('GPTBot', '/admin/reports');
$decision->allowed;     // false
$decision->rule->line;  // 4 — the rule that decided it
$decision->rule->value; // '/admin'
$decision->byDefault(); // false; true when no rule matched

Resolution follows RFC 9309 §2.2.2:

Groups naming the agent apply; all of them, if the document declares it more than once.
Only when no group names the agent does the * group apply.
The longest matching pattern wins.
At equal length the least restrictive rule wins — Allow beats Disallow.
Disallow: with an empty value forbids nothing.
Matching is case-insensitive on the agent, case-sensitive on the path.

Patterns support * (any run of characters) and a trailing $ (end anchor). Everything else is literal, including a $ that is not the final character.

Explaining a rule

$explanation = $document->disallowed('*')[0]->explanation();

$explanation->specificity;  // 6
$explanation->wildcards;    // 0
$explanation->endAnchor;    // false
$explanation->pathToMatch;  // 'Prefix match: any URL whose path starts with "/admin"…'

Explanations are built on first access, so a document with thousands of rules costs nothing unless something reads them. The structural fields are there for callers who want to write their own wording or translate it.

Auditing

Parsing answers what does this file say. The audit answers what does that mean for this site, and what should change — which is the question people actually arrive with.

use Leopoletto\RobotsTxtParser\Audit\Auditor;

$report = (new Auditor())->audit($response);

$report->worst();       // Status::Warning — the worst status worth acting on
$report->counts();      // ['critical' => 0, 'warning' => 3, 'notice' => 4, 'info' => 5, 'pass' => 2]
$report->actionable();  // everything that is not a pass, worst first
$report->find('blanket-block');
$report->toArray();     // JSON-ready

Every finding answers four questions in order:

$finding->title;      // "6 of 6 AI training crawlers are blocked"
$finding->summary;    // what is the case
$finding->intent;     // what this achieves, if it was deliberate
$finding->impact;     // what it costs, either way
$finding->fix;        // what to change if it was not deliberate
$finding->evidence;   // the lines it points at
$finding->crawlers;   // per-crawler detail, for crawler-access findings
$finding->status;     // Status::Info

The intent field is the one most robots.txt tooling skips, and the reason so much of it reads as scolding. A file that turns away every AI crawler is not broken; a report that cannot say so is not much use.

Status

Severity answers how confident the audit is that something is unintended — never how expensive a rule would be if it were a mistake. What a rule costs belongs in impact, where the reader can weigh it against their own intent, rather than in a badge that decides for them.

Status	Meaning	Example
`Critical`	Cannot be what anyone intended	`Disallow: /` with nothing overriding it
`Warning`	Internally incoherent — the rule did not hit its target	A user-triggered fetch blocked while the same operator collects content
`Notice`	Deliberate-looking, with a consequence worth stating	Rules that can never take effect
`Info`	A policy readout, with no judgement	N of M search engines are blocked
`Pass`	Nothing to do
`NotChecked`	The input was never observed, so nothing is claimed; the summary says why	The page answered HTTP 403

Info, Pass and NotChecked are not actionable: isActionable() is false for all three, actionable() omits them, and worst() ignores them. So a site that blocks every AI crawler on purpose reports Pass overall, with the full policy readout still there to read.

Blocking a crawler is a choice a site is entitled to make, so CrawlerAccessCheck never grades it. The genuinely diagnostic signal lives in CrawlerCoherenceCheck, which compares one operator's crawlers against each other: blocking OpenAI's ChatGPT-User while allowing GPTBot donates the content in bulk and refuses the person who asked for the page — the opposite of either intent, and almost always a rule that named the wrong user agent.

What it checks

Check	Looks for
`CrawlerAccessCheck`	Whether notable crawlers can reach the site, grouped by purpose
`CrawlerCoherenceCheck`	One operator's crawlers contradicting each other
`BlanketRuleCheck`	`Disallow: /` for everyone, a default-deny allowlist, or a file that restricts nothing
`IndexingDirectiveCheck`	Page `noindex` reconciled against robots.txt
`SitemapCheck`	Declared, absolute, reachable, and not blocked by this same file
`SyntaxCheck`	Lines crawlers will skip
`DeprecatedDirectiveCheck`	`Crawl-delay`, `Noindex:` — directives that no longer do what their author expects
`SensitivePathCheck`	Paths the file gives away by naming them: secrets and data files, names nobody would have guessed, tool panels
`PrecedenceCheck`	Contradictions, shadowed rules, duplicates
`FileSizeCheck`	The 500 KB ceiling crawlers enforce

The single most expensive misunderstanding in the subject gets its own check: Disallow and noindex are not the same instruction, and combining them produces the opposite of what is intended. A crawler forbidden to fetch a page never reads its noindex, so the URL stays eligible for search — it just gets listed without a description.

Crawler groups

Which crawlers are worth reporting on, and how they are grouped, follows Cloudflare Radar, which measures observed crawler traffic rather than estimating importance. The useful idea taken from it is purpose, because the same mechanism has very different consequences:

Group	Blocking them costs
Search engines	Removal from search. Reported as `Critical`.
AI answer engines	Citation traffic in AI-generated answers
User-triggered AI fetches	A person asked for the page and gets an error
Social link previews	Shared links render as bare URLs
AI training crawlers	Nothing but the content. Reported as `Notice`, never a fault.

That last row matters: blocking training crawlers is the most common configuration on the web and an entirely legitimate editorial choice, so the audit reports it without calling it a problem.

Validating sitemaps

A declared sitemap that 404s is worse than none — it looks handled. Pass a probe to fetch each one and check it responds and opens as a urlset or sitemapindex:

use GuzzleHttp\Client;
use Leopoletto\RobotsTxtParser\Audit\SitemapProbe;

$auditor = new Auditor(probe: new SitemapProbe(
    new Client(['http_errors' => false]),
    'Mozilla/5.0 (compatible; MyBot/1.0; https://example.com/bot)',
));

Without a probe the sitemap checks still run; they just do not make requests.

Each result carries a SitemapOutcome. Only an answer is judged: a 4xx or 5xx is sitemap-unreachable, and a 2xx that does not open as XML is sitemap-not-xml. A sitemap that was not seen — blocked for your bot, skipped by a limit, a timeout, or an error with no answer — is listed under one sitemap-not-checked finding (NotChecked), with the reason for each, and is not held against the file. SitemapProbeResult::blocked() and ::skipped() build those results.

A robots.txt speaks only for its own origin, so SitemapProbe does not fetch a sitemap on another origin unless told whether that origin's robots.txt allows it. The package never fetches another origin's robots.txt itself; pass a decision:

$probe = new SitemapProbe($client, $userAgent, decideElsewhere: function (string $sitemapUrl): ?bool {
    return $myOtherOriginCheck($sitemapUrl); // true: fetch, false: blocked, null: could not tell
});

Without one, such a sitemap is skipped as "on a different origin". The audit scopes any prober implementing Contract\OriginScopedSitemapProber to the origin of the response's robots.txt.

The auditor takes any Contract\SitemapProber; SitemapProbe is the one that makes requests. To audit a stored file again without the network, keep what the probe found and answer from it: SitemapProbeResult::toArray() writes a result out and SitemapProbeResult::fromArray() reads it back.

The page's own directives work the same way. Response::pageProbe() holds what the request for the page found; store its toArray(), and later hand a Contract\PageProber that returns it to withPageProber() and call parseTextForUrl($storedRobotsTxt, $url). Whether the page may be requested is decided again from the stored file each time, so only what came from the network is kept.

Posture at a glance

Alongside the findings, the report groups every user agent the file names by what that crawler is for. On a file that declares dozens of agents, reading them one by one says nothing — grouped, the policy states itself:

foreach ($report->breakdown->categories as $tally) {
    printf("%-24s %s\n", $tally->category, $tally->describe());
}

$report->breakdown->fullyBlocked();   // categories where nothing is allowed
$report->breakdown->fullyAllowed();

Search Engine Crawler    all 17 allowed
AI Data Scraper          all 9 blocked
AI Assistant             all 5 blocked
Undocumented AI Agent    all 3 blocked

Adding a check

Checks are independent and additive, so a new question never means editing the report:

use Leopoletto\RobotsTxtParser\Contract\AuditCheck;
use Leopoletto\RobotsTxtParser\Response;

final class HostCheck implements AuditCheck
{
    /** @return list<\Leopoletto\RobotsTxtParser\Audit\Finding> */
    public function run(Response $response): array
    {
        return [];
    }
}

$auditor = new Auditor([new HostCheck(), /* … */]);

Issues

Malformed and ineffective lines are reported rather than silently dropped — a misspelled Dissalow: is ignored by every real crawler, which is exactly the mistake worth surfacing.

foreach ($document->issues() as $issue) {
    $issue->line;             // 12
    $issue->type;             // 'unknown_directive'
    $issue->severity;         // Severity::Medium
    $issue->message;
}

Nothing is dropped in silence. Every line is either parsed into a record or reported as an issue, so a reader can account for all of them.

Defects in the document

Type	Severity	Meaning
`malformed_line`	high	A line with no `:` separator
`orphan_directive`	high	A rule declared before any `User-agent:`
`unknown_directive`	medium	A field no crawler acts on, usually a typo
`ineffective_directive`	medium	A field crawlers recognise but no longer honour, such as `Noindex:`
`invalid_path`	medium	An `Allow`/`Disallow` value not starting with `/`
`invalid_value`	medium	A non-numeric `Crawl-delay`
`empty_sitemap`	medium	A `Sitemap:` line with no URL
`nonstandard_directive`	low	A published extension outside the standard — see below

About the request, not the file

Type	Meaning
`fetch_failed`	The robots.txt could not be retrieved
`too_many_redirects`	The redirect chain exceeded the limit
`empty_response`	The response carried no body
`truncated`	The document exceeded the size limit
`page_disallowed`	The page was not fetched because robots.txt forbade it
`robots_missing`	robots.txt answered 404 or 410: there is no file, so every path is allowed
`robots_refused`	robots.txt answered another 4xx (not 429): the site turned the bot away
`robots_unreachable`	robots.txt answered 5xx or 429, so every path counts as disallowed

fetch_failed, too_many_redirects and empty_response mean no body was read, and each sets $response->error(), so a response with no error always carries a file that was actually read.

Note the split. A page_disallowed note says nothing about whether the file is correct, so counting it as an error would report a fault against a valid document.

Non-standard but widely published fields — Host, Clean-param, Request-rate, Visit-time and Cloudflare's Content-Signal — are recorded at the lowest severity rather than faulted. They are real, and worth knowing about, but they are not mistakes.

Meta tags and X-Robots-Tag

parseUrl() also collects the indexing directives that live outside robots.txt.

foreach ($document->metaDirectives() as $meta) {
    $meta->name;                    // 'robots'
    $meta->validation->has('noindex');
    $meta->validation->conflicts;   // e.g. index + noindex
}

foreach ($document->headerDirectives() as $header) {
    $header->userAgent;   // '*' or 'googlebot'
    $header->origin;      // 'robots.txt' or 'page'
}

Both sources combine into one answer, resolving toward the more restrictive reading:

$rules = $document->effectiveRules();
$rules->indexable();   // false if anything said noindex
$rules->followable();
$rules->toArray();     // full effective_rules + source counts

Fetching behaviour

Two rules govern parseUrl(), both deliberate:

The robots.txt comes from the origin; the meta tags come from your URL. robots.txt is per-origin, so it is always fetched from scheme://host[:port]/robots.txt — the port included. Meta tags and X-Robots-Tag are per-page, so those are read from the exact URL you passed, never the home page.

If robots.txt disallows your URL, the page is not fetched. A tool that reports on robots.txt has no business ignoring one. When this happens you get a page_disallowed issue and:

$response->pageInspected();          // false
$response->pageDecision()->allowed;  // false
$response->pageDecision()->rule;     // the rule that blocked it

What the robots.txt status means. RFC 9309 governs what each answer implies:

robots.txt answered	Treated as	Body parsed	Page fetched	`error()`
2xx	The file	Yes	If allowed	No
404, 410	No file: every path allowed (§2.3.1.3), `robots_missing`	No	If allowed	No
401, 403, other 4xx	A refusal, `robots_refused`	No	No	Yes
429, 5xx	Unreachable: every path disallowed (§2.3.1.4), `robots_unreachable`	Yes	No	No

The RFC reads every 4xx as "no file". This package reads 401, 403 and the rest as a refusal instead, so a bot that was turned away does not go on to request the page.

A page that was not read is not audited. Directives are read only from a page that answered 2xx and whose body arrived whole. A 4xx or 5xx, a timeout, or a body that fails partway leaves the probe unobserved: nothing is read from it, not even its X-Robots-Tag headers, and the audit reports indexing-not-observed as NotChecked instead of passing on a page it never saw.

$response->pageProbe()->observed();  // false for a 403, a 5xx or a timeout
$response->pageProbe()->truncated;   // true when maxHtmlBytes cut the head before </head>
$response->pageProbe()->challenge;   // true when a bot challenge answered instead of the page

A bot challenge or block page (Cloudflare, Akamai, HUMAN, DataDome) answered in place of the page is recognised by markers only those pages carry, and is not read: its own noindex is not the site's. A CDN's name alone is never taken as a challenge.

Meta tags inside HTML comments, <script> and <noscript> are ignored, since no browser applies them. Tags added by JavaScript at runtime are not seen, because the page is not rendered.

HTTP metadata

$response->robotsUrl();      // https://example.com/robots.txt
$response->requestedUrl();   // https://example.com/products/widget
$response->finalUrl();       // after redirects
$response->redirects();
$response->statusCode();     // the robots.txt response
$response->pageStatusCode(); // the page response, if fetched
$response->error();
$response->size();           // bytes of robots.txt read
$response->truncated();

A failed fetch is a value, not an exception: a site with no robots.txt is a finding to report.

Limits

Defaults follow what real crawlers do, not what is technically possible. Google reads at most 500 KB of a robots.txt, so content past that is ignored and flagged rather than parsed into rules nothing will honour.

use Leopoletto\RobotsTxtParser\Http\HttpConfiguration;

$parser = new RobotsTxtParser(new HttpConfiguration(
    maxBytes: 500 * 1024,
    maxHtmlBytes: 1024 * 1024,
    maxRedirects: 5,
    robotsTimeout: 10,
    pageTimeout: 10,
    connectTimeout: 5,
));

robotsTimeout and pageTimeout cap each request's total duration, body included, measured from when the request starts: a server that sends a byte every few seconds is cut off at the limit rather than awaited. A robots.txt cut off that way, or one whose body fails partway, is a failed fetch with error() set, never parsed as a shorter file. connectTimeout is honoured by curl-backed clients; Guzzle's stream handler, which serves the streamed requests the default client makes, ignores it and bounds the connection by the request timeout instead.

A file cut at maxBytes keeps only its complete lines: the fragment after the last line break is not parsed, since Disallow: /private-area cut to Disallow: /priv would be a broader rule than the site wrote. A file of exactly maxBytes is not truncated. Pass the same limit to the auditor so the size check judges against it:

$report = (new Auditor(maxBytes: 100 * 1024))->audit($response);

The agent dataset

User agents are enriched from a bundled dataset of ~1,800 known crawlers, sharded by leading letter so a lookup reads one file rather than decoding the whole set.

$userAgent = $document->userAgents()[0];
$userAgent->token;                // 'GPTBot' — exactly as declared
$userAgent->agent?->category;     // 'AI Data Scraper'
$userAgent->agent?->description;  // null — see below

The dataset currently carries names, categories and paths only. description is null for every agent until descriptions written for this package are added.

Skip the dataset entirely when you do not need it:

use Leopoletto\RobotsTxtParser\Agents\NullAgentRepository;

new RobotsTxtParser(agents: new NullAgentRepository());

Refreshing it

The roster and categories come from knownagents.com; its descriptive text is not collected. The roster is read from that site's sitemap rather than its paginated listing, so one request replaces thirty-six and nothing is missed to pagination shifting mid-crawl. Detail pages are fetched only for agents missing locally, so a routine sync costs one request per genuinely new crawler.

It also dogfoods the library: the sync reads knownagents.com's own robots.txt through this parser and stops if it is not permitted, honours any declared Crawl-delay, and paces its requests.

composer agents:sync -- --dry-run          # report what would change
composer agents:sync -- --limit=25         # cap the fetches while trying it out
composer agents:sync                       # fetch new agents
composer agents:build                      # rebuild the shards

The crawler list

Separate from the agent dataset, and much shorter: the crawlers the audit reports on, in src/data/crawlers.json, with the prose explaining what blocking each group costs.

Selection and grouping follow Cloudflare Radar. Traffic share moves month to month, so no ranking is stored — only membership and purpose, which are stable.

Radar publishes no JSON for its directory, so regenerating is two steps. Paste bin/radar-scrape.js into the browser console on radar.cloudflare.com/bots/directory; it pages through the grid, extracts each card and downloads radar-crawlers.json. Then:

composer crawlers:import -- --input=radar-crawlers.json --dry-run
composer crawlers:import -- --input=radar-crawlers.json

The importer maps Radar's purpose labels onto the report's groups and reports anything it cannot place rather than filing it somewhere plausible — a crawler in the wrong group would produce confidently wrong advice.

Extending the parser

Each directive is handled by its own LineParser, consulted in order:

use Leopoletto\RobotsTxtParser\Contract\LineParser;
use Leopoletto\RobotsTxtParser\Parsing\ParseContext;
use Leopoletto\RobotsTxtParser\Parsing\Token;

final class HostParser implements LineParser
{
    public function supports(Token $token): bool
    {
        return $token->fieldIs('host');
    }

    public function parse(Token $token, ParseContext $context): array
    {
        return [new HostRecord($token->number, $token->value)];
    }
}

$parser = new DocumentParser($agents, [
    new CommentParser(),
    new HostParser(),
    // …the catch-all must come last
    new UnknownFieldParser(),
]);

Input sources are equally pluggable — implement Contract\Source to parse from anywhere.

Development

composer test        # phpunit
composer phpstan     # level 8
composer format      # php-cs-fixer
composer quality     # all three

Migrating from wizardcompass/robots-txt-parser

wizardcompass/robots-txt-parser is retired in favour of this package, which has a different public API. Its 2.0.0 release is a compatibility layer that delegates here, so existing installs keep working while they migrate. Changes between releases of this package are listed in CHANGELOG.md.

Credits

Leonardo Poletto
Crawler roster and categories from Known Agents
Crawler selection and purpose classification from Cloudflare Radar

License

The source code is released under the MIT License (MIT). See LICENSE.md.

The bundled datasets in src/data/ are derived from third-party sources and are not covered by the MIT License. See src/data/NOTICE.md.

leopoletto / robots-txt-parser

Package info

Security

Statistics