koh / sylius-robots-txt

v1.1.32 2026-10-09 23:08 UTC

Requires

• php: ^8.5.11
• koh/sylius-agent-contracts: ^1.0
• sylius/sylius: ^2.2.9 || ^2.3.0
• symfony/config: ^7.4 || ^8.0
• symfony/console: ^7.4 || ^8.0
• symfony/dependency-injection: ^7.4 || ^8.0
• symfony/http-foundation: ^7.4 || ^8.0
• symfony/http-kernel: ^7.4 || ^8.0
• symfony/routing: ^7.4 || ^8.0

Requires (Dev)

• koh/dev-sylius: ^1.0
• phpstan/phpstan: ^2.3.1
• phpunit/phpunit: ^11.5.57

MIT

• Kai Ole Hartwig

• crawler
• robots.txt
• sylius
• Agent
• ai
• content-signals

Published on Packagist.org on 2026-10-13 21:17 UTC

README

koh/sylius-robots-txt

Channel-aware robots.txt for Sylius, with explicit AI crawler rules and Content Signals.

Three things that usually live in a hand-edited static file, modelled instead: one robots.txt per channel, a named group per AI crawler, and a Content Signal that is emitted both in the file and as a response header.

Install

composer require koh/sylius-robots-txt

// config/bundles.php
Koh\SyliusRobotsTxt\KohSyliusRobotsTxtBundle::class => ['all' => true],

# config/routes/koh_agent.yaml
koh_robots_txt:
    resource: '@KohSyliusRobotsTxtBundle/config/routes.php'
    ignore_errors: not_found

Delete public/robots.txt. The web server answers from disk before Symfony is consulted, so a leftover file makes this bundle silently do nothing. The bundle logs a warning at cache warmup if it finds one.

Configure

Defaults are sensible for a shop that wants to be found and answered from, but not trained on:

# config/packages/koh_robots_txt.yaml
koh_robots_txt:
    disallow: ['/admin', '/_mcp', '/mcp/oauth/']
    sitemaps: ['/sitemap.xml']          # relative paths get expanded per channel

    ai_crawlers:
        allowed: true                   # false ⇒ Disallow: / for every named token
        agents: [GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, …]
        disallow: []                    # extra paths hidden from AI crawlers only

    content_signal:
        search: true                    # may be indexed and linked
        ai_input: true                  # may be retrieved at answer time (RAG)
        ai_train: false                 # may not be used to train a model
        response_header: true           # also emit Content-Signal on HTML pages

Output

User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /
Disallow: /admin
Disallow: /_mcp

# AI crawlers, named explicitly.
# A wildcard group is not read as a policy for these agents.
User-agent: GPTBot
User-agent: OAI-SearchBot
…
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

Sitemap: https://shop.example.com/sitemap.xml

Why the AI crawlers get their own group rather than being folded into *: several of them only honour a policy that names their product token, and scanners check for the tokens specifically. User-agent: * is not read as a statement about GPTBot.

Why the Content Signal also goes out as a header: robots.txt states a policy for a crawl, the header states it for the exact resource that was fetched — which is what a retrieval-time agent sees. Different consumers, same configuration.

Command

bin/console koh:agent:robots:dump                  # print for the first enabled channel
bin/console koh:agent:robots:dump -c WEB_AT        # print for one channel
bin/console koh:agent:robots:dump --write          # write public/robots.txt

--write is for deployments that prefer a static file. It makes the dynamic route unreachable, so re-run it whenever the configuration changes.

Decommission

Remove the composer require, the config/bundles.php entry, the route import and config/packages/koh_robots_txt.yaml. No database table, no cache pool. Restore a static public/robots.txt if you still want one.

Security

Report vulnerabilities privately — see SECURITY.md. Advisories are published as CSAF 2.0 at https://ole-hartwig.eu/.well-known/csaf/.

The family threat model, covering this package, is at https://github.com/ohartwig/sylius-agent-ready/blob/main/docs/threat-model.md.

Licence

MIT — see LICENSE.