crwlr/robots-txt

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Fund package maintenance!
otsch

Installs: 18 199

Dependents: 1

Suggesters: 0

Security: 0

Stars: 11

Watchers: 1

Forks: 2

Open Issues: 0

pkg:composer/crwlr/robots-txt

v1.1.2 2025-01-27 17:32 UTC

This package is auto-updated.

Last update: 2025-09-27 19:02:06 UTC


README

crwlr.software logo

Robots Exclusion Standard/Protocol Parser

for Web Crawling/Scraping

Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Documentation

You can find the documentation at crwlr.software.

Contributing

If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).