spatie/robots-txt dependents (26) Order by: name | downloads Show: all | require | require-dev
-
acanto/crawler
Crawl all internal links found on a website
-
PHP
alex-kassel/roach-php-core
Community fork of roach-php/core with on-demand lazy DOM loading and native JSON response parsing
-
-
PHP
bringyourownideas/laravel-sitemap
A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+
-
-
-
dkm/typo3-robots-guard
Centrally manages robots.txt, the robots head meta tag and XML sitemap inclusion for any TYPO3 site (single-site or multisite), with environment-aware safety logic that guards production against accidental crawl-blocking while keeping staging/dev fully disallowed.
-
PHP
haggag/laravel-sitemap
A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+
-
-
-
-
-
-
-