Determine if a page may be crawled from robots.txt and robots meta tags

1.0.5 2019-08-07 22:29 UTC

This package is auto-updated.

Last update: 2020-01-07 23:28:51 UTC


Latest Version on Packagist Build Status Quality Score Total Downloads StyleCI

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers.


You can install the package via composer:

composer require spatie/robots-txt


$robots = Robots::create();



You can also specify a user agent:

$robots = Robots::create('UserAgent007');

By default, Robots will look for a robots.txt file on Another location can be specified like so:

$robots = Robots::create()

$robots = Robots::create()
    ->withTxt(__DIR__ . '/public/robots.txt');


composer test


Please see CHANGELOG for more information what has changed recently.


Please see CONTRIBUTING for details.


If you discover any security related issues, please email instead of using the issue tracker.


You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.

Our address is: Spatie, Samberstraat 69D, 2060 Antwerp, Belgium.

We publish all received postcards on our company website.


Support us

Spatie is a webdesign agency based in Antwerp, Belgium. You'll find an overview of all our open source projects on our website.

Does your business depend on our contributions? Reach out and support us on Patreon. All pledges will be dedicated to allocating workforce on maintenance and new awesome stuff.


The MIT License (MIT). Please see License File for more information.