spatie/robots-txt

Determine if a page may be crawled from robots.txt and robots meta tags

2.0.2 2022-05-18 15:14 UTC

This package is auto-updated.

Last update: 2022-05-18 15:15:07 UTC


README

68747470733a2f2f6769746875622d6164732e73332e65752d63656e7472616c2d312e616d617a6f6e6177732e636f6d2f737570706f72742d756b7261696e652e7376673f743d31

Parse robots.txt, robots meta and headers

Latest Version on Packagist GitHub Tests Action Status Quality Score Total Downloads

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers.

Support us

68747470733a2f2f6769746875622d6164732e73332e65752d63656e7472616c2d312e616d617a6f6e6177732e636f6d2f726f626f74732d7478742e6a70673f743d31

We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products.

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall.

Installation

You can install the package via composer:

composer require spatie/robots-txt

Usage

$robots = Spatie\Robots\Robots::create();

$robots->mayIndex('https://www.spatie.be/nl/admin');

$robots->mayFollowOn('https://www.spatie.be/nl/admin');

You can also specify a user agent:

$robots = Spatie\Robots\Robots::create('UserAgent007');

By default, Robots will look for a robots.txt file on https://host.com/robots.txt. Another location can be specified like so:

$robots = Spatie\Robots\Robots::create()
    ->withTxt('https://www.spatie.be/robots-custom.txt');

$robots = Spatie\Robots\Robots::create()
    ->withTxt(__DIR__ . '/public/robots.txt');

Testing

composer test

Changelog

Please see CHANGELOG for more information what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Postcardware

You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.

Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium.

We publish all received postcards on our company website.

Credits

License

The MIT License (MIT). Please see License File for more information.