tekord/robots-txt-provider

Provides various framework-agnostic ways to generate the contents of the robots.txt file

1.0 2021-06-06 17:06 UTC

Requires

• php: ^7.4|^8.0

Requires (Dev)

• phpunit/phpunit: ^9.3

MIT

• Cyrill Tekord

• search
• seo
• robots.txt
• robots
• indexing

README

This package provides various framework-agnostic ways to generate the contents of the robots.txt file.

Installation

Install the package via Composer:

composer require tekord/robots-txt-provider

Usage

Here are the available content providers:

• StringContentProvider - returns a plain string
• FileContentProvider - returns the contents of the specified file
• CallbackContentProvider - used a callback to get the value
• CompositeContentProvider - used to compose multiple providers, returns the value of the first provider that returns a non-NULL value

ContentBuilder

The ContentBuilder class provides handy methods for generating content:

ContentBuilder::make()
    ->line("User-Agent: *")
    ->emptyLine()
    ->comment("This is a comment")
    ->parameter("Host", "https://example.com")
    ->allow("/about")
    ->disallow("/login")
    ->build();

Will return the following:

User-Agent: *

# This is a comment
Host: https://example.com
Allow: /about
Disallow: /login

The following methods have a conditional version: line, emptyLine, comment, parameter, allow and disallow. Just add If to the end of the method name like lineIf. The first parameter is a condition (boolean or callback). If the condition is TRUE then the method executes. For instance:

$debugMode = true;

$useSpecialUserAgent = function () {
    return false;
};

ContentBuilder::make()
    ->lineIf($useSpecialUserAgent, "User-Agent: Bot/1.0")
    ->lineIf(!$useSpecialUserAgent, "User-Agent: *")
    ->emptyLine()
    ->commentIf($debugMode, "Debug mode is active")
    ->parameter("Host", "https://example.com")
    ->disallow("/")
    ->build();

Will return the following:

User-Agent: *

# Debug mode is active
Host: https://example.com
Disallow: /

StringContentProvider

$content = <<<'TXT'
User-Agent: *
Disallow: /
TXT;

$contentProvider = new StringContentProvider($content);

FileContentProvider

$contentProvider = new FileContentProvider(__DIR__ . "/storage/static/default-robots.txt");

CallbackContentProvider

$contentProvider = new CallbackContentProvider(function() {
    return ContentBuilder::make()
        ->line("User-Agent: *")
        ->emptyLine()
        ->comment("This content was generated by the CallbackContentProvider class")
        ->parameter("Host", "https://example.com")
        ->disallow("/")
        ->build();
});

CompositeContentProvider

$productionFileContentProvider = new FileContentProvider(__DIR__ . "/public/robots.production.txt");
$defaultFileContentProvider = new FileContentProvider(__DIR__ . "/public/robots.default.txt");
$fallbackFileContentProvider = new StringContentProvider("User-Agent: *\nDisallow: /");

$compositeContentProvider = (new CompositeContentProvider())
    ->addContentProvider($productionFileContentProvider)
    ->addContentProvider($defaultFileContentProvider)
    ->addContentProvider($fallbackFileContentProvider);

Testing

composer test

Security

If you discover any security related issues, please email [e-mail] instead of using the issue tracker.