bnomei/kirby3-robots-txt

Manage the robots.txt from the Kirby config file

1.7.0 2023-08-02 10:01 UTC

Requires

• php: >=8.0.0
• getkirby/composer-installer: ^1.1

Requires (Dev)

• getkirby/cms: ^3.5
• php-coveralls/php-coveralls: ^2.4
• phpunit/phpunit: ^9.5

Suggests

• bnomei/kirby3-feeds: Adds a xml sitemap which will be automatically be included in the robots.txt

MIT

• Bruno Meilick

• robots-txt
• kirby3
• kirby3-cms
• kirby3-plugin
• robots-exclusion-protocol

README

Manage the robots.txt from the Kirby config file.

Similar Robots.txt Plugins

• kirby3-helpers
• kirby3-wellknown

Related SEO Plugins

• kirby3-seo
• kirby3-helpers
• kirby-meta-knight

Commercial Usage

Support open source!

This plugin is free but if you use it in a commercial project please consider to sponsor me or make a donation.
If my work helped you to make some cash it seems fair to me that I might get a little reward as well, right?

Be kind. Share a little. Thanks.

‐ Bruno
 

M	O	N	E	Y
Github sponsor	Patreon	Buy Me a Coffee	Paypal dontation	Hire me

Installation

• unzip master.zip as folder site/plugins/kirby3-robots-txt or
• git submodule add https://github.com/bnomei/kirby3-robots-txt.git site/plugins/kirby3-robots-txt or
• composer require bnomei/kirby3-robots-txt

Staging Server? Debug Mode = Disallow all

When you set the global Kirby debug config to true the plugin will disallow all indexing for all user-agents. This is especially useful on a staging server but you could consider xml-sitemap and rss-feed among other things as well.

⚠️⚠️⚠️ THIS MEANS IF YOU HAVE KIRBY'S DEBUG MODE ENABLED IN PRODUCTION ALL SEARCH ENGINES WILL BE BLOCKED FROM INDEXING YOUR SITE!

Adding Sitemap Link to Robots.txt

This plugin will add the sitemap link automatically to the robots.txt file for most available SEO plugins. You can skip setting the bnomei.robots-txt.sitemap config value to sitemap.xml in that case.

Setup

The plugin generates automatic defaults for the starterkit. You do not have to enter them in the config file. But if you would it would look like this.

defaults for starterkit

<?php
return [
    'bnomei.robots-txt.content' => null, // string or callback
    'bnomei.robots-txt.sitemap' => null, // null (aka. AUTOMATIC) or string or callback
    'bnomei.robots-txt.groups' => [ // array or callback
        '*' => [ // user-agent
            'disallow' => [
                '/kirby/',
                '/site/',
                '/cdn-cgi/',
            ],
            'allow' => [
                '/media/',
            ]
        ]
    ]
];

using a plain string

<?php
return [
    'bnomei.robots-txt.content' => 'user-agent: *
disallow: /kirby/
disallow: /site/
disallow: /cdn-cgi/
allow: /media/',
];

using a callback

<?php
return [
    'bnomei.robots-txt.content' => function() {
        return site()->myRobotsTxtContentField()->value();
    },
];

sitemap and multiple user-agents

<?php
return [
    'bnomei.robots-txt.sitemap' => 'sitemap.xml',
    'bnomei.robots-txt.groups' => [
        '*' => [
            'disallow' => [
                '/',
            ],
        ],
        'googlebot-images' => [
            'allow' => [
                '/media/',
            ]
        ]
    ]
];

Disclaimer

This plugin is provided "as is" with no guarantee. Use it at your own risk and always test it yourself before using it in a production environment. If you find any issues, please create a new issue.

License

MIT

It is discouraged to use this plugin in any project that promotes racism, sexism, homophobia, animal abuse, violence or any other form of hate speech.