Simple robots generation module for Silverstripe.
This module provides simple robots.txt generation for Silverstripe, with various configuration options available.
When a site is not in live mode (such as on a testing domain) it will respectively block the entire domain, ensuring that (at least respectful) search engines will refrain from indexing your test site.
Composer is the recommended way of installing SilverStripe modules.
composer require gorriecoe/silverstripe-robots
- silverstripe/cms ^4.0
This module is heavily inspired by silverstripe-robots by Damian Mooyman
You can add a page or pattern to be blocked by adding it to the disallowedUrls configuration
gorriecoe\Robots\Robots: disallowed_urls: - 'mysecretpage.html' - '_private' - 'Documents-and-Settings/Ricky/My-Documents/faxes/sent-faxes'
Also by default, any page with 'ShowInSearch' set to false will also be excluded. This can be useful for hiding auxiliary pages like "thanks for signing up", or error pages.
You can turn this off (if you really absolutely think you need to) using the below.
gorriecoe\Robots\Robots: disallow_unsearchable: false
By default the module will check for a sitemap file in
/sitemap.xml. You can set a custom file location using the below configuration.
gorriecoe\Robots\Robots: sitemap: '/sitemap.xml'
For multiple sitemaps.
gorriecoe\Robots\Robots: sitemap: - '/sitemap_index.xml' - 'http://www.gstatic.com/s2/sitemap.xml'