rotassator/silverstripe-subsites-robotstxt

Generate custom robots.txt for subsites

1.0 2017-05-17 05:45 UTC

Requires

• silverstripe/subsites: *

MIT

• Steve Dixon

• robots.txt
• robots
• silverstripe
• subsites

README

Generate custom robots.txt for each subsite.

This module aims to prevent indexing of subsite-specific folder assets that belong to other subsites. It creates a robots.txt file with Disallow rules for folders belonging to other subsites (ie. not folders that are common or for the current subsite).

Installation

composer require rotassator/silverstripe-subsites-robotstxt

Live mode

Set the site to live mode to see subsite-specific robots.txt. On dev or test environments, robots are disallowed for all files.

See Environment management documentation for more details.

Example robots.txt for live site

For example1.com subsite:

# robots.txt for Example 1

User-agent: *
Disallow: assets/example2/
Disallow: assets/example2-documents/

For example2.com subsite:

# robots.txt for Example 2

User-agent: *
Disallow: assets/example1/

Example for non-live site

# robots.txt for Example 1

User-agent: *
Disallow: /