taurus-media/module-robots

Add filterable attributes to robots.txt in Magento 2

Maintainers

Details

github.com/taurus-media/module-robots

Source

Issues

Installs: 9

Dependents: 0

Suggesters: 0

Security: 0

Stars: 0

Watchers: 0

Forks: 0

Open Issues: 0

Type:magento2-module

pkg:composer/taurus-media/module-robots

1.0.0 2026-01-16 14:17 UTC

Requires

  • magento/framework: *
  • magento/module-robots: ^101.1

MIT 6aa1b0888ba11c5d6da6744516ae5172866864d8

This package is auto-updated.

Last update: 2026-01-16 14:17:29 UTC

README

A Magento 2 module that automatically adds filterable product attributes to the robots.txt file to prevent search engines from crawling layered navigation filter combinations, which can lead to duplicate content issues.

Description

The Taurus_Robots module extends the default Magento 2 Robots functionality. It identifies all product attributes configured to be "Use in Layered Navigation" (filterable) and adds a Disallow rule for each of them under the User-agent: * section.

For example, if you have a filterable attribute color, it will add: Disallow: /*color=*

Features

  • Automatically detects filterable product attributes.
  • Dynamically injects Disallow rules into the generated robots.txt.
  • Sorts rules alphabetically for better readability.
  • Ensures rules are placed under the global User-agent: * block.

Installation

composer require taurus-media/module-robots
bin/magento module:enable Taurus_Robots
bin/magento setup:upgrade
bin/magento setup:static-content:deploy