hryvinskyi / magento2-bot-verification-google

1.0.0 2026-10-30 18:01 UTC

Requires

• php: >=8.1
• hryvinskyi/magento2-bot-verification: ^1.0
• magento/framework: >=103.0.0

MIT

• Volodymyr Hryvinskyi

Published on Packagist.org on 2026-10-30 20:59 UTC

README

Google's crawlers and their published address lists for hryvinskyi/magento2-bot-verification. The package is configuration only: one etc/di.xml.

composer require hryvinskyi/magento2-bot-verification-google
bin/magento module:enable Hryvinskyi_BotVerificationGoogle
bin/magento setup:upgrade
bin/magento bot-verification:refresh

It declares two crawlers.

Crawler	User-Agent tokens	Checked against
google	Googlebot, Storebot-Google, Google-InspectionTool, GoogleOther, Google-CloudVertexBot, AdsBot-Google, Mediapartners-Google, APIs-Google, DuplexWeb-Google, Google-Safety	common-crawlers.json, special-crawlers.json, user-triggered-fetchers-google.json, user-triggered-agents.json
google_fetcher	FeedFetcher-Google, Google-Read-Aloud, Google-Site-Verification, GoogleProducer, Google-NotebookLM, Google-Pinpoint	user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json

The lists are published at https://developers.google.com/static/crawling/ipranges/.

Googlebot and the special-case crawlers are checked against every list Google publishes as its own infrastructure, not only the one its documentation names for each crawler. In a production shop's traffic every real Storebot-Google request came from user-triggered-fetchers-google.json rather than common-crawlers.json, where the documentation places it. user-triggered-fetchers.json holds the addresses of Google Cloud customers fetching on a user's behalf, so it vouches for the fetchers only, never for Googlebot: a scraper on a Google Cloud machine stays an impostor.

License

MIT