monperrus / crawler-user-agents
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
Package info
github.com/monperrus/crawler-user-agents
Language:Go
pkg:composer/monperrus/crawler-user-agents
Fund package maintenance!
Requires
None
Requires (Dev)
None
Suggests
None
Provides
None
Conflicts
None
Replaces
None
- dev-master
- v1.71.0
- v1.70.0
- v1.69.0
- v1.68.0
- v1.67.0
- v1.66.0
- v1.65.0
- v1.64.0
- v1.63.0
- v1.62.0
- v1.61.0
- v1.60.0
- v1.59.0
- v1.58.0
- v1.57.0
- v1.56.0
- v1.55.0
- v1.54.0
- v1.53.0
- v1.52.0
- v1.51.0
- v1.50.0
- v1.49.0
- v1.48.0
- v1.47.0
- v1.46.0
- v1.45.0
- v1.44.0
- v1.43.0
- v1.42.0
- v1.41.0
- v1.40.0
- v1.39.0
- v1.38.0
- v1.37.0
- v1.36.0
- v1.35.0
- v1.34.0
- v1.33.0
- v1.32.0
- v1.31.0
- v1.30.0
- v1.29.0
- v1.28.0
- v1.27.0
- v1.26.0
- v1.25.0
- v1.24.0
- v1.23.0
- v1.22.0
- v1.21.0
- v1.20.0
- v1.19.0
- v1.18.0
- v1.17.0
- v1.16.0
- v1.15.0
- v1.14.0
- v1.13.0
- v1.12.0
- dev-web-bot-auth-metadata
- dev-fix-silk-false-positive
- dev-copilot/fix-snapchat-user-agent-detection
- dev-clf-filter-binary
- dev-fix-pr436
- dev-fix-pr441
- dev-pr-432
- dev-copilot/fix-no-es-module-support
- dev-copilot/update-pacote-version
This package is auto-updated.
Last update: 2026-11-22 07:46:30 UTC
README
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
- NPM package: https://www.npmjs.com/package/crawler-user-agents
- Go package: https://pkg.go.dev/github.com/monperrus/crawler-user-agents
- PyPi package: https://pypi.org/project/crawler-user-agents/
Each pattern is a regular expression. It should work out-of-the-box wih your favorite regex library.
Sponsor
💼 Using crawler-user-agents in a commercial product? This package is free to use, but it takes real time to maintain and expand. If it's providing value (and it probably is), please consider sponsoring at the commercial tier.
It keeps the project alive and actively maintained. Your company can afford it. 🙏
Install
Direct download
Download the crawler-user-agents.json file from this repository directly.
Javascript
crawler-user-agents is deployed on npmjs.com: https://www.npmjs.com/package/crawler-user-agents
To use it using npm or yarn:
npm install --save crawler-user-agents
# OR
yarn add crawler-user-agents
In Node.js, you can require the package to get an array of crawler user agents.
const crawlers = require('crawler-user-agents'); console.log(crawlers);
Python
Install with pip install crawler-user-agents
Then:
import crawleruseragents if crawleruseragents.is_crawler("Googlebot/"): # do something
or:
import crawleruseragents indices = crawleruseragents.matching_crawlers("bingbot/2.0") print("crawlers' indices:", indices) print( "crawler's URL:", crawleruseragents.CRAWLER_USER_AGENTS_DATA[indices[0]]["url"] )
Note that matching_crawlers is much slower than is_crawler, if the given User-Agent does indeed match any crawlers.
Go
Go: use this package,
it provides global variable Crawlers (it is synchronized with crawler-user-agents.json),
functions IsCrawler and MatchingCrawlers.
Example of Go program:
package main import ( "fmt" "github.com/monperrus/crawler-user-agents" ) func main() { userAgent := "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)" isCrawler := agents.IsCrawler(userAgent) fmt.Println("isCrawler:", isCrawler) indices := agents.MatchingCrawlers(userAgent) fmt.Println("crawlers' indices:", indices) fmt.Println("crawler's URL:", agents.Crawlers[indices[0]].URL) }
Output:
isCrawler: true
crawlers' indices: [237]
crawler' URL: https://discordapp.com
Web Bot Auth
Some records include optional verification metadata for Web Bot Auth, which uses HTTP Message Signatures (RFC 9421):
"verification": [{ "type": "web-bot-auth", "directory": "https://agent.bot.goog/.well-known/http-message-signatures-directory" }]
directory is the canonical HTTPS URL of the operator's JSON Web Key Set. Consumers can match the URL from a request's Signature-Agent header, resolve keys, and verify its Signature/Signature-Input using an RFC 9421 implementation such as Cloudflare's Web Bot Auth libraries. Key rotation is handled by fetching the directory, not by copying keys into this package.
A User-Agent match or a directory URL is not authentication. Only a successfully verified request signature authenticates the signing identity; signatures do not imply authorization or that the User-Agent itself was signed. Some requests may remain unsigned, and hosted browser identities may be shared by many customers.
Contributing
I do welcome additions contributed as pull requests.
The pull requests should:
- contain a single addition
- specify a discriminant relevant syntactic fragment (for example "totobot" and not "Mozilla/5 totobot v20131212.alpha1")
- contain the pattern (generic regular expression), the discovery date (year/month/day) and the official url of the robot
- result in a valid JSON file (don't forget the comma between items)
Example:
{
"pattern": "rogerbot",
"addition_date": "2014/02/28",
"url": "http://moz.com/help/pro/what-is-rogerbot-",
"instances" : ["rogerbot/2.3 example UA"],
"tags": ["seo"]
}
License
The list is under a MIT License. The versions prior to Nov 7, 2016 were under a CC-SA license.
Related work
There are a few wrapper libraries that use this data to detect bots:
- Voight-Kampff (Ruby)
- isbot (Ruby)
- crawlers (Clojure)
- isBot (Node.JS)
Other systems for spotting robots, crawlers, and spiders that you may want to consider are:
- Crawler-Detect (PHP)
- BrowserDetector (PHP)
- browscap (JSON files)