README

Verify that web crawlers are actually who they claim to be.

composer require jacyimp/crawler-verifier

Usage

use JacyImp\CrawlerVerifier\CrawlerVerifier;

$verifier = new CrawlerVerifier();

$result = $verifier->verify(
    userAgent: $_SERVER['HTTP_USER_AGENT'] ?? '',
    ip: $_SERVER['REMOTE_ADDR'],
);

if ($result->verified) {
    // Genuine crawler.
}

The result contains the claimed crawler and verification method:

$result->verified;
$result->crawler;
$result->method;

use JacyImp\CrawlerVerifier\Crawler;
use JacyImp\CrawlerVerifier\VerificationMethod;

$result->crawler === Crawler::Googlebot;
$result->method === VerificationMethod::IpRange;

Supported crawlers

Crawler	Verification
Googlebot	IP ranges + FCrDNS
Bingbot	IP ranges + FCrDNS
Applebot	IP ranges + FCrDNS
DuckDuckBot	IP ranges
Pinterestbot	FCrDNS
Baiduspider	FCrDNS
GPTBot	IP ranges
OAI-SearchBot	IP ranges
OAI-AdsBot	IP ranges
PerplexityBot	IP ranges
Perplexity-User	IP ranges

Official IP range snapshots are bundled with the package.

Runtime verification never downloads IP ranges. DNS-backed crawlers may perform DNS lookups.

Identify without verifying

$crawler = $verifier->identify(
    $_SERVER['HTTP_USER_AGENT'] ?? '',
);

This only identifies what the User-Agent claims to be.

Verify a known crawler

use JacyImp\CrawlerVerifier\Crawler;

$result = $verifier->verifyCrawler(
    Crawler::Googlebot,
    $_SERVER['REMOTE_ADDR'],
);

Documentation

Requirements

PHP 8.2+
PSR-16 (psr/simple-cache ^3.0)

jacyimp / crawler-verifier

Maintainers

Package info

Statistics

Security