X-Robots-Tag HTTP header parser class

v0.2.1 2016-07-15 21:18 UTC

This package is auto-updated.

Last update: 2024-07-05 05:07:53 UTC


Build Status Scrutinizer Code Quality Code Climate Test Coverage License Packagist Gitter

X-Robots-Tag HTTP header parser

PHP class to parse X-Robots-Tag HTTP headers according to Google X-Robots-Tag HTTP header specifications.



Note: HHVM support is planned once facebook/hhvm#4277 is fixed.


The library is available via Composer. Add this to your composer.json file:

    "require": {
        "vipnytt/robotstagparser": "~0.2"

Then run composer update.

Getting Started

Basic example

Get all rules affecting you, this includes the following:

  • All generic rules
  • Rules specific to your User-Agent (if there is any)
use vipnytt\XRobotsTagParser;

$headers = [
    'X-Robots-Tag: noindex, noodp',
    'X-Robots-Tag: googlebot: noindex, noarchive',
    'X-Robots-Tag: bingbot: noindex, noarchive, noimageindex'

$parser = new XRobotsTagParser('myUserAgent', $headers);
$rules = $parser->getRules(); // <-- returns an array of rules

Different approaches:

Get the HTTP headers by requesting an URL

use vipnytt\XRobotsTagParser;

$parser = new XRobotsTagParser\Adapters\Url('', 'myUserAgent');
$rules = $parser->getRules();

Use your existing GuzzleHttp request

use vipnytt\XRobotsTagParser;
use GuzzleHttp\Client;

$client = new GuzzleHttp\Client();
$response = $client->request('GET', '');

$parser = new XRobotsTagParser\Adapters\GuzzleHttp($response, 'myUserAgent');
$array = $parser->getRules();

Provide HTTP headers as an string

use vipnytt\XRobotsTagParser;

$string = <<<STRING
HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindex
X-Robots-Tag: nofollow

$parser = new XRobotsTagParser\Adapters\TextString($string, 'myUserAgent');
$array = $parser->getRules();

Export all rules

Returns an array containing all rules for any User-Agent.

use vipnytt\XRobotsTagParser;

$parser = new XRobotsTagParser('myUserAgent', $headers);
$array = $parser->export();


  • all - There are no restrictions for indexing or serving.
  • none - Equivalent to noindex and nofollow.
  • noindex - Do not show this page in search results and do not show a "Cached" link in search results.
  • nofollow - Do not follow the links on this page.
  • noarchive - Do not show a "Cached" link in search results.
  • nosnippet - Do not show a snippet in the search results for this page.
  • noodp - Do not use metadata from the Open Directory project for titles or snippets shown for this page.
  • notranslate - Do not offer translation of this page in search results.
  • noimageindex - Do not index images on this page.
  • unavailable_after - Do not show this page in search results after the specified date/time.