README

Re-announce a site's URLs to Yandex, Bing and the other IndexNow engines from its own sitemap: a sitemap index, gzip-compressed and text sitemaps are streamed entry by entry and submitted in batches, so a million-URL sitemap never lives in memory. The sitemap command of every framework adapter of the family (indexnowkit/symfony-bundle, laravel, yii2) is this package; in plain PHP it is three lines over indexnowkit/core.

Русская версия

Install

composer require indexnowkit/sitemap        # brings indexnowkit/core; needs ext-xmlreader, ext-zlib for .gz

With a framework adapter you install nothing: the adapter requires this package and registers the command (bin/console indexnow:sitemap, php artisan indexnow:sitemap, php yii indexnow/sitemap) with its sitemap configuration block.

Plain PHP

use IndexNowKit\Config;
use IndexNowKit\IndexNowKit;
use IndexNowKit\Sitemap\SitemapConfig;
use IndexNowKit\Sitemap\SitemapReader;

$kit = IndexNowKit::create(Config::fromEnv());
$reader = SitemapReader::fromConfig(SitemapConfig::fromArray(['spool' => 'auto']), $kit->transport);

$batch = [];
foreach ($reader->read('https://www.example.com/sitemap.xml', new DateTimeImmutable('-1 day')) as $entry) {
    $batch[] = $entry->url;
    if (\count($batch) === $kit->config->batchMaxUrls) {
        $kit->submit($batch);
        $batch = [];
    }
}
$kit->submit($batch);

read() yields SitemapEntry objects (url, lastmod), optionally only those whose <lastmod> is newer than $changedSince (entries without lastmod are then skipped). The root may be an http(s) URL, a local path or a file:// URL; nested sitemaps of an index are fetched over the transport you pass (the one the facade submits through, so http.client and http.timeout apply). $kit->transport is null when the facade was built around a custom submitter: use Http\TransportFactory::lazy($kit->config) then.

Configuration

SitemapConfig::fromArray() reads the sitemap block every adapter exposes; SitemapConfig::OPTIONS lists its dotted keys for Config::unknownOptions().

Key	Default
`sitemap.enabled`	`true`	`false`: the adapter registers no command and no reader
`sitemap.url`	`null`	sitemap read when the command gets no argument; `null` = `<base_url>/sitemap.xml`
`sitemap.max_depth`	`3`	levels of `<sitemapindex>` followed below the root (`0` = the root only)
`sitemap.max_sitemaps`	`1000`	documents fetched per run, root included
`sitemap.max_bytes`	`52428800`	size cap of one uncompressed document (50 MiB, the protocol maximum; at least 1024)
`sitemap.allow_foreign_hosts`	`false`	follow nested sitemaps on other origins (CDN-hosted parts); `--allow-foreign-hosts` enables it for one run
`sitemap.spool`	`auto`	where a document is kept while parsing: `auto` = temp file, memory when the temp dir is not writable; `disk` = temp file or fail; `memory`
`sitemap.spool_dir`	`null`	directory of the temp files (`sys_get_temp_dir()`); point it at a writable volume on a read-only filesystem
`sitemap.fetch_retries`	`2`	extra attempts (1 s, 2 s, 4 s apart) after a network failure or 5xx while fetching a document; 4xx and broken documents are never retried

How it stays safe and small

Memory stays flat whatever the sitemap size: every document is spooled (Sitemap\Spool: a temp file, or memory on a read-only filesystem; straight from the socket when the transport implements Http\StreamingTransportInterface, as Psr18Transport does), gzip is inflated chunk by chunk into a second spool, and XMLReader walks the spool through the indexnowkit-spool:// wrapper with a few KiB of buffers. Nested sitemaps must live on the origin of the root unless allow_foreign_hosts says otherwise; recursion depth, document count and document size (before and after gunzip) are capped; external entities and network access are disabled in the XML parser. A failing nested sitemap is logged and skipped; a failing root throws Http\Exception\TransportException, and a response shorter than its Content-Length is a truncated download, never a document. Details in SECURITY.md.

The command

Sitemap\Console\SitemapRunner is the body of sitemap [url] (--changed-since "1 day", --allow-foreign-hosts, --force, --dry-run, --json); it streams, submits every batch.max_urls URLs, and submits the pending batch before reporting a mid-run failure (the re-run is idempotent, what was read is still worth announcing). The check command of every adapter carries Sitemap\Check\SitemapSpoolCheck: where documents are spooled, and whether that directory is writable — the kind of thing that otherwise only shows up on the first scheduled run.

An application decorates the source (filter, rewrite) or replaces it (another format, a database) by implementing Sitemap\SitemapSourceInterface and binding it under the adapter's alias. Writing an adapter? docs/adapters.md.

Requirements

PHP 8.2+, ext-xmlreader, indexnowkit/core ^0.4; ext-zlib for gzip-compressed sitemaps; symfony/console for the command body.

Versioning

SemVer; until 1.0 minor versions may contain breaking changes, listed in CHANGELOG.md. What the compatibility promise covers: docs/bc.md.

MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org.

indexnowkit / sitemap

Maintainers

Package info

Statistics

Security