README

IndexNow for Laravel

Tell search engines a URL changed, the moment it changes.

IndexNow is a small protocol: you publish a key file on your domain, then POST the URLs that changed. Bing, Yandex, Seznam and Naver consume it, and one submission to the shared endpoint reaches all of them. Google does not participate, so this complements your sitemap, it does not replace it.

This package wires the protocol into Eloquent, so publishing a post submits its URL and unpublishing it submits the URL again, which is the part most implementations miss.

IndexNow::track(Post::class)
    ->url(fn (Post $post) => route('blog.show', $post->slug))
    ->when(fn (Post $post) => $post->status === PostStatus::Published);

That is the whole integration. No trait, no interface, nothing added to the model.

Installation

composer require edulazaro/laraindexnow
php artisan vendor:publish --tag=indexnow-config

Getting the key in place

Publish the config and generate a key:

php artisan vendor:publish --tag=indexnow-config
php artisan indexnow:key

The key is written to your .env, the way key:generate writes APP_KEY:

INDEXNOW_KEY=ecd5c0c8f2ad4b0e9a0d1f2c3b4a5968

With no .env to write to it goes into config/indexnow.php instead, and if neither can be written without guessing the command says so and prints the line to add by hand.

Worth knowing before you decide where to keep it: unlike most keys this one is public by design, since you publish it as a text file, and it has to match a file that lives in the repository. Keeping both in version control means they cannot drift apart and a deploy needs nothing else. Keeping it in .env means a server that misses the variable serves a key file that no longer matches, and nothing says so until a submission is rejected. To pin it, replace env('INDEXNOW_KEY') in the config with the key itself.

The key has to be publicly readable at https://yourdomain.com/{key}.txt. This package serves that route for you, so there is nothing to drop into public/.

If you deploy behind php artisan down, write a real file as well:

php artisan indexnow:key --file

The route and the file cover the same URL, and on an ordinary day the route is enough. The file is for the days that are not ordinary: maintenance mode answers every route with a 503, and a search engine checking the key mid-deploy would be told the key is gone. A file in public/ is served without booting the framework, so it survives that, a fatal error, and a half finished composer install. Commit it, or generate it again on the server after deploying. The command warns you about a key file left over from a previous key, which stays claimable until you delete it.

Confirm it works before you rely on it:

php artisan indexnow:verify

An unreachable key file is the reason behind almost every 403 from the endpoint, and the endpoint itself tells you nothing useful about which half failed.

Tracking models

// app/Providers/IndexNowServiceProvider.php

public function boot(): void
{
    IndexNow::track(Post::class)
        ->url(fn (Post $post) => route('blog.show', $post->slug))
        ->when(fn (Post $post) => $post->status === PostStatus::Published && $post->published_at?->isPast())
        ->ignoring(['views', 'last_viewed_at'])
        ->affects(fn (Post $post) => [route('blog.index')]);
}

Method	Asks about	Purpose
`url($routeOrFn)`		The canonical URL of the record: a route name, a literal URL, or a closure. Return `null` and it is never submitted.
`when($attr)`	the record now	The attribute holds something true.
`when($attr, $value)`	the record now	The value an attribute has to hold. A closure judges the whole record.
`when($attr, $op, $value)`	the record now	Compared: `=`, `==`, `!=`, `<>`, `>`, `>=`, `<`, `<=`.
`whenNot($attr)`	the record now	The attribute holds something false.
`whenNot($attr, $value)`	the record now	Anything but that value.
`whenIn($attr, [...])`	the record now	Any of several values.
`whenHolds($attr, $value)`	the record now	The same as `when()`, spelled out to read next to `whenWas()`.
`whenWas($attr, $value)`	the change	The value the attribute held before this write.
`whenWasIn($attr, [...])`	the change	Any of several previous values.
`whenChanged(...$attrs)`	the change	Those attributes changed, whatever their values.
`whenBecame($attr, $value)`	the change	It just took that value, having held something else.
`whenBecameIn($attr, [...])`	the change	It just took any of several values.
`whenBecameFrom($attr, $to, $from)`	the change	The exact transition, both ends pinned down.
`whenLeft($attr, $value)`	the change	It stopped holding that value.
`whenLeftIn($attr, [...])`	the change	It stopped holding any of several values.
`whenIncrements($attr)`	the change	A numeric attribute went up, whatever the amount.
`sets($attr, $value)`		Written on the record once its URL has been handed over. Defaults to true.
`ignoring([...])`		Attributes whose change is not a content change.
`affects($urls)`		Other URLs this write left out of date: a route name, a URL, an array, or a closure.

Conditions combine with AND, values with OR. Every condition you chain has to hold. Use the In variants for several possible values of one attribute, and pass a map instead of an attribute name to declare several conditions at once:

    ->whenIn('status', ['published', 'featured'])              // one attribute, several values
    ->when(['status' => 'published', 'review' => 'correct'])   // several attributes
    ->whenBecame(['published' => true, 'status' => 'live'])

IndexNow::track(Post::class)
    ->url(fn (Post $post) => route('blog.show', $post->slug))
    ->whenBecame('published', true)      // it just flipped to true
    ->when('review', 'correct');         // and this holds right now

Backed enums are compared by their value, so whenBecame('status', 'published') matches an attribute cast to an enum without unwrapping it yourself.

Reading an attribute as a flag

Named on its own, an attribute has to hold something true, and whenNot() is the opposite. Truth is read the way PHP reads it, so 1, "1" and true all pass while 0, "0", null and "" do not. That covers a tinyint(1) with no cast, and it covers a counter: when('comment_count') is "has at least one".

    ->when('published')            // published is true
    ->whenNot('spam')              // spam is false
    ->when('comment_count')        // has at least one
    ->when('comment_count', '>=', 5)

These ask about the record as it stands, so indexnow:sync can answer them too and a registration built only from them is not a transition rule.

Submitting once, and remembering it

sets() writes an attribute on the record once its URL has been handed over. The value defaults to true, which is the flag column case; pass another one, or a closure, for anything else. Put the same attribute in the conditions and the record drops out the moment it is marked, which is how you say "submit this once, ever":

IndexNow::track(Game::class)
    ->url('games.show')
    ->whenNot('index_now_sent')
    ->when('published')
    ->sets('index_now_sent');

It also makes indexnow:sync resumable: each pass only picks up what is left, instead of resubmitting the whole table. A dry run never writes.

Two things to know. The write is quiet, because a normal save would fire another event and the record, having just left the conditions, would be submitted a second time. And marking means handed over, not accepted by a search engine: the buffer carries URLs and nothing else, so by the time a submission succeeds or fails there is no record left to point back at. Listen for SubmissionFailed if a failure has to undo it.

A flag like this is the wrong tool when a page should be resubmitted after an edit, since a written record never comes back. For that, declare the edit as its own registration with whenChanged().

Several ways in: registrations combine with OR

Conditions on one registration are an AND, and sometimes that is not the shape of the problem. A record can become worth indexing through more than one route, each with conditions of its own, and no single AND describes all of them at once.

Call track() again for the same model and you get a second registration. Registrations are an OR: a write is submitted when any of them wants it.

// A game page is worth indexing once it is approved and has playthroughs.
// Either half can be the one that arrives last, so declare both routes.

IndexNow::track(Game::class)
    ->url('games.show')
    ->whenLeft('playthroughs', 0)        // playthroughs arrived, already approved
    ->whenHolds('approved', true);

IndexNow::track(Game::class)
    ->url('games.show')
    ->whenBecame('approved', true)       // approval arrived, playthroughs already there
    ->when(fn (Game $game) => $game->playthroughs > 0);

Writing that as one registration does not work: whenLeft(...) and whenBecame(...) on the same chain would demand both transitions in the same write, which never happens.

Each registration keeps its own url(), ignoring(), affects() and policy, so they are read and reasoned about separately. The model is observed once however many registrations it has, and a URL that two of them resolve is submitted once.

registrationFor() returns the first; registrationsFor() returns all of them.

Naming the URL

A string is read as a route name and resolved with the record bound to it, which covers most models. Anything that already looks like a URL is left alone, and a closure handles the rest.

    ->url('blog.show')                                   // route('blog.show', $post)
    ->url('https://example.com/about')                   // taken literally
    ->url('/about')                                      // taken literally
    ->url(fn (Post $post) => route('blog.show', $post->slug))

affects() takes the same shapes, plus an array of them:

    ->affects('blog.index')
    ->affects(['blog.index', 'https://example.com/novedades'])
    ->affects(fn (Post $post) => [route('blog.category', $post->category)])

track() registers the observer for you and is safe to call twice: the second call returns the same registration instead of adding a second observer.

Policy classes

When the rules grow, or you want them testable on their own:

use EduLazaro\IndexNow\Policy;

class PostIndexNow extends Policy
{
    public function url(Model $post): ?string
    {
        return route('blog.show', $post->slug);
    }

    public function when(Model $post): bool
    {
        return $post->status === PostStatus::Published;
    }
}

IndexNow::track(Post::class, PostIndexNow::class);

Only url() is required.

What actually triggers a submission

With a transition condition, that condition decides, full stop. whenBecame, whenBecameFrom, whenLeft, whenIncrements, whenChanged and whenWas all need both states, so they only apply to writes.

whenIncrements cares about direction, not amount: a counter going from 3 to 4 matches like one going from 0 to 90. To catch only the first rise, which is the usual "this page now has enough content to be worth indexing" case, reach for whenLeft($attr, 0) instead.

Without one, the default rule applies, and it asks a single question: does this record belong in the index, before and after? That answer is whatever your when() conditions return, as a plain true or false.

`when()` before	`when()` after	Submitted
false	false	no
false	true	yes, it entered the index
true	true	only if a content attribute changed
true	false	yes, it left the index

That last row is the one most implementations get wrong. A record leaving the index matters as much as one entering it: the URL now 404s and the engines should be told to look. The same goes for a deleted record, and for a changed slug, where both the old and the new URL go out.

Four Eloquent events are observed: created, updated, deleted and restored. saved is deliberately not one of them, it fires alongside created and updated and would double every submission. Neither is forceDeleted, which arrives after deleted already fired for the same removal.

Submissions are held until the surrounding transaction commits, so a rolled back write never reaches a search engine.

The gap you need to know about

Eloquent events do not fire for query builder writes. This publishes a hundred posts and IndexNow hears about none of them:

Post::where('status', 'draft')->update(['status' => 'published']);

The same is true of delete() on a query, insert() and upsert(). No package can intercept those. That is what indexnow:sync is for:

php artisan indexnow:sync Post

It walks the table, evaluates the same when() condition the observer uses, and submits everything that qualifies. Use --dry-run first.

Submitting by hand

IndexNow::submit('https://example.com/page');        // buffered, sent by the queue
IndexNow::submit([$urlOne, $urlTwo]);

IndexNow::submitNow($url);                           // sent inline, returns the results
IndexNow::submitModel($post);                        // resolves the URL from the registration

And when you are importing, seeding or backfilling:

IndexNow::withoutSubmissions(function () {
    Post::factory()->count(50_000)->create();
});

Buffering, queueing and deduplication

Model events do not send HTTP requests. They push URLs into a cache backed buffer and schedule one delayed job to drain it, so saving a hundred rows produces one request rather than a hundred. Exactly one flush job exists per window, no matter how many events arrive.

The same URL is not submitted twice within the deduplication window. Editing a post five times in ten minutes is one submission. Keep that window short (the default is an hour). It is an anti-bounce measure, not a record of what is already indexed, and a long window swallows real edits: publish at 09:00 with a 24 hour window and the correction you make at 15:00 never goes out.

Retries live on the job, not on the transport. A 429 is released with the endpoint's own Retry-After; a 5xx or a connection failure backs off at 1, 5 and 15 minutes. A rejected key or a malformed URL is never retried, because retrying will not fix it.

Several domains from one application

Map a key per host:

'keys' => [
    'example.com' => 'ecd5c0c8f2ad4b0e9a0d1f2c3b4a5968',
    'example.org' => '5f3a9d7e1b2c4a6d8e0f1a2b3c4d5e6f',
],

The key file route answers with the right key for whichever host was requested, submissions are grouped per host before being sent, and URLs whose host is not configured are dropped rather than submitted against the wrong key. indexnow:verify checks every host at once.

Commands

Command
`indexnow:key`	Generate a key and show where it has to be published
`indexnow:verify`	Check the key file is publicly readable and correct
`indexnow:submit {url*}`	Submit URLs by hand, inline by default
`indexnow:sync {model}`	Submit every indexable record of a tracked model
`indexnow:flush`	Send whatever is waiting in the buffer
`indexnow:models`	List the tracked models

Commands send inline by default, because when you run one you want the status code on screen, not a queued job. Pass --queue for the opposite.

Testing

$fake = IndexNow::fake();

$post->update(['status' => 'published']);

$fake->assertSubmitted(route('blog.show', $post->slug));
$fake->assertSubmittedCount(2);
$fake->assertNothingSubmitted();

fake() swaps the transport and bypasses the queue, so assertions work without a worker.

Not submitting from your laptop

Submissions only happen in the environments listed in the config, production by default. This is the guard that stops a development machine from asking search engines to crawl URLs that only resolve locally, and it is the first thing to check when nothing is going out.

For a staging machine that should exercise the whole path without talking to the endpoint, set driver to null and log to log.

To stop submissions in production without deploying, set INDEXNOW_ENABLED=false and clear the config cache. That is the only setting this package reads from the environment: it is an operational kill switch, not configuration.

Configuration

Everything lives in config/indexnow.php and is documented there. The settings worth knowing about:

Key	Default
`enabled`	`true`	Master switch, the only setting reading `.env` (`INDEXNOW_ENABLED`)
`environments`	`['production']`	Where submissions may happen
`driver`	`'http'`	`http` or `null`
`key` / `keys`	`null` / `[]`	Single key, or one per host
`hosts`	`[]`	Allowed hosts, derived from the keys when empty
`queue.enabled`	`true`	Buffer and queue, or send inline
`buffer.delay`	`60`	Seconds before the buffer is drained
`dedupe.ttl`	`3600`	Deduplication window in seconds
`log`	`'null'`	`null`, `log` or `database`

Set log to database and publish the migration to keep a record of every request:

php artisan vendor:publish --tag=indexnow-migrations
php artisan migrate

One row per request rather than per URL, so a flush of ten thousand URLs does not write ten thousand rows.

Events

UrlsSubmitted and SubmissionFailed, both carrying the SubmissionResult with the host, the URLs, the status code and a message written to be read by a human.

Author

Created by Edu Lazaro

License

Larascraper is open-sourced software licensed under the MIT license.

edulazaro / laraindexnow

Maintainers

Package info

Statistics

Security