IndexNow: How to Tell Search Engines About Changes on Your Site

A search engine learns about a new page on a site in two ways: it finds it by following links while crawling, or it reads sitemap.xml. Both work on the search engine's schedule: it decides by itself when to visit the site next. For a small blog that means days, sometimes weeks. The new article is already published, but it isn't in search results.

IndexNow turns this around: instead of the search engine asking the site, the site tells the search engine. The protocol was created by Microsoft (Bing) and Yandex in 2021; today it is supported by Bing, Yandex, Naver, Seznam and a few other search engines. Google doesn't take part in the protocol and still learns about pages from the sitemap.

How it works

The whole protocol is a single HTTP request. When a page on the site appears, changes or is deleted, the site sends its address to the search engine's endpoint. You can send one address with a plain GET:

GET https://api.indexnow.org/indexnow?url=https://www.vitaliy.org/post/indexnow-protocol-explained&key=<key>

Or up to 10,000 addresses at once with a single POST of JSON:

POST https://api.indexnow.org/indexnow
Content-Type: application/json

{
  "host": "www.vitaliy.org",
  "key": "<key>",
  "keyLocation": "https://www.vitaliy.org/<key>.txt",
  "urlList": [
    "https://www.vitaliy.org/post/indexnow-protocol-explained",
    "https://www.vitaliy.org/post/csp-against-xss"
  ]
}

The shared endpoint api.indexnow.org forwards the list to all participants of the protocol, so there's no need to notify Yandex and Bing separately. A 200 or 202 response means "accepted"; 400, 403, 422 mean an error in the request or the key, and retrying is pointless; 429 means too often, you need to wait.

It's important to understand what has been accepted and what hasn't. The search engine has received a signal "take a look at this address", nothing more. It decides by itself when to come for the page and whether to add it to the index. IndexNow speeds up discovery but guarantees nothing about ranking.

Why a key is needed

If anyone could submit addresses on behalf of any site, search engines would be flooded with junk. So a request is verified with a key, and the verification is ridiculously simple: the site owner makes up a random string of 8 to 128 characters (letters, digits, hyphen) and puts a text file on the site containing that same string:

https://www.vitaliy.org/<key>.txt

On receiving a request, the search engine opens the file at the keyLocation address (or https://<host>/<key>.txt if keyLocation isn't given) and compares its contents with the key from the request. If they match, the addresses come from the owner. There's no need to register anywhere, and nobody issues the key: you made it up yourself and put it on the site yourself. The key isn't a secret — the whole internet can read it; only its stability matters: change the key, and the search engine will reject requests until it reads the new file.

How it's done in this blog

In my blog the notification takes the same path as emails: the handler of a command that publishes, edits, renames, unpublishes or deletes a post puts the list of addresses into an outbox table in the same transaction as the change itself. A background worker picks the record up from the outbox and passes it on through a RabbitMQ queue, and a separate service sends the POST to api.indexnow.org.

There are two reasons for this. First, the external service doesn't take part in the publishing transaction: if api.indexnow.org is unavailable, the article is saved anyway and the notification goes out later. Second, retries and errors live in one place: 429 and 5xx mean a delivery retry, 4xx means a log entry without a retry.

What gets sent: the /post/<slug> address of a published post. When the slug changes, both addresses, the new one and the old one, so that the search engine learns both about the appearance and the disappearance. Drafts and posts awaiting moderation are never mentioned. On the dev environment the feature is turned off: it shows the same data under a different domain, and its addresses have no business being in search results.

After the feature was turned on, all posts already published were sent once in a batch: 134 addresses, response 202. From then on, search engines learn about every change the moment it happens.

The bottom line

• IndexNow is a single HTTP request with a list of addresses and a key; the shared endpoint api.indexnow.org forwards it to all participating search engines.
• The key is an arbitrary string verified by a file on the site; no registration is needed.
• The protocol speeds up page discovery but doesn't replace sitemap.xml: Google doesn't support it, and the participants still decide for themselves what to index.
• It's better to send the notification not from the request handler but through an outbox and a queue: then the search engine being unavailable doesn't get in the way of publishing.