What indexing actually decides

13 August 2026 · Indexing

Indexing is often described as storage, which makes it sound automatic. It is closer to a filing decision: the page has been read, and something now has to decide whether keeping a copy is worth the shelf space, and whether the copy should be allowed to appear in results.

Four states, not two

People talk as though a page is either indexed or not. In practice a page moves through at least four states, and confusing them causes most of the diagnostic mistakes I see.

• Unknown. No address has been discovered. Nothing else can happen.
• Known, not fetched. The address is in the queue. This can persist for a long time and is not an error.
• Fetched, not kept. The bytes arrived, were parsed, and the page was judged not worth storing, or worth storing only as a near-duplicate of something else.
• Kept and eligible. The page is stored and can be returned for a query, though whether it ever is returned is a further question entirely.

Reporting tools tend to collapse these into a single pass or fail, which is why a page can be described as excluded without anything being broken.

What gets a fetched page dropped

The common causes are dull. The page is one of several near-identical documents and another was chosen to represent the group. The page is a thin wrapper around content that lives elsewhere. The page returned a soft error: a status of 200 with a body that says nothing was found. The page requires a script to produce any text at all, and the parse produced an empty document.

That last one deserves a note. A page whose text is assembled in the browser can be indexed, but the work is done in a second pass that runs later and less reliably than the first. Text present in the delivered document is simply cheaper to handle, and cheap things happen sooner and more often.

Storage is not a ranking promise

A stored page competes for a position on a results page only when a query makes it relevant. Sites with large archives often have most of their pages kept, and most of those pages receiving no impressions at all, because nobody asks the questions those pages answer. That is a demand problem, not an indexing problem, and no amount of technical work will fix it.

How to tell the two apart

Ask whether the page has ever been shown for anything. Zero impressions with the page reported as stored means the content has no audience or no distinct angle. Impressions with no clicks means the listing is being seen and passed over, which is a writing problem. Nothing stored at all means you are still upstream, in crawling and duplication territory.

Permalink: /notes/what-indexing-actually-decides/