How a crawler decides what to fetch next

13 August 2026 · Crawling

A crawler is a queue with a fetcher attached to it. Everything that feels mysterious about being crawled becomes ordinary once you picture the queue: a long list of addresses waiting their turn, reordered constantly, and trimmed whenever it grows faster than the machines can drain it.

Your page being known and your page being fetched are two separate events, and the gap between them can be months. A crawler learns an address from a link, from a sitemap, from a redirect target, from a previous version of the same page. Learning the address costs almost nothing, so the list of known addresses for any large site is far longer than the list of addresses anyone has actually requested.

What moves an address up the queue

No public description of a production crawler is complete, but the ordering signals that show up consistently in server logs are unsurprising ones. An address linked from many places on your own site is requested sooner than one linked from a single deep page. An address that changed the last three times it was fetched is revisited sooner than one that has been byte-identical for two years. An address that returned a server error is retried, then retried more slowly, then largely left alone.

The practical version of this is unromantic. If you want a page fetched, link to it from somewhere that is already fetched often, and make sure the response is fast and stable when the request arrives.

Politeness and the rate ceiling

Crawlers deliberately hold back. They spread requests out so that a small server is not knocked over by the act of being catalogued, and they watch response times while they work. A site that answers in eighty milliseconds gets a different request rate from one that answers in four seconds, because slow responses are read as a signal to back off.

This is the honest mechanism behind most advice about crawl efficiency. Nothing is rationed to punish you. There is simply a rate the crawler considers safe, and every request spent on a duplicate, a session-parameter variant or a paginated dead end is a request not spent on something you care about.

Where crawl waste comes from

Three sources produce most of it. Faceted lists that multiply one collection into thousands of filter combinations. Tracking parameters appended to internal links, so that the same document is requested under a dozen addresses. And calendar or pagination systems with no end, which generate an address for every month until the end of time.

You find these by reading your own access logs rather than by guessing. Group the requests by user agent, then by path pattern, and look at what the crawler actually spent its requests on last month. The answer is frequently a section you had forgotten existed.

Being fetched is not the finish line

A successful fetch means the machine has your bytes. It does not mean the page is stored, and it certainly does not mean the page is eligible to be shown to anyone. That is a separate decision made later, on different criteria, and it is worth understanding on its own terms.

Permalink: /notes/how-crawlers-decide-what-to-fetch/