Notes on search engines and how pages get found
13 August 2026 · Crawling
A crawler is a queue with a fetcher attached to it. Everything that feels mysterious about being crawled becomes ordinary once you picture the queue: a long list of addresses waiting their turn, reordered constantly, and trimmed whenever it grows faster than the machines can drain it.
Permalink: /notes/how-crawlers-decide-what-to-fetch/
13 August 2026 · Indexing
Indexing is often described as storage, which makes it sound automatic. It is closer to a filing decision: the page has been read, and something now has to decide whether keeping it is worth the shelf space.
Permalink: /notes/what-indexing-actually-decides/
13 August 2026 · Crawling
One of these tells a crawler not to make the request. The other tells it not to list the page it has already read. Using the first to achieve the second is the most durable self-inflicted problem in this subject.
Permalink: /notes/robots-txt-and-noindex/
13 August 2026 · Indexing
Most duplicate content is not plagiarism. It is one document reachable through several addresses, each created by an ordinary technical decision nobody thought of as a decision.
Permalink: /notes/duplicate-urls-and-canonicals/
13 August 2026 · Structure
Nearly every structure works at forty pages. The interesting question is what happens at four hundred, when the menu can no longer list everything and the person who invented the categories has moved on.
Permalink: /notes/site-structure-that-holds-up/
13 August 2026 · Structure
A page with nothing pointing at it is a page nobody will fetch often and nobody will stumble upon. Internal links are the cheapest, least glamorous lever you have, and they are almost always underused.
Permalink: /notes/internal-links-as-plumbing/
13 August 2026 · Writing
A title element is read in a context you do not control: stacked against nine other lines, in a narrow column, by someone scanning rather than reading.
Permalink: /notes/writing-page-titles/