Robots.txt and noindex are not the same instruction
13 August 2026 · Crawling
One of these tells a crawler not to make a request. The other tells it not to list a page it has already read. Using the first to achieve the second is the most durable self-inflicted problem in this subject, and it survives every redesign because nobody ever revisits the file.
The file at the root
A robots file sits at the root of a host and lists paths that co-operating crawlers agree not to request. It is a request-level instruction, published in public, and it is advisory: it works because the major crawlers choose to honour it, not because anything enforces it. Anyone can read yours, which is a reason not to use it as a place to hide sensitive paths.
The instruction inside the page
A noindex directive lives in the page itself, either as a meta element in the head or as a response header. It says: you may read this, but do not list it. Because it lives inside the document, it only works if the document is fetched.
The trap
Block a path in the robots file and add noindex to the pages under it, and you have built a contradiction. The crawler never requests the page, so it never sees the noindex, so the address can still be listed on the strength of links pointing at it, usually with no description because nothing was ever read. The page you most wanted hidden becomes the one with the emptiest listing.
If you want a page out of the results, let it be fetched and let it say noindex. If you want to save crawl requests on a section you do not care about, block it in the robots file and accept that stray listings may persist. Choose one intention per path.
Removal, expiry and patience
A noindex takes effect at the next fetch, which is governed by everything in the crawling queue. A page nobody links to may not be revisited for a long time. If a listing needs to disappear quickly, the removal tools in a search console are the short-term instrument; the noindex is what makes the removal stick after the temporary suppression expires.
A short checklist
- Fetch your own robots file over the public address and read it. Rules for a staging host have a way of arriving on the live one.
- Never block a path whose pages carry a noindex you rely on.
- Never block script or style files that the page needs in order to be understood.
- Keep the sitemap reference in the file, and keep the host in it lowercase and consistent with the one people actually visit.
Permalink: /notes/robots-txt-and-noindex/