What is robots.txt? Structure, rules and common mistakes

That second sentence is the whole reason this page exists. Most of the damage the file causes comes from people using it as a privacy control, and most of the confusion around it comes from the gap between "do not fetch this" and "do not list this".

What the file is actually for

Google's introduction is narrow about the purpose: robots.txt tells crawlers which URLs they can access, mainly to avoid overloading your site with requests. Its three legitimate uses are managing crawl traffic on a server that would struggle under it, keeping crawlers away from unimportant or near-identical pages, and preventing media files — images, video, audio — from turning up in results.

It is explicitly not a way to hide a page. Google's own warning: if other pages point at your page with descriptive text, Google could still index the URL without ever visiting it, and the result then appears without a description. To keep a page out of results, use password protection, noindex, or remove the page.

Three limitations sit alongside that, and they are worth reading as design constraints rather than caveats:

• The rules are not enforceable. Googlebot and other respectable crawlers obey them; anything else may not. Google says outright that secrets belong behind a password on the server, not behind a disallow line.
• Not every search engine supports every rule, and crawlers interpret syntax differently.
• A disallowed URL can still be indexed if it is linked from elsewhere on the web, with its address and the anchor text used to describe it.

Structure and syntax

A valid line is a field, a colon and a value. Field names are case-insensitive; path values are not. Comments start with #.

# Controls crawling of URLs under https://example.com.
User-agent: *
Disallow: /includes/

User-agent: Googlebot
Allow: /includes/

Sitemap: https://example.com/sitemap.xml

Google supports exactly four fields, and ignores everything else — crawl-delay, for instance, does nothing:

Field	What it does
user-agent	Names the crawler the following rules apply to
disallow	A URL path that may not be crawled
allow	A URL path that may be crawled
sitemap	The full absolute URL of a sitemap

Two wildcards are supported in paths: * matches zero or more of any character, and $ marks the end of the URL. So /fish matches /fish.html, /fishheads and /fish/salmon.html, but not /catfish or /Fish.asp — matching is case-sensitive, and a rule matches anything starting with the path unless you anchor it.

The sitemap line has to be a fully qualified URL including protocol and host, and it does not have to be on the same host as the robots.txt file. You can list as many as you like.

Which rule wins

Two precedence rules decide every argument about a robots.txt file, and both are counter-intuitive the first time.

Only one group applies per crawler. Google's crawlers find the group whose user-agent most specifically matches, and ignore every other group. A Googlebot group and a * group are not combined — if you write specific rules for Googlebot, it stops reading the general ones entirely. This is the single most common way a robots.txt file does something nobody intended.

Within a group, the most specific path wins, and ties go to the least restrictive rule. Specificity is measured by the length of the rule path. Given allow: /p and disallow: /, the URL /page is allowed because /p is longer. Given allow: /folder and disallow: /folder, the allow wins, because on a genuine conflict Google takes the least restrictive rule.

Location, caching and failure modes

The file must sit in the top-level directory of the host, on HTTP, HTTPS or FTP, and its rules apply only to that exact host, protocol and port. So https://example.com/robots.txt does not govern https://www.example.com/, http://example.com/ or https://example.com:8181/; a robots.txt in a subdirectory is not a robots.txt at all, because crawlers never look there; and every subdomain needs its own file, which is how staging environments end up crawlable.

How Google handles a failed request matters more than most teams realise. A 4xx response other than 429 is treated as though no robots.txt exists, meaning no crawl restrictions at all. A 5xx is different: for the first 12 hours Google stops crawling the site while it retries, then falls back on the last good version for up to 30 days; if the errors persist beyond that and the site is otherwise available, it behaves as though the file were absent. The file is generally cached for up to 24 hours, so an edit is never instant.

Two more limits: the file must be UTF-8 plain text, and Google enforces a size limit of 500 KiB, ignoring everything past it.

robots.txt or noindex?

The two instruments answer different questions, and using the wrong one is the most expensive mistake on this page.

Goal	Instrument
Stop a crawler wasting requests on a huge, worthless section	disallow in robots.txt
Keep a page out of search results	noindex on the page
Keep a non-HTML file (a PDF) out of results	X-Robots-Tag: noindex header
Keep something genuinely private	Authentication

They also interact badly. A noindex rule only works if the crawler can fetch the page and read it — Google has to crawl the page to see the meta tag or header. Block the URL in robots.txt and the noindex is never seen, so the page stays eligible to appear. And noindex in the robots.txt file itself is not supported by Google. If a page must disappear, allow the crawl and let it read the instruction; the mechanics are covered under crawling and indexing.

The mistakes we actually find

Disallow: / left over from staging. The single line that removes a site from search. It survives launches because nobody diffs the file.

Blocking CSS and JavaScript. Google renders pages; if the resources needed to render are blocked, it analyses a page it cannot see properly. Google's guidance is to block resource files only when the page is not significantly affected by their loss.

Using robots.txt to fix duplicate content. Blocking a duplicate URL leaves it indexable-by-link and prevents the crawler from ever reading the canonical tag that would have consolidated it.

Specific rules that silently disable the general ones. A User-agent: Googlebot group added for one exception, quietly discarding every rule in the * group.

A file nobody has read since it was generated. Plugins and platforms write these. They get checked when traffic disappears.

How Xerx does this

robots.txt is the first file we open in an audit, and about once a quarter it is the whole finding. It costs nothing to check, and the failure mode is total — a site that is otherwise healthy simply is not there. That work belongs to technical SEO.

On sites we build, the file is generated from the routing layer rather than hand-edited or emitted by a plugin, so it cannot drift out of step with the routes that exist, and the sitemap lines cannot point at a path renamed two deploys ago. Staging is blocked by authentication rather than by a disallow line, because a disallow line is a suggestion and a login is not.

The free SEO report covers the crawl-control layer: what is blocked, what that block is costing, and whether the instrument matches the intention.

FAQ