SEO fundamentals

Crawling and indexing: How Google discovers and stores pages

Q: How do I stop a page being indexed?

Serve a noindex rule — <meta name="robots" content="noindex"> in the page's <head>, or an X-Robots-Tag: noindex HTTP header for files that are not HTML, such as PDFs. It only works if Google can still crawl the page and read it, so do not block the same URL in robots.txt, and do not try to write noindex into robots.txt, which Google does not support. If the page must be gone today rather than at the next crawl, use the removals tool in Search Console as well.

Crawling and indexing are the two stages a page has to pass before it can rank. Crawling is a search engine fetching the page; indexing is the engine analysing what it found and deciding whether to store it in its index. Google is explicit that neither step is guaranteed.

Last reviewed August 29, 2026 · 7 min read

On this page

Almost every "why don't we rank" conversation is really about one of these two. A page that was never fetched, or was fetched and not stored, is not competing badly — it is not competing at all, and no amount of copy or link work changes that.

The three stages of Google Search

Google describes Search as three stages, and says plainly that not all pages make it through each.

Crawling

The discovery stage. There is no central registry of web pages, so Google keeps looking for new and updated ones — a process it calls URL discovery. Pages become known because Google visited them before, because a link on a known page points at them, or because you submitted a sitemap.

Once a URL is known, Googlebot may fetch it. Which sites it crawls, how often, and how many pages it takes from each is decided algorithmically, and the crawlers throttle themselves to avoid overloading a server — a run of HTTP 500 responses is read as "slow down". Google states directly that Googlebot does not crawl every page it discovers.

During the crawl Google renders the page and runs any JavaScript it finds, using a recent version of Chrome, because sites often rely on JavaScript to put content on the page.

The named causes of crawl failure are unglamorous: problems with the server, network issues, and robots.txt rules preventing access — which is why the robots.txt file is the first thing worth checking.

Indexing

After the fetch, Google works out what the page is about: processing the text, key tags and attributes such as the <title> element and alt attributes, and the images and video on it. Part of this stage is deciding whether the page is a duplicate or the canonical version — Google clusters pages with similar content and selects the most representative one, with the consequences covered under canonical tags.

Then the important sentence: indexing is not guaranteed, and not every page Google processes will be indexed. The reasons Google names are low content quality, robots meta rules disallowing indexing, and site design that makes indexing difficult.

Serving

Only now does ranking happen. Google searches the index for matching pages and returns what it judges most relevant, with relevance determined by hundreds of factors that can include location, language and device.

This stage explains a symptom that confuses a lot of people: Search Console can report a page as indexed while you never see it in results. Google lists three reasons — the content is irrelevant to the queries being asked, its quality is low, or robots meta rules prevent serving.

Why a page does not get indexed

Ranked roughly by how often we find each one:

Cause	What it looks like
Blocked from crawling	Disallowed in robots.txt; the page is never fetched
`noindex` left in place	Usually inherited from a staging environment or a template
Not discoverable	No internal links pointing at it, missing from the sitemap
Consolidated into another URL	Google picked a different page as canonical for the cluster
Thin or duplicated content	Nothing on the page the index does not already hold
Broken rendering	Content that only exists after client-side JavaScript fails
Server errors	5xx responses at the moment of the crawl

Note how many are configuration rather than content. That is the useful bias of this topic: indexing problems are cheap to fix once correctly identified, and expensive only while they are misdiagnosed as a content problem.

`noindex`: the pages you keep out on purpose

Not every page belongs in the index. Thank-you pages, internal search results, filtered listing variants, old price-list PDFs — leaving them in wastes crawl attention and puts weak pages in front of searchers. Google supports two ways to say so:

A meta tag in the <head>: <meta name="robots" content="noindex"> for every engine that supports the rule, or <meta name="googlebot" content="noindex"> for Google alone.
An HTTP response header, X-Robots-Tag: noindex — the only option for non-HTML files such as PDFs.

Two conditions decide whether it works. Google has to be able to crawl the page to see the rule at all, so blocking the URL in robots.txt hides the noindex from the crawler that was supposed to obey it. And noindex in the robots.txt file itself is not supported by Google. Combining a robots.txt block with a noindex tag is a common own goal: neither instrument does what was intended.

Being indexed is also the entry ticket to everything downstream, citation by answer engines included — which is where generative engine optimization starts.

How long does indexing take?

There is no promised number, and Google avoids giving one. What its documentation does say:

Crawling can take anywhere from a few days to a few weeks.
Requesting a crawl does not guarantee inclusion instantly, or at all; Google's systems prioritise fast inclusion of high-quality, useful content.
There is a quota on submitting individual URLs, and asking repeatedly for the same one does not get it crawled faster.
A page you have just added noindex to leaves the index at the next visit — which, depending on the page's importance, can take months.

In practice, on an established site with healthy internal linking, a new page tends to be indexed within days; on a young domain with few links, weeks is normal. The lever that moves this is not the submit button — it is being linked to from pages Google already crawls often.

How to check, rather than guess

URL Inspection in Search Console is ground truth for one URL: whether it is indexed, which canonical Google selected, what the crawler received, and a button to request a recrawl. It requires being an owner or full user of the property.

The Page Indexing report answers the same at site scale, grouped by reason. It is also where you monitor pages Googlebot extracted a noindex rule from — the way to catch an accidental one before it spreads.

A sitemap is the tool for many URLs at once, particularly after a launch or a site move.

The method worth retiring is the site: operator: a rough estimate, not an index count.

How Xerx does this

Crawl and index control is the first half of our technical SEO work, and on most audits it is where the largest single finding sits. We crawl the site the way a search engine does, pull Search Console and log data, and compare three pictures: what the site publishes, what the crawler received, and what Google actually stored. Where those three disagree is the diagnosis.

Findings are then ranked by what they cost. A blocked section, a template-wide noindex and a redirect chain are not the same size of problem as a missing alt attribute, and an audit that weights them equally is not an audit. We fix the ones that matter in the templates and server config, or hand your developers tickets precise enough to ship without a meeting — then verify by re-crawling rather than declaring it done at deploy.

The diagnosis is free; ongoing work runs inside SEO Care from €790 per month, in real logged hours, cancel monthly. What nobody can offer is a guarantee: Google states that following every requirement still does not guarantee it will crawl, index or serve a page, and we will not promise around that.

FAQ

What is the difference between crawling and indexing?

Crawling is the fetch: a crawler requests the URL and downloads what the server returns, rendering any JavaScript along the way. Indexing is what happens next — the engine analyses the content, decides whether the page is a duplicate or the canonical version of a cluster, and stores it if it judges it worth storing. A page can be crawled and never indexed, which is the most commonly misread state of the two.

How long does Google take to index a new page?

Google says crawling can take from a few days to a few weeks, and does not promise inclusion at all. On an established site with good internal linking, days is typical; on a young domain with few inbound links, weeks is normal. Requesting indexing through URL Inspection is worth doing once for an important page, but it is quota-limited and repeating it does not speed anything up.

How do I stop a page being indexed?

Serve a noindex rule — <meta name="robots" content="noindex"> in the page's <head>, or an X-Robots-Tag: noindex HTTP header for files that are not HTML, such as PDFs. It only works if Google can still crawl the page and read it, so do not block the same URL in robots.txt, and do not try to write noindex into robots.txt, which Google does not support. If the page must be gone today rather than at the next crawl, use the removals tool in Search Console as well.