How Search Engines Actually Work: Crawl, Index, Rank, Serve
A page passes through four stages before anyone sees it, and it can fail at any of them. Google crawls it, decides whether to index it, ranks it against the few hundred other documents that match a query, then serves a results page that increasingly carries an AI answer above everything else. The scale of the filtering is the part most business owners miss: Google has discovered more than 130 trillion URLs and holds roughly 400 billion documents in its index, and Ahrefs finds 96.55% of indexed pages get no organic traffic at all. This page walks the pipeline stage by stage, lists the technical limits that quietly decide outcomes (the 2 MB Googlebot fetches, mobile-first indexing, how server speed changes crawl rate), and shows where the AI answer layer sits on top. Updated when Google's documentation changes.
The numbers at a glance
- 130T+ Unique URLs Google has discovered on the web
- 400B Documents held in the Google search index
- 96.55% Of indexed pages get no organic traffic
- 2 MB Of an HTML file Googlebot actually fetches
- 20%+ Of searches now show an AI answer above results
Read the text version
The pipeline
Crawl: Googlebot discovers pages through links and submitted sitemaps and fetches them, reading the first 2 MB of an HTML file and the first 64 MB of a PDF. Anything past that is never processed, which is why critical content and tags belong high in the page.
Index: the fetched content is parsed and rendered, near-duplicates are clustered, and one canonical version is stored. Crawled does not mean indexed. The majority of crawled pages are classified as spam, duplicate or low quality and never enter the index.
Rank: when a query arrives, core algorithms retrieve matching documents from the inverted index and cut the set down to a few hundred, which are then scored. Indexed does not mean ranked.
Serve: results assemble for that specific query, now frequently with an AI Overview above the organic listings.
The funnel
Google has identified more than 130 trillion unique URLs. Its index holds roughly 400 billion documents at over 100 petabytes, a figure that came out in US antitrust testimony rather than from Google directly. Of the pages that do get indexed, Ahrefs finds 96.55% receive no organic search traffic. Each stage discards far more than it keeps.
Where pages fall out
Googlebot fetches only the first 2 MB of an HTML or text file. Google uses the mobile version of a page for both indexing and ranking under mobile-first indexing, so content hidden or dropped on mobile is effectively invisible. Server speed changes crawl rate directly: improving server response time by 100 milliseconds can increase crawling by around 15%, which matters most for large sites where crawl budget constrains how much gets seen.
The fifth stage
Synthesis now sits on top of the four. AI answers appear on more than 20% of searches, 68% of US searches end without a click to any site, and only about 12% of URLs cited by AI engines also rank in Google's top 10 for the same prompt. Being cited and being ranked are related outcomes, not the same one.
What this means for a growing business
Each stage is a filter, and a page that fails an early one cannot be rescued by better content later. Check in order: is the page crawled, is it indexed, is it ranking, is it cited. Most wasted SEO budget goes to improving pages that Google never stored in the first place, which no amount of content work will fix.
Frequently asked questions
What are the stages of how a search engine works?
Four: crawling, indexing, ranking and serving. Google discovers and fetches pages, decides which to store, retrieves and scores the matching documents when a query arrives, then assembles the results page. A fifth synthesis layer now generates AI answers on top.
How many pages does Google have in its index?
Roughly 400 billion documents, at over 100 petabytes, according to US antitrust trial testimony. Google has discovered more than 130 trillion URLs, so the vast majority of what it finds never enters the index.
Does being indexed mean my page will rank?
No. Indexing only makes a page eligible. Ahrefs finds 96.55% of indexed pages get no organic traffic from Google at all, because ranking requires competing against the few hundred documents core algorithms retrieve for a query.
How much of my page does Googlebot read?
The first 2 MB of an HTML or text file and the first 64 MB of a PDF. Anything past that is not processed, so important content and tags should sit high in the source.
Does site speed affect crawling?
Yes. Improving server response time by 100 milliseconds can increase crawling by about 15%. On large sites this decides how much of the site Google sees at all.
Sources
Keep reading
Article
How search engines work: crawling, indexing, and ranking in the age of AI
Home › Learn › How Search Engines Work Technical SEO How Search Engines Work: Crawling, Indexing & Ranking Fundamentals Google handles 8.5 billion searches…
Service
Managed SEO
We check the pipeline in order, crawled then indexed then ranking then cited, so budget goes to the stage that is actually failing.
See how it works
More infographics
View all
See where your site stands
Book a free 30-minute call with our team. We will review your search visibility and show you which fixes to make first.