Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Google Rebuilt Its Search Index for AI
Transcript
- Lucas: When Google launched AI search overviews in May 2024, most people focused on the weird answers — the glue on pizza, the 'eat one rock a day' thing. But underneath that launch, Google quietly replaced the indexing pipeline that had powered web search for almost twenty years. Luna: Replaced, not upgraded? That's a big claim for a system handling billions of queries a day. Lucas: It's not hyperbole. The old index was built around inverted document sorting — basically, for every word in every language, store a list of documents containing that word, then rank them by PageRank and a few hundred other signals. The new index is fundamentally different. It's built around semantic embeddings — vector representations of entire documents and queries that capture meaning, not just keyword matches. Luna: So Google moved from a giant lookup table to a neural search engine for its primary index. That's not a minor refactor. Lucas: It's arguably the largest infrastructure migration in the company's history since the original MapReduce rollout. And here's the specific architecture: they call it the Incremental Document Index, or IDI. Instead of batch-processing the entire web every few weeks, IDI ingests documents continuously, computes embeddings on the fly, and updates the vector index in near real time. That's how AI overviews can cite a piece published twelve minutes ago. Luna: Right — freshness was always the Achilles' heel of the old batch pipeline. But embeddings at web scale — how do you even store that? We're talking about hundreds of billions of vectors. Lucas: Google uses a custom approximate nearest neighbor index — think ScaNN, but massively distributed across its own TPU pods. Each document gets an embedding vector, say 768 dimensions, and those vectors are partitioned across shards using a learned hash. When a query comes in, it's also embedded, and the system retrieves the top-K vectors from the most relevant shards. That retrieval is then combined with traditional keyword signals in a two-stage ranking pipeline. First stage: fast approximate retrieval from the embedding index. Second stage: a deeper neural ranking model — something like a distilled version of BERT — that re-ranks the top few hundred results. Luna: So it's not purely semantic; they kept a foot in the keyword world. That makes sense for precision. Lucas: Exactly. Google's own research showed that pure semantic search can miss exact matches that users expect — if you type 'Python download', you want the python.org page, not a semantically related article about programming languages. So the final ranking layer blends both. The embedding index handles recall — finding documents that might be relevant even if they don't share exact keywords. The neural re-ranker handles precision — ordering those results so the most useful one comes first. And the traditional keyword index acts as a safety net for navigational queries. Luna: That dual approach is fascinating. But what about the training data for the embedding model? You can't just throw the entire web at a transformer and hope it works for search. Lucas: Google trained its embedding model on a custom dataset of query-document pairs — actual search logs from anonymized user interactions. Click-through data, dwell time, whether the user reformulated the query after clicking. They also used synthetic data: pairs generated by a teacher model — likely a larger LLM — that labeled documents as relevant or not to a given query. The model itself is a variant of the T5 architecture, distilled down to something that can embed a document in under 10 milliseconds on a TPU. That's the kind of optimization that only makes sense when you have Google's hardware. Luna: And the latency requirements for the end user? AI overviews need to generate an answer while simultaneously showing traditional blue links. That's a tight window. Lucas: Google's public latency targets for search are something like 300 milliseconds for the first result. With AI overviews, they have to embedded the query, retrieve from the vector index, run the neural re-ranker, then feed the top documents into a generative model — all before the user sees anything. They managed it by pre-computing embeddings for the entire web index, so the only real-time work is the query embedding and retrieval. The generative model itself is also heavily optimized — it's not a general-purpose LLM, it's a specialized model trained specifically to summarize and cite search results, not to have a conversation. That model is probably smaller than Gemini, maybe 7 billion parameters, and it runs on custom inference accelerators. Luna: So they optimized every piece of the pipeline for speed. What broke during the rollout? Because those weird answers suggested something was off. Lucas: The weird answers mostly came from the generative model misinterpreting satirical or low-quality sources. The embedding index itself was retrieving the right documents — that part worked. But the generative model didn't have strong enough guardrails to reject a false premise. Google fixed that by adding a factuality classifier that scores each generated sentence against the retrieved documents before displaying it. That classifier runs in a few milliseconds and is trained on a dataset of human-labeled factual errors. It's essentially a small natural language inference model. Luna: That's a pretty surgical fix. Rather than redesign the whole pipeline, add a verification step. Lucas: Right. And the lesson for other engineering teams is: when you're building a system that combines retrieval and generation, the retrieval part is usually reliable if you've trained it well. The generation part is where the black swans live. You need a separate quality gate specifically for factual consistency. That's a pattern we're going to see replicated across every search product that adds generative capabilities. Luna: It's interesting — and this is a good moment to mention something about the economics behind how we can cover this level of engineering detail every week. Lucas: Yeah, look — if these deep dives into how companies rebuild their infrastructure are useful for what you're building or running, there's a small way to help keep this going. A handful of listeners chip in monthly at buy me a coffee dot com slash fexingo, and that literally funds making this many episodes. No ads, no sponsors, just people who find the content valuable. Luna: And it's not about big donations — even a couple of coffees a month adds up when enough people contribute. It keeps the show independent and lets us pick topics because they're interesting, not because they're advertiser-friendly. Lucas: Exactly. And speaking of interesting — let's go back to the architectural challenge that made this whole migration possible. The real unsung hero is the incremental indexing layer. Luna: Yes — I want to understand how IDI handles deletions. If a page goes down, does the embedding get removed in real time? Lucas: It does. IDI maintains a versioned log of all changes — inserts, updates, deletions — and the vector index supports point deletions via a tombstone mechanism. When a document is deleted, its embedding is marked as deleted in the index, and the shard periodically compacts to reclaim space. But the clever part is that deletions also trigger re-embedding of related documents in some cases — if a page you cited changes, the generative model's context might be stale. So there's a dependency tracker that maps which documents are commonly cited together, and when one changes, it flags related embeddings for re-computation. Luna: That's a level of sophistication most search systems don't have. It's almost like a graph database layered on top of the vector index. Lucas: Exactly. And that graph is what allows AI overviews to maintain coherence. When you see a summary that cites three sources, those sources have been checked for mutual consistency — not just relevance to the query. That's the next frontier in search: not just retrieving the right documents, but ensuring the documents you retrieve agree with each other. Google's engineering team built that capability into the index itself, not as a post-processing step. Luna: So the index isn't just a storage layer anymore — it's part of the reasoning pipeline. Lucas: That's the fundamental shift. For twenty years, the index was a passive repository. You dump documents in, you query them. Now the index actively maintains relationships, flags inconsistencies, and even suggests which documents to re-fetch based on downstream usage. It's a living system. And that's the kind of architectural thinking that separates a search engine from a real AI platform. Luna: Final thought — for engineers listening who want to apply some of this without Google-scale resources, what's the one pattern you'd steal? Lucas: The dual-rank pattern. Use embeddings for recall, use a smaller model — or even simple heuristics — for precision. You don't need a billion-parameter re-ranker. You can use a logistic regression on top of embedding cosine similarity and keyword match features and get 90 percent of the benefit. The key is decoupling the two stages so you can optimize latency and accuracy independently. That's the architecture that won Google search, and it's accessible to any team building a retrieval system today.