Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Pinterest Rebuilt Its Image Search at 5 Billion Pins
Transcript
- Lucas: Five billion. That's how many pins Pinterest had in its image search index as of early this year — and the old architecture was starting to buckle. Luna: That's a serious number of images. What was breaking first? Lucas: Latency. The search team told me that median query time had drifted from about 80 milliseconds to over 200 milliseconds in just two years — and the tail was hitting two seconds. The problem wasn't the database. It was the similarity search algorithm itself. Luna: So they weren't just adding more servers. They had to rethink how search works under the hood. Lucas: Exactly. The old system used a brute-force k-nearest neighbor approach on a cluster of GPU nodes. For each query, it would compute cosine similarity between the query embedding and every single pin embedding in the index. That works when you have, say, 100 million vectors. At 5 billion, the math becomes impossible — you hit a wall on both compute and memory bandwidth. Lucas: And if today's deep dive into image search infrastructure is useful for what you're building or running, the reason we can keep doing deep dives like this without any ad breaks is listener support — head over to buy me a coffee dot com slash fexingo. It's a small thing that makes a big difference in keeping this show independent. Luna: Yeah, we hear from people who say they use these engineering stories to pitch ideas to their own teams. That's exactly why listener support matters. Lucas: So the Pinterest team stepped back and asked: what's the fastest way to find similar images among billions without checking every single one? The answer was approximate nearest neighbor search — specifically, they adopted a hierarchical navigable small world graph, or HNSW. Luna: HNSW — that's one of those algorithms that sounds complicated but the basic idea is pretty intuitive, right? Lucas: It is. Imagine you're in a city with 5 billion houses and you want to find the closest one to a given address. You don't knock on every door. You build a network of highways and local roads — long-range connections get you close fast, then short-range connections refine the result. HNSW does exactly that with vectors. It builds a multi-layer graph where each layer has fewer nodes and longer edges. Luna: So the query starts at the top layer, hops between distant clusters, then drops down to finer layers until it finds the nearest neighbors. Lucas: Exactly. Pinterest reported that switching from brute-force to HNSW cut median query latency from 200 milliseconds back down to 30 milliseconds, with 99th percentile under 100 milliseconds. And they did it with fewer machines. But the graph construction itself is expensive — building the initial index for 5 billion vectors took about three days on a dedicated cluster. Luna: Three days of compute just to build the search index. That's a non-trivial operational cost. Lucas: Right, but it's a one-time build. Then you need incremental updates — pins are added constantly. For that, Pinterest used a technique called 'dynamic index maintenance' where new vectors are inserted into the graph by connecting them to their nearest neighbors in the existing graph. That's O per insert, so it scales. Luna: But the bigger question — how do you get the embeddings in the first place? They moved from GPU inference to TPUs, right? Lucas: Yes. Originally every new pin went through a ResNet-50 model on NVIDIA V100 GPUs. That took about 50 milliseconds per image. As pin uploads grew to over 100 million per day, they needed a cheaper, faster inference path. Google's TPU v4 gave them about 5 milliseconds per image at a lower cost per inference. Luna: But TPUs are more rigid. They're great for large batch inference with a fixed model architecture, but less flexible for experimentation. Lucas: That was exactly the trade-off. Pinterest decided to freeze the embedding model for production inference and keep experimentation on GPU clusters. So they essentially have two pipelines — a stable TPU pipeline for real-time embedding generation, and a GPU research cluster for training new models. Once a model is validated, it gets compiled for TPU and pushed to prod. Luna: That sounds like a smart separation of concerns. But what about cold start — a brand new pin that hasn't been indexed yet? Lucas: Cold start is the classic problem in recommendation systems. For new pins, there's no interaction data, so you can't rely on collaborative filtering. Pinterest tackled it in two ways. First, they use the visual embedding directly — the HNSW index works on visual similarity alone, so a new pin can be searched immediately after its embedding is computed. Second, they built a 'fast-follow' pipeline that updates the index every 15 minutes instead of batch-rebuilding nightly. Luna: So a new pin is searchable within 15 minutes of upload, purely based on visual features. Lucas: Right. And over time, as the pin gets saves and clicks, the system blends in collaborative signals using a learned weighting. But the baseline is always visual. One interesting detail — they also added a 'diversity constraint' in the search results. Without it, HNSW tends to return a cluster of very similar pins. So if you search for 'minimalist desk setup,' you might get 50 variations of the same Ikea desk. Pinterest introduced a maximum marginal relevance algorithm that penalizes redundancy. Luna: That's a user-experience change that directly affects engagement. If results are too repetitive, people swipe away. Lucas: Exactly. They tuned the diversity parameter using A/B tests. The final system returns 30 results per query, with embedding similarity weighted 70 percent and diversity penalty 30 percent. They saw a 12 percent increase in saves per session after launching. Luna: Let's talk about the team and timeline. How long did the full rebuild take? Lucas: About 18 months, with a team of roughly 15 engineers — mostly from the search and ML infrastructure teams. They shipped in phases: first the HNSW index for a subset of pins, then the TPU inference pipeline, then incremental updates, then diversity ranking. Each phase was validated with canary traffic before full rollout. Luna: Did they run into any major surprises during the rollout? Lucas: One big one: the HNSW graph used about 100 gigabytes of RAM per billion vectors. For 5 billion vectors, that's 500 gigabytes — too large for a single machine. They had to shard the index across 16 nodes using a consistent hashing scheme based on pin ID. But then query routing became a challenge: a single query might need to search multiple shards and merge results. Luna: So they traded index size for query complexity. How did they handle the merge? Lucas: They built a lightweight scatter-gather layer. Each shard returns its top 100 candidates, and the aggregator re-ranks them using the same similarity metric. That adds maybe 5 milliseconds of overhead, which they considered acceptable. The alternative — keeping the index on a single monster machine — would have been too expensive and failure-prone. Luna: What about the cost comparison? Was the new architecture cheaper overall? Lucas: Pinterest shared some numbers at a conference. The old gpu based system cost about $8 million per year in compute — including both inference and search. The new tpu plus hnsw system runs at about $3.5 million per year. That's a 56 percent reduction, with better latency and higher throughput. Luna: That's a pretty compelling business case. What's the biggest lesson for other teams thinking about rebuilding their search? Lucas: Don't underestimate the data engineering work. The algorithm change was important, but most of the team's time went into building reliable pipelines for embedding generation, index maintenance, and monitoring. The HNSW library itself — they used a fork of Facebook's FAISS — was the easy part. The hard part was making sure new pins were indexed within 15 minutes, that the shard mapping stayed consistent during node failures, and that latency didn't spike during index rebuilds. Luna: So the infrastructure around the algorithm mattered more than the algorithm itself. Lucas: Absolutely. And that's the pattern we see in most successful rebuilds. The smart math gets the headlines, but the engineering is about boring, reliable pipelines. One last thing — Pinterest open-sourced their HNSW sharding and routing code. It's on GitHub under the Pinterest engineering org. Luna: Good to know. So if someone's running a similar system at a smaller scale, they can learn from what Pinterest actually shipped, not just the theory. Lucas: Exactly. And that's the real value of these post-mortems — seeing the decisions behind the architecture, not just the architecture itself.