Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How LinkedIn Rebuilt Search for 950 Million Members
Transcript
- Lucas: LinkedIn's search team has a problem you probably don't think about every day: how do you serve relevant results to 950 million members across jobs, people, companies, posts, and groups — all while respecting who can see what? Luna: And that's not just a relevance problem. It's a permissions problem too, right? Because I can't see someone's profile if they're three degrees away and I'm not a premium member. Lucas: Exactly. And that constraint changes everything about how you build a search index. LinkedIn's old search stack was based on Apache Solr, and by 2020 it was showing its age. The team knew they needed a ground-up rebuild. Luna: So what was the breaking point? Was it latency, scale, or something else? Lucas: It was all three, honestly. But the biggest driver was that Solr couldn't handle the real-time indexing they needed. When a member updates their profile — adds a new job, changes their headline — that change needs to be searchable within seconds. Solr's batch-oriented architecture meant index updates could take minutes, sometimes hours. Luna: That's a huge problem for a platform where timeliness matters. If I just updated my profile to 'Senior Engineer at Google', I want recruiters finding me under that title today, not tomorrow. Lucas: Right. So the team designed a completely new indexing pipeline. They moved to a log-based architecture using Apache Kafka as the backbone. Every profile change publishes an event to Kafka, which feeds into a stream processor that updates the search index in near real-time. Luna: And what did they replace Solr with? Did they build their own search engine? Lucas: They did. It's called Galene — named after a Greek nymph, which is a fun bit of trivia. Galene is a custom retrieval engine built specifically for LinkedIn's data model. It's written in C++ and designed to handle the complex permission filtering that Solr struggled with. Luna: So the permission model is baked into the retrieval layer, not just applied as a post-filter. That must have been a massive architectural shift. Lucas: It was. In Solr, the typical approach was to index everything, then apply permission checks after retrieval. But at LinkedIn's scale — 2.5 billion search queries per week — that post-filter approach was too slow. Galene encodes permissions directly into the inverted index, so the retrieval step only returns documents the user is allowed to see. Luna: That's clever. But doesn't that make the index much larger? You're effectively storing multiple copies of the same document for different visibility levels. Lucas: It does increase index size, but the team found that the tradeoff was worth it. They reduced query latency by about 40 percent on average. For a search that previously took 200 milliseconds, it's now down to 120. That's a noticeable improvement when you're typing into the search bar. Luna: Yeah, every millisecond counts in search. Google found that a 100-millisecond delay can drop engagement by 0.2 percent. So if LinkedIn's query volume is that high, the business impact is real. Lucas: And it's not just speed. Relevance also improved. Galene uses a technique called 'real-time learning to rank' — they incorporate user engagement signals like clicks and saves to adjust ranking models on the fly. So if everyone starts clicking on a certain type of result, the model adapts within minutes. Luna: That's impressive. But I've heard that LinkedIn search has a lot of edge cases — like typos in job titles, or people searching in multiple languages. How does Galene handle that? Lucas: Great question. The team built a custom query understanding layer that runs before the retrieval engine. It handles spelling correction, synonym expansion, and language detection. For example, if you search 'sotware engineer' — misspelled — it will correct to 'software engineer' and also expand to related terms like 'developer' or 'programmer'. And for multilingual queries, it can detect the language and apply appropriate stemming. Luna: That sounds like a lot of machine learning models running in the query path. How do they keep latency low with all that processing? Lucas: They moved a lot of the heavy lifting to offline precomputation. The spelling correction model, for instance, is pre-trained and runs as a lightweight lookup. And the language detection is fast because they only run it on the query, not on all the documents. The key insight is that query understanding is computationally cheap compared to the retrieval and ranking steps. Luna: But there's another challenge: LinkedIn has multiple search surfaces. There's the main search bar, but also job search, people search, company search — each with different relevance criteria. Does Galene handle all of them? Lucas: It does, but through separate indices. Each vertical has its own Galene index because the data model is different. A job posting has fields like salary range and location that a people profile doesn't. But they share the same underlying infrastructure — the same Kafka pipeline, the same Galene cluster. That's what makes it scalable. Luna: So the infrastructure is unified, but the indices are separate. That's a common pattern in large-scale search systems. But what about the migration itself? How did they switch from Solr to Galene without disrupting the user experience? Lucas: That's one of the most interesting parts. They ran a shadow comparison for months. Every query was sent to both the old Solr cluster and the new Galene cluster, but only Solr's results were shown to users. They compared relevance metrics like click-through rate and zero-results rate. Only when Galene consistently outperformed Solr did they start ramping traffic. Luna: That's a textbook approach. But I imagine there were surprises even in shadow mode. Did anything break? Lucas: Yeah, one big surprise was that Galene initially returned too many results. Because the permission filtering was so efficient, it was returning documents that Solr had missed due to permission errors. But some of those results were actually irrelevant — they were just more visible. So the team had to tune the relevance model to compensate. Luna: So the permission-aware retrieval introduced new kinds of noise. That's a classic second-order effect. Lucas: Exactly. And speaking of second-order effects, the team also had to deal with the fact that Galene's real-time indexing changed how they handled deletions. When a member deletes their account, that change needs to propagate instantly. In Solr, deletions were batched, so there was a window where deleted profiles still appeared. Galene processes deletions in the same stream — so they disappear within seconds. Luna: That's a big win for data privacy compliance. With regulations like GDPR, you need to be able to delete data quickly. Lucas: Right. And that's a theme across the whole rebuild — every architectural decision had a compliance angle. The permissions encoding, the real-time deletions, the auditing logs. LinkedIn operates in over 200 countries, so they have to satisfy multiple regulatory regimes simultaneously. Luna: It's rare to see a search engine rebuild that's driven as much by compliance as by performance. Most companies focus on latency and relevance. Lucas: True, but LinkedIn's data is sensitive by nature. If you search for someone and see results you shouldn't, that's a privacy violation. So the constraints are tighter than, say, a web search engine. Luna: And now they have a platform that can serve 2.5 billion queries a week with better latency and better relevance. That's a pretty impressive outcome. Lucas: Yeah, and the interesting thing is that the rebuild unlocked new product features too. For example, they launched 'people also viewed' recommendations in the search results, which rely on the real-time index to suggest similar profiles. That wasn't possible with Solr's batch updates. Luna: So the infrastructure change directly enabled a new user-facing feature. That's the kind of impact that gets leadership buy-in for the next big engineering project. Lucas: And for other teams building similar systems, the lesson is clear: if you're stuck with a legacy search engine that can't handle real-time updates or complex permissions, a custom solution like Galene might be worth the investment — but only if you have the engineering depth to maintain it. Luna: And if these conversations are useful for what you're building or running, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: It really does make a difference. And speaking of building, let's talk about another company that rebuilt their search infrastructure from scratch — but with a very different set of constraints.