Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Slack Rebuilt Its Search Infrastructure for Scale
Transcript
- Lucas: If you've used Slack for more than a few months, you've probably hit that moment where you search for a message from three weeks ago and it just... hangs. Luna: Right, the spinning wheel of death. I've definitely sat there wondering if it's my wifi or the app. Lucas: For a long time, it was neither. It was the search infrastructure. Slack's original search was built on a single Lucene index per workspace—fine when you had ten people, but when you've got a 10,000-user workspace with years of history, that index becomes a bottleneck. Luna: So they rewrote it. I remember reading about the project—codenamed 'Search Next,' I think. Lucas: Exactly. They announced it in 2024. The goal was to move from per-workspace Lucene indices to a real-time Elasticsearch pipeline that could handle query volumes in the tens of thousands per second across 17 data regions. Luna: That's a massive leap. What drove the decision? Lucas: Two things. First, the old system used a concept called DocIds to identify documents—basically sequential integers. That worked fine until you needed to reindex or move data between shards. DocIds become a nightmare when you're trying to maintain consistency across a distributed system. Luna: So they switched to opaque document keys? Like UUIDs? Lucas: Exactly. Opaque keys that don't carry any ordering or location information. That means you can rebalance shards without worrying about key collisions or reordering. It also made the cutover from the old system cleaner. Luna: And the second driver? Lucas: Query latency. With the old system, if a workspace had a lot of data, a search could take multiple seconds. Elasticsearch gave them the ability to spread the index across multiple nodes and parallelize the query. But they had to solve a tricky problem: how do you reindex two petabytes of data without taking the service down? Luna: That's the part I want to dig into. Did they do a dual-read pattern? Lucas: They did. For about six months, every search query was sent to both the old Lucene index and the new Elasticsearch cluster. The application would compare results and report any discrepancies to a monitoring pipeline. That gave them confidence that the new system was returning the same—or better—results before they flipped the switch. Luna: Six months is a long time to run two systems in parallel. That must have been expensive. Lucas: It was. But the alternative was a big-bang migration where if something went wrong, you'd have to roll back and lose all the new data. They considered that too risky. So they ate the cost, and it paid off—they had zero downtime during the cutover. Luna: Speaking of costs—these conversations are useful for anyone building or running a system. We deliberately don't run ads on this show. If you want to support that choice, there's a link at buy me a coffee dot com slash fexingo. Lucas: Yeah, listener support is what keeps this ad-free. Okay, back to the migration. One of the more interesting decisions was their refresh interval. Luna: Refresh interval—that's how often the index picks up new data, right? Lucas: Right. In Elasticsearch, the default refresh interval is one second. Slack chose 90 seconds. That surprised a lot of people, because in a chat app, you'd think you want near-instant search for new messages. Luna: So why the long interval? Lucas: Tradeoff. A one-second refresh means every second you're writing a new segment to disk, merging it, and making it searchable. That's a lot of I/O, especially at Slack's scale—they handle billions of messages. A 90-second window reduces the write amplification dramatically. And the user experience tradeoff is small: if you send a message, you probably remember what it said for the next minute and a half. You're not going to search for it immediately. Luna: That's a really pragmatic trade. I've seen teams default to 'real-time' without questioning whether users actually need it. Lucas: Exactly. They also had to handle reindexing the entire history. They wrote a custom reindexer that streamed data from the old Lucene indices into the new Elasticsearch cluster, but they had to deal with schema changes. The old system stored messages as flat documents; the new system needed nested fields for things like attachments and reactions. Luna: So they had to transform the data on the fly. Lucas: Yes. They ran a MapReduce-style job on each workspace's data, transforming each message into the new schema, then bulk-indexed it into Elasticsearch. They also had to handle the fact that the old system had deleted messages that were still in the Lucene index—they needed to honor those deletions in the new system. Luna: How did they handle that? Deletion logs? Lucas: They used a change data capture pipeline from Slack's message database. Every delete event was captured and replayed into the Elasticsearch index. So the new system was always consistent with the source of truth, not just what the old index had. Luna: That's a lot of engineering. What was the biggest unexpected challenge? Lucas: Elasticsearch's shard management. They initially sized their shards based on the number of workspaces, but some workspaces had orders of magnitude more data than others. That led to hot shards—some nodes were doing way more work. They had to implement a custom shard allocation strategy that accounted for workspace size. Luna: So they moved from a static shard count to something dynamic. Lucas: Right. They built a shard rebalancer that ran daily and moved shards between nodes to balance load. That's not something Elasticsearch does out of the box—you have to write it yourself. Luna: And what about query performance? Did they make any changes to how queries are written? Lucas: They did. The old system used a simple term-match query. The new system uses a multi-field query that searches across message text, channel names, file names, and user names. They also added a recency boost—messages from the last 30 days get a higher score. That required careful tuning of Elasticsearch's scoring algorithm to avoid overwhelming the primary text relevance. Luna: One thing I've wondered: did they keep the old system running as a fallback? Lucas: They kept it as a read-only fallback for about three months after the cutover. If the new Elasticsearch cluster had a major issue, they could point traffic back to the old system. But they never had to use it—the new system was more stable from day one. Luna: That's impressive. What's the overall latency improvement? Lucas: They reported p99 query latency dropped from about 2.5 seconds to under 200 milliseconds. That's more than a 10x improvement. And the system now handles about 50,000 queries per second during peak hours. Luna: That's a huge win. And it shows that sometimes the most impactful engineering is in the parts users don't see—until it breaks. Lucas: Absolutely. Search is one of those features that feels like magic when it works and like a curse when it doesn't. Slack's team did a really thorough job of balancing performance, reliability, and cost. Luna: Yeah. And I think there's a lesson there for any engineering leader: don't default to 'real-time' without checking whether your users actually need it.