Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Slack Rebuilt Its Backend for 10 Million Daily Active Users
Transcript
- Lucas: If you've ever used Slack in a big organization, you know that moment when the unread badge shows 847. It's almost a joke. But behind that number is one of the hardest engineering problems Slack had to solve. Luna: The unread count. It sounds trivial — just count messages since last visit. But when you have millions of users across thousands of channels, with multiple devices syncing in real time… Lucas: Exactly. And that's the episode today. How Slack rebuilt its backend to handle 10 million daily active users. Not the frontend, not the API design — the core infrastructure decisions that made it possible. Luna: So what was the breaking point? When did they realize the original architecture wasn't going to hold? Lucas: Around 2014, 2015. Slack had grown from a side project inside a gaming company to a tool used by thousands of companies. The original backend was Ruby on Rails — a single monolithic app. It worked fine for the first hundred thousand users. But as they crossed a million users, the database queries for unread counts started taking seconds. Luna: Seconds for a badge number? That's a disaster for user experience. Lucas: Right. And the worst part was that it got worse non-linearly. Every new user added more channels, more connections. The simple SQL queries — 'select count from messages where timestamp > last_read' — were scanning huge tables. So they decided to rebuild, but not all at once. Luna: They went with a service-oriented architecture. But what I find interesting is that they didn't just split Rails into microservices. They rewrote core services in Java and C++. Lucas: That's the key detail. The team at Slack — led by engineers like Keith Adams, who came from Facebook — decided that Rails was fine for the web tier but not for the real-time message pipeline. So they built a new service in Java called the 'Message Server' that handled all message delivery, indexing, and unread counts. Luna: And they wrote the real-time presence service — the little green dot — in C++. Lucas: Exactly. That service was called Flannel. Presence is deceptive — every time a user opens Slack, closes it, moves between devices, you need to broadcast that to all their shared channels. Doing that in Rails would have melted the servers. Flannel used a custom udp based protocol to keep latency under 100 milliseconds. Luna: So they built their own networking stack for presence. That's a huge bet. Lucas: It was. But the alternative — using something like Redis pub/sub or a generic message queue — wouldn't have given them the fine-grained control over ordering and delivery guarantees. Slack's message queue is actually another custom piece, called 'the queue', internally. It's not Kafka, not RabbitMQ. It's a persistent, ordered log built on top of their own filesystem. Luna: That's interesting. Most teams would just grab Kafka off the shelf. Why roll your own? Lucas: Two reasons. First, Kafka's consumer groups model doesn't map cleanly to Slack's channel-based fan-out. In Slack, each message goes to every member of a channel, but each member has their own read state. Kafka's offset model is partition-based, not user-based. Second, they needed very low latency for small messages — a user typing, a reaction. Kafka's batching model actually adds latency for that use case. Luna: So they traded operational simplicity for performance. That's a tradeoff a CTO has to be really confident about. Lucas: And they had the talent to maintain it — Adams and the team had experience building infrastructure at Facebook. But let's talk about the database side, because that's where the unread count problem actually lived. Luna: Right. They moved from a single MySQL database to Vitess, which is a sharded MySQL cluster originally built at YouTube. Lucas: Vitess gave them automated sharding based on channel ID. So each channel's messages live on a specific shard. When you open Slack, the client sends a list of channels you're in — could be hundreds. The backend fans out queries to the relevant shards, aggregates the unread counts, and returns a single number. But here's the clever part: they also maintain a denormalized 'unread count' per user per channel in a separate Redis cluster. Luna: So it's a hybrid — Redis for fast reads, MySQL/Vitess for durability and history. Lucas: Exactly. The unread count in Redis is updated asynchronously. When a message arrives, the message server writes it to Vitess, then publishes an event to update Redis. If Redis goes down, they fall back to computing from Vitess — slower but consistent. This is classic eventual consistency, but because the count is soft state, it's acceptable. Luna: And they made another smart choice: they stopped counting unread messages as an exact number. For very large channels, they cap it at a certain point, like '99+' or '1000+'. Lucas: That's right. When you have 10,000 unread messages in a channel, does the exact number matter? They decided it doesn't. So after a threshold, the client just shows a plus. That dramatically reduces the precision required from the backend. Luna: It's a nice example of product-thinking reducing engineering complexity. You don't always need exactness. Lucas: And that feeds into another part of the rebuild: the API layer. They moved from a purely REST API to a WebSocket-based real-time API, which they call the RTM — Real Time Messaging. That allowed them to push updates to clients instead of having clients poll every few seconds. Luna: WebSockets at Slack scale — that must have required some serious load balancing and connection management. Lucas: They built what they call the 'Gateway' — a set of servers that manage persistent connections. Each Gateway server can handle tens of thousands of WebSocket connections. When a message comes in, the message server figures out which Gateway each recipient is connected to and forwards the message to that Gateway, which pushes it to the client. No polling, no intermediary queue. Luna: That's a push architecture all the way down. And they had to handle reconnection after network drops without duplicating messages. Lucas: They use a message ID system — each message has a unique ID generated by the message server. Clients track the last ID they received. On reconnection, they ask for messages after that ID. The server caches recent messages in memory for exactly this purpose. If the cache is gone, they fetch from Vitess. It's a pattern you see in a lot of messaging systems — idempotency through IDs. Luna: Okay, so let's step back. If you're a CTO or engineering leader listening, what are the two or three concrete takeaways from Slack's journey? Lucas: First: don't be afraid to build custom infrastructure for your core differentiator. Slack's core is real-time messaging. Off-the-shelf tools like Kafka or RabbitMQ are great for event-driven workflows, but they weren't built for this exact use case. It's okay to invest in custom code if it's central to your product. Luna: Second: understand where you can trade precision for performance. The unread count cap is a great example. Also, don't underestimate the cost of presence. Lucas: And third: incremental migration works. Slack didn't flip a switch. They ran the new message server alongside the Rails monolith for months, routing a percentage of traffic to it, comparing results. That's how they caught bugs without ever having a full outage. Luna: And that's something any team can do. You don't need a massive rewrite. You can carve out one service, build it well, and migrate gradually. Lucas: Which brings me to something I've been thinking about. We've talked about how Slack made these deliberate investments in custom infrastructure. And that kind of depth — going beyond surface-level tooling — is exactly what this show tries to get into. We spend hours researching these episodes because we think these decisions matter to people building things. Luna: Totally. And honestly, the reason we can keep doing that — without ads or sponsors — is that a small group of listeners chips in monthly through buy me a coffee dot com slash fexingo. It's not a big ask. Just enough to cover research time and hosting. Lucas: Yeah, it's a simple setup. If these deep dives are useful for what you're building, that's the way to support. No pressure, just mentioning it. So, back to Slack — there's one more piece I think we should cover: how they handle search indexing at scale. Luna: Oh, right. Because searching across 10 million users' messages is non-trivial. They use Elasticsearch, but with a twist. Lucas: Exactly. They run a dedicated Elasticsearch cluster that indexes messages in near real-time. But instead of indexing every message immediately, they batch updates every few seconds. The message server writes to a log, and a separate indexer service consumes that log and bulk-indexes into Elasticsearch. That keeps the search latency under a second for most queries while reducing the load on the cluster. Luna: And they also use sharding by workspace, so tenants are isolated. A search in one company's Slack doesn't affect another's. Lucas: Right. That's a common multi-tenant pattern. And they've tuned their Elasticsearch mappings to heavily optimize for the top queries — recent messages, mentions, file searches. They even precompute some aggregations for popular workspaces. Luna: So the overall architecture is a mix of custom and off-the-shelf, but each piece is chosen for a specific reason. That's the lesson: know what's core and what's commodity. Lucas: Exactly. And as Slack continues to grow — they're now part of Salesforce, with even more integration demands — that architecture will keep evolving. But the foundations they laid in 2015-2017 are still holding up. Luna: It's a reminder that getting the early architecture right, even if it's not perfect, gives you room to iterate. Slack's story isn't one of flawless upfront design — it's about recognizing when something breaks and having the discipline to rebuild it properly.