Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Slack Rebuilt Its Real-Time Messaging for 10 Million Concurrent Connections
Transcript
- Lucas: If these conversations are useful for what you're building or running, this next bit is for you. A small number of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what pays for making all of these. More on that later. Luna: Today we're digging into a piece of infrastructure almost everyone uses but almost nobody thinks about until it breaks: real-time messaging at Slack scale. Lucas: Right. Slack sits somewhere between a chat app and a platform. At peak, they're handling something like 10 million concurrent WebSocket connections. That's 10 million open TCP sockets, each pushing and pulling messages in real time. Luna: And that's not counting the internal event traffic — bot messages, API calls, webhooks. Lucas: Exactly. So the core problem is: how do you deliver a message from one user to another, or to a channel with thousands of members, with latency under a few hundred milliseconds, while preserving ordering, and without melting your network? Luna: I remember Slack's original architecture was a lot simpler. A single event bus, essentially. Lucas: Yes — early Slack used a single Redis-backed pub-sub bus for all real-time events. Every server subscribed to every channel. It worked fine for hundreds of thousands of connections, but as they grew, the bus became a bottleneck. Redis couldn't keep up with the volume, and the fan-out pattern meant every message was broadcast to every server, regardless of whether that server had a connected client interested in it. Luna: So they had to shard. But sharding a real-time chat system is trickier than sharding a database, because of ordering. Lucas: That's the key constraint. In Slack, message ordering within a channel matters. If Alice sends a message, then Bob replies, you want Bob's reply to appear after Alice's. If you shard by channel, you preserve ordering within that shard, but you need a way to route messages to the right shard without adding a central coordinator. Luna: What did they settle on? Lucas: They built a custom WebSocket gateway layer that acts as a connection manager and a message router. Each gateway node holds WebSocket connections for a set of clients. When a client sends a message, the gateway looks up the channel's shard assignment — which is deterministic based on channel ID — and forwards the message to the appropriate event bus shard. Luna: So the event bus itself is sharded per channel. Lucas: Yes. They moved from a single Redis bus to multiple Kafka topics, each topic representing a set of channels. The gateway subscribes only to the topics relevant to its connected clients. That cut the message fan-out dramatically. Luna: But that introduces a new problem: what if a channel is very large, like the #general channel at a big company? Lucas: That's the hot-channel problem. A single shard can get overloaded if a channel has tens of thousands of members all active simultaneously. Slack's approach was to split large channels into sub-shards by hashing the user ID, but then you lose global ordering. They decided that for very large channels, they'd relax ordering guarantees — messages within a sub-shard are ordered, but messages across sub-shards may arrive out of order. Luna: That's a practical tradeoff. Most users don't notice if two messages in a high-volume channel arrive milliseconds apart. Lucas: Exactly. And they made that tradeoff explicit. Now, the gateway layer itself became a scaling challenge. Each gateway node has to maintain a persistent WebSocket connection with each client. That's a lot of state. Luna: How do they handle reconnection when a gateway node fails? Lucas: They use a consistent hash ring to assign clients to gateways. When a client connects, the load balancer hashes the client ID and routes to one of N gateways. If a gateway dies, the clients reconnect and get rehashed to a new gateway. The new gateway pulls the client's state — like which channels they're in and their last seen message ID — from a shared cache layer, which is basically a Redis cluster. Luna: So the gateway is stateless in terms of persistent storage, but stateful in terms of open connections. Lucas: Right. And that brings us to the load balancer. Slack found that off-the-shelf load balancers like HAProxy or NGINX couldn't handle the number of concurrent WebSocket connections they needed — tens of thousands per node. The bottleneck was the kernel's connection table and the epoll overhead. Luna: So they wrote their own? Lucas: They built a custom Layer 4 load balancer in Go, called 'Siphon'. It uses raw sockets and a custom event loop to minimize context switches. They claim it handles over a million concurrent connections on a single machine with relatively low CPU usage. Luna: That's impressive. But I imagine the bigger challenge isn't just connections — it's the message delivery itself. How do they ensure a message from one user reaches all the others in the same channel without overwhelming the network? Lucas: They use what they call 'intelligent fan-out'. The event bus shard delivers the message to all gateway nodes that have at least one client subscribed to that channel. Each gateway then fans out to its local clients. But they also apply backpressure at the client level. Luna: Backpressure meaning if a client is slow to acknowledge messages, you stop sending? Lucas: Exactly. Each WebSocket connection has a send buffer. If the buffer fills up because the client is on a slow network or the browser tab is backgrounded, the gateway stops pushing new messages until the client catches up. This prevents a single slow client from blocking the entire gateway. Luna: But what about the ordering guarantee? If you stop sending to one client, do you risk messages arriving out of order when it reconnects? Lucas: They handle that with something called 'cursor-based reconnection'. Each client maintains a cursor — the ID of the last message they received. When they reconnect, they send that cursor to the new gateway, which replays any missed messages from the cache. That way, even if the client missed some messages due to backpressure, they'll get them in the correct order. Luna: That's elegant. So the system is designed for disconnection as a normal state. Lucas: Yes. And that's a key design principle for any real-time system at scale: assume clients will disconnect and reconnect frequently. Build for that gracefully. Luna: Let's talk about emoji reactions. I've heard that was a surprising bottleneck. Lucas: Oh, it's a classic. Emoji reactions are deceptively expensive. When someone adds a reaction to a message, that event needs to be broadcast to every client viewing that channel. In a channel with thousands of members, a single reaction generates thousands of events. And people love to react — Slack sees tens of millions of reactions per day. Luna: So how did they optimize that? Lucas: They batch reactions. Instead of sending each reaction as a separate event, they accumulate reactions over a short window — like 200 milliseconds — and send a single 'reactions updated' event with a list of all new reactions. That reduces the event count dramatically. They also deduplicate: if the same user removes and re-adds a reaction within the window, it's a no-op. Luna: But doesn't batching add latency? Lucas: A little — 200 milliseconds is below human perception for reactions. Users don't notice. It's a great example of trading a tiny amount of latency for a huge reduction in throughput. Luna: Another thing I've read about is their 'presence' system — the green dot that shows if someone is online. That's also real-time. Lucas: Presence is actually handled separately from messaging. They have a dedicated presence service that uses a different data model. Instead of broadcasting every presence change to every client, they use a pull-based model: clients poll the presence service for a list of users they care about — typically the ones visible in the current channel or sidebar. That polling happens every 30 seconds or so. Luna: So the green dot isn't truly real-time? Lucas: It's eventually consistent. If you go offline, it might take up to 30 seconds for your dot to turn gray. That's acceptable for most use cases. They also have a fast-path for active conversations: if two users are in a direct message, the system uses a push-based update for presence, so it feels immediate. Luna: Smart — separate tiers of consistency based on context. Lucas: And that's a pattern we see across many Slack subsystems: they don't try to make everything strongly consistent all the time. They identify where strong consistency matters — like message ordering in a channel — and where eventual consistency is fine — like presence or reaction counts. Luna: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these. We keep the whole thing ad-free because of that. Lucas: Yeah, it's a small group that makes it possible for us to spend the time digging into architecture like this. Appreciate every single one. Luna: And if it's not your thing, no worries. Let's keep going — I want to ask about the multi-region aspect. Lucas: Sure. Slack runs in multiple AWS regions. For real-time messaging, they faced a choice: do you keep all traffic in one region for simplicity, or do you route users to the nearest region for lower latency? Luna: They chose the latter, I assume. Lucas: Yes. Users are assigned to a home region based on latency. All their real-time traffic goes to that region. But messages can be sent between users in different regions. So a message from a user in us-east-1 to a user in eu-west-1 needs to be replicated across regions. Luna: That sounds like a cross-region consistency nightmare. Lucas: It can be. Slack uses a cross-region Kafka mirroring setup. Each region has its own Kafka cluster. When a message is produced in one region, it's mirrored to all other regions asynchronously. The gateway in each region subscribes to both local and mirrored topics. So a user in Europe sees the message with a slight delay — typically under a second — but it's eventually consistent. Luna: And ordering across regions? Lucas: They don't guarantee global ordering across regions. If Alice in the US sends a message, then Bob in Europe sends a reply, Alice might see Bob's reply before her own message — but only if the cross-region replication is delayed. In practice, that's extremely rare. The system is designed so that within a region, ordering is strict; across regions, it's best-effort. Luna: That seems like a sensible tradeoff for a global chat app. Lucas: It is. And it's worth noting that Slack's architecture has evolved a lot. They started with a monolithic Rails app and a single Redis bus. Now they have a distributed system with custom gateways, sharded Kafka, and multi-region replication. But they didn't build all of that upfront. They iterated. Luna: What's the one lesson you'd take away from this for someone building their own real-time system? Lucas: I'd say: design for disconnection. Assume clients will come and go. Use cursors or offsets to allow state reconstruction on reconnect. And separate your concerns — don't try to make every feature strongly consistent. Pick your battles. Luna: And maybe don't underestimate emoji reactions. Lucas: Never underestimate the emoji. That tiny thumbs-up can bring down your system if you're not careful.