Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Discord Scaled Its Real-Time Infrastructure
Transcript
- Lucas: If these conversations are useful for what you're building or running, then today's episode is going to hit exactly that note. We're talking about a company that went from a single server running a Go binary to handling 140 million monthly active users — all on real-time messaging with sub-100-millisecond latency. Luna: Discord. I've been waiting for this one. Every time I open a WebSocket connection, I think about how they manage so many persistent connections. Lucas: Right. And the story is really instructive because it's not about some exotic new technology. It's about a series of deliberate, sometimes boring choices that compound. The founding team started with a single Go service — they called it the 'chat service' — that handled every WebSocket connection from the browser or the desktop app. Luna: One service for everything? Authentication, message routing, presence, guild state? Lucas: All of it. And it worked for a surprisingly long time. When Discord launched in 2015, they had maybe a few thousand concurrent users. That single Go binary could handle a million concurrent WebSocket connections without breaking a sweat. The Go runtime's goroutine model was basically purpose-built for this — each connection is a goroutine, the scheduler handles the I/O multiplexing, and you get massive throughput without the callback hell of Node.js or the thread per connection overhead of Java. Luna: But that doesn't scale forever. Eventually you hit a ceiling. Lucas: Exactly. The ceiling wasn't CPU or memory, actually. It was the operational complexity of a single binary. Every deploy touched every feature. One buggy message handler could bring down the entire chat system. So around 2017, they started splitting out services. The first split was authentication — moving that to its own Go service. Then message search, then the real-time presence system, then the voice connection manager. Luna: What I find interesting is they didn't go full microservices from day one. They waited until the pain was real. Lucas: That's a key lesson. And it's not just a 'startup scrappiness' thing. Even at their scale now — 140 million monthly actives, millions of concurrent WebSocket connections — they still run services that are fairly coarse-grained. The chat service is still one deployable unit, but they've added layers. A message goes in through a gateway service, gets validated, hits a sequencer for ordering, then an event pipeline fans it out to all connected clients. Luna: The sequencer is the part I want to dig into. Because message ordering in a distributed chat system is deceptively hard. You have multiple servers, multiple data centers, and users expect messages to appear in the order they were sent. Lucas: Right. Discord uses a concept they call 'channel-local ordering'. Each text channel has a sequencer — basically a single process that assigns monotonically increasing IDs to messages within that channel. It's a single point of ordering, but since the sequencer is lightweight and runs in memory, it's not a bottleneck. They use Redis-backed counters behind the scenes, so if the sequencer crashes, the next instance picks up from the last persisted ID. Luna: And users don't notice a gap because the client buffers and reorders on the receiving end. Lucas: Exactly. The client sees a stream of events, and it applies them in ID order, not arrival order. That's a classic eventual-consistency trade-off. You get low latency because you're not waiting for a global consensus, and the user experience is fine because, within a single channel, the ordering is deterministic. Luna: I want to talk about storage too. Discord migrated from MongoDB to Cassandra in 2016, then later added ScyllaDB for certain workloads. What drove those moves? Lucas: The MongoDB move was driven by operational pain. They had a single MongoDB replica set that stored all chat history, and as they grew, the primary node couldn't keep up with writes. They tried sharding, but the MongoDB sharding architecture at the time had issues with hotspots and resharding. Cassandra gave them a hash-ring-based partitioning model that spread writes evenly across nodes, and they could add capacity without rebalancing — just spin up new nodes and the token ranges adjust. Luna: But Cassandra has its own trade-offs. Read performance can be inconsistent, especially for range queries like 'get messages from last hour'. Lucas: Exactly. That's why they added ScyllaDB later for the 'hot' data — recent messages that users are likely to scroll through. Scylla is C++ reimplementation of Cassandra that's more consistent on read latency because it's better at CPU scheduling and avoids the Java garbage collection pauses. They keep messages in Scylla for, I think, 30 days, then move older messages back to Cassandra for cold storage. Luna: So it's a tiered storage architecture. That's a pattern we see at a lot of large chat systems — WhatsApp, Slack, Telegram all do something similar. Lucas: Let's talk about the voice side, because that's where Discord really differentiates. Their voice chat runs on a custom system written in Erlang — something most people wouldn't expect from a modern startup. Luna: Erlang? The telecom language from the 80s? That's a surprising choice for 2026. Lucas: It's surprising until you think about the requirements. Voice chat is soft real-time — you need to encode, transmit, decode, and play audio with end to end latency under 50 milliseconds. And you have millions of concurrent voice channels, each with up to 25 participants, where every participant's audio has to be mixed and sent to everyone else. Erlang's actor model, fault tolerance, and hot-code swapping are perfect for that. Discord's voice servers run on bare metal, not containers, because they need deterministic network performance. Each server handles about 10,000 simultaneous voice connections, and they have thousands of these servers globally. Luna: And the Erlang runtime — the BEAM — has a scheduler that's specifically designed for soft real-time. No garbage collection pauses that could drop a packet. Lucas: Exactly. The BEAM scheduler is preemptive at the process level — each Erlang process gets a small time slice, and if it doesn't yield, the scheduler preempts it. That means a cpu bound voice encode task can't cause a latency spike on another connection. When they benchmarked it against Go and Rust prototypes, Erlang was the only runtime that could guarantee sub-50-millisecond latency at 95th percentile under load. Luna: That's a great example of picking the right tool for the job, not the trendy tool. If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. It keeps us independent and lets us do deep dives like this. Lucas: Yeah, we really appreciate that. So back to voice — the trade-off is that Erlang developers are harder to find. Discord had to build an internal training program for new hires. Luna: But they decided the operational reliability was worth the hiring friction. That's a CTO decision that goes beyond pure technology. Lucas: Absolutely. And they've been consistent about it. The voice system has had, by their public postmortems, 99.99 percent uptime over the last four years. That's single-digit minutes of downtime per year. Luna: Let's zoom out. What's the one architectural principle that runs through all of Discord's decisions? I think it's 'optimize for the tail.' Lucas: That's a great way to put it. They care deeply about the 99.9th percentile user. When you have millions of concurrent connections, the slowest one percent is still tens of thousands of people. So they instrument everything — every WebSocket, every database query, every voice packet. They have dashboards that show the P99 latency for each microservice, and if it drifts above 100 milliseconds, an alert fires. Luna: And they've been willing to rewrite critical paths when the tail gets too long. The move from MongoDB to Cassandra was essentially a tail-latency optimization. Lucas: Right. And they did a similar thing with their gateway service — the entry point for all WebSocket connections. They rewrote it in Rust in 2023 to reduce CPU overhead per connection, which let them handle 50 percent more connections on the same hardware. That's a direct cost saving and a latency improvement. Luna: Rust for the gateway, Go for business logic, Erlang for voice — that's a polyglot architecture. How do they manage developer productivity across three languages? Lucas: They use a shared gRPC layer for service to service communication, and all services expose metrics in a common Prometheus format. The deployment is Kubernetes for stateless services, with the voice servers running on bare metal outside the cluster. They have a single CI/CD pipeline that builds and deploys all services, regardless of language. The key is they don't let the polyglot nature become an excuse for silos — every engineer is expected to be able to work in at least two of the three languages. Luna: That's a cultural decision as much as a technical one. So for a CTO listening, what's the actionable takeaway? I'd say: resist the urge to over-architect early, but invest in observability from day one. Lucas: I would add: know your non-negotiables. For Discord, it's real-time latency and availability. Everything else — consistency, storage efficiency, even developer hiring — is subordinate to those. If you're building a product where a 200-millisecond delay is a failure, structure your whole system around that. Luna: And don't be afraid to use 'boring' technology if it solves the problem. Cassandra is 16 years old, Redis is 15, Erlang is 40. None of it is new, but it's battle-tested. Lucas: Exactly. Discord's stack is a masterclass in pragmatic engineering. They're not chasing hype — they're chasing reliability.