Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Discord Rebuilt Its Voice Engine for Zero Packet Loss
Transcript
- Lucas: If these conversations are useful for what you're building or running, stick around — today we're looking at how Discord rebuilt its voice engine from the ground up to handle 200 million monthly active users without dropping a single packet. Luna: And not just during quiet moments — I'm thinking about those massive Call of Duty or Fortnite launches where millions are talking simultaneously. Lucas: Exactly. So Discord started life as a pretty standard WebRTC app. Back in 2015, they used Google's libjingle library, which was designed for one-on-one calls, not the kind of group voice channels they were trying to build. Luna: Right, and WebRTC has this built-in assumption that you want to prioritize low latency over perfect reliability. Which makes sense for video calls — better to skip a frame than to stall. Lucas: But for gaming voice chat, you actually want the opposite in some ways. Gamers can tolerate a few extra milliseconds if it means the audio stays clear and doesn't cut out. So Discord had a fundamental mismatch between the protocol they inherited and the user expectations they had to meet. Luna: So they built their own transport layer over UDP instead of relying on WebRTC's built-in TCP fallbacks. Lucas: That's the core of it. They moved to a custom udp based system where they control every aspect of packet sequencing, jitter buffering, and forward error correction. The key insight was that WebRTC's congestion control algorithms are too aggressive for gaming — they'd drop audio quality the moment there's any network jitter. Luna: Walk me through forward error correction in this context. How does that work for voice? Lucas: So instead of waiting for the receiver to say 'hey I missed packet 47' and then retransmitting it — which adds a full round trip — they send redundant data alongside the original stream. Essentially, every packet contains enough information to reconstruct a small window of audio even if a few packets are lost. Luna: That's clever because with voice, you can afford a tiny bit of redundancy. The extra bandwidth is negligible compared to the latency saved. Lucas: And they tuned it carefully. Discord's engineering team published numbers showing they could recover from up to 20 percent packet loss with negligible impact on perceived audio quality. That's massive when you're routing traffic through congested residential ISPs. Luna: There's also the jitter buffer redesign. WebRTC's default jitter buffer is designed for adaptive bitrate video — it's constantly adjusting based on network conditions. But for voice, you want a fixed-size buffer that introduces a known latency. Lucas: Right, so Discord moved to a fixed 40-millisecond jitter buffer. That's the trade-off: you add 40 milliseconds of delay in exchange for smooth playback even when packets arrive out of order. And they made it configurable per server region, so if you're connecting from a data center you can drop it even lower. Luna: So the architecture now is: a custom UDP transport layer, fixed jitter buffer, and forward error correction. That's the trifecta. Lucas: Plus they integrated their own noise suppression engine called Krisp. That's actually a separate machine learning model that runs on the client side, but it's deeply integrated into the audio pipeline so that voice activity detection — knowing when someone is actually talking — happens before the packet even gets assembled. Luna: What about the server side? Because Discord doesn't do peer to peer for group channels, right? Everything goes through their servers. Lucas: That's right. They use a selective forwarding unit model — the server receives audio from each participant and forwards it to everyone else in the channel. They built their own SFU from scratch because the off-the-shelf WebRTC SFUs couldn't handle the scale or the low-latency requirements. Luna: So now we're talking about a custom transport, custom jitter buffer, custom SFU, custom noise suppression. At what point does it stop being a chat app and become a voice infrastructure company? Lucas: That's the fascinating thing. Discord essentially had to become a real-time communications company just to be a good gaming chat app. And they did it with a relatively small engineering team — under 200 engineers when they made most of these changes. Luna: Hang on, I want to ask about the migration. They had millions of active users on the old WebRTC stack. How did they switch without users noticing? Lucas: They did it region by region over about six months. Each cloud region would get a new voice server deployment running the new stack, and they'd gradually shift traffic over via DNS and load balancer changes. They also kept the old stack running for fallback — if any user's client had compatibility issues, it would automatically fall back to WebRTC. Luna: That's a classic dark launch pattern. But with voice, even a brief blip is audible. Did they have any major incidents? Lucas: They had one notable outage during a Call of Duty Warzone launch in 2021. The new SFU had a memory leak under a specific load pattern that only happened when hundreds of thousands of users joined channels nearly simultaneously. They patched it within a few hours, but it was a stark reminder that testing at scale is never quite the same as production. Luna: And you know, it's episodes like this — where we dig into real engineering decisions — that take a lot of research. Quick honest thing: a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making these episodes possible. Lucas: Yeah, we don't run ads, so it's listener support that keeps the show going. If you find these breakdowns useful, that's the place. And we'll keep digging into the architecture. Luna: So after that Warzone incident, what did Discord change in their testing process? Lucas: They built a chaos engineering framework specifically for audio pipelines. They'd simulate packet loss, latency spikes, and sudden user surges on a shadow fleet running the new code. The goal was to catch memory leaks and race conditions before they hit production. Luna: Sounds like they borrowed from Netflix's Simian Army but for voice. Lucas: Exactly. They called it 'Voice of Chaos' internally — they'd inject packet loss into a percentage of shadow servers and monitor audio quality metrics. It's a great example of how infrastructure testing has to be domain-specific. Luna: Let's talk about the client side. Because noise suppression is done locally — Krisp runs on device. How does that integrate with the network pipeline? Lucas: Krisp runs as a separate process that intercepts the raw audio from the microphone before it even reaches the voice engine. It's a neural network trained on millions of hours of background noise — keyboard clicks, fans, street noise — and it strips that out while preserving speech. The clean audio is then passed to the encoder and sent over UDP. Luna: So the network only ever carries clean audio. That reduces bandwidth because you're not wasting bits encoding background noise. Lucas: Right, and it improves quality because the codec can focus its bitrate on the speech signal instead of wasting it on noise. Discord uses the Opus codec, which is already very efficient — it can deliver broadcast-quality audio at as low as 32 kilobits per second per user. Luna: For a 10-person channel, that's 320 kbps total server bandwidth. That's tiny. Lucas: And with their custom SFU, they can do something called 'audio mixing' — instead of sending 10 separate streams to each user, they mix the audio of all participants into a single stream on the server. That means each user receives only one audio stream, regardless of how many people are talking. Luna: But mixing introduces latency. How do they keep it under 50 milliseconds end to end? Lucas: They do the mixing in the same process as the SFU, with a carefully tuned thread model. The audio packets arrive, get decoded, mixed in a lock-free ring buffer, re-encoded, and sent out — all within a 10-millisecond window. It's a tour de force of low-level optimization. Luna: And the client doesn't need to do any mixing itself. That's a huge CPU savings on the user's device. Lucas: Exactly. That's why Discord can run smoothly on a ten-year-old laptop while Zoom chugs. They offloaded as much work as possible to their servers. Luna: What about quality of service? In gaming, if you're in a firefight, you don't want your voice packets competing with game traffic. Lucas: They can't control the user's home network, but they do use DSCP — differentiated services code points — to mark voice packets as high priority. Most consumer routers respect those markings, so the voice traffic gets prioritized over, say, a Steam download. Luna: And if the router doesn't respect DSCP? Lucas: Then the forward error correction and jitter buffer are your safety net. Discord's architecture is designed to handle the worst-case consumer internet — high packet loss, high jitter, and no QoS. That's the reality for a lot of their users, especially outside the US. Luna: Let me ask about the future. With spatial audio becoming popular — Apple's AirPods Pro, for example — is Discord looking at that? Lucas: They've actually shipped a beta of spatial audio for voice channels. It uses HRTF — head-related transfer functions — to position different speakers in a 3D space around the listener. It's computationally expensive, so they're doing it on the server side for now, mixing the spatialized stream just like they mix regular audio. Luna: That's ambitious. So the server has to calculate a unique spatial mix for every user in a channel? That's potentially 10x the mixing work. Lucas: That's the challenge. They're solving it with a technique called 'binaural rendering at source' — each user's voice is spatialized once at the server and then combined. It's still early, but it's a good example of how Discord continues to push its custom stack further. Luna: It's impressive that a chat app born from a gaming community now has one of the most sophisticated real-time audio pipelines in the world. Lucas: And it all started because WebRTC wasn't good enough for gaming. That's the lesson: sometimes the best engineering decision is to throw out the standard library and build exactly what your users need, even if it means reinventing the wheel. Luna: Yeah, and they've proven that a focused team can out-engineer the incumbents when they deeply understand their use case. Lucas: That's the takeaway for me. Discord didn't try to build a general-purpose communications platform. They built a voice engine for gamers, and everything else flowed from that constraint.