Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Netflix Rebuilt Its Encoding Pipeline for 4K HDR Streaming
Transcript
- Lucas: So here's a number that stopped me: Netflix says that every single video title they serve goes through a per-title, per-scene encoding pipeline that picks from something like a hundred different encoding recipes. Not one recipe, not ten — a hundred. And that pipeline runs daily for over ten thousand titles. Luna: That's wild. I remember when Netflix just used a fixed bitrate ladder — one size fits all. What pushed them to rebuild? Lucas: The simple answer is 4K HDR. When you're streaming in 4K with high dynamic range, the old approach wastes bandwidth on simple scenes and leaves quality on the table for complex ones. A static ladder means you're either over-encoding a talking head or under-encoding an action sequence. Luna: So they basically realized that a single encoding configuration is never optimal for every scene of every movie. Lucas: Exactly. And they had the data to prove it. In 2015, Netflix published a paper showing that per-title encoding could save 20 percent of bandwidth while maintaining the same perceived quality. That's not just a cost save — that's a huge deal for users on capped data plans. Luna: Twenty percent bandwidth savings at scale — that's millions of dollars in delivery costs, not to mention a better experience. How did they design the pipeline to handle that complexity? Lucas: They built a massively parallel system that analyzes each source video scene by scene. The pipeline has three stages: first, a quick analysis pass that computes complexity metrics like spatial and temporal activity. Then a decision engine that selects the optimal encoding parameters for each scene — resolution, bitrate, codec settings. Finally, the actual encode, which runs on GPU clusters. Luna: So the decision engine is the real innovation. It's not just 'use a higher bitrate for action scenes' — it's actually choosing different resolution and codec profiles per scene? Lucas: Right. For a slow, static scene, you might encode at 1080p or even lower resolution and still look great. For a fast-moving action scene with lots of detail, you jump to 4K with a higher bitrate. The engine uses a metric called VMAF — Video Multimethod Assessment Fusion — which Netflix developed to predict human-perceived quality. The goal is to hit a target VMAF score while minimizing bitrate. Luna: VMAF — that's the open-source metric Netflix contributed, right? I've seen it used in FFmpeg and other tools. Lucas: Exactly. Netflix open-sourced VMAF in 2016, and it's become a standard in the industry. It combines multiple quality metrics into a single score, trained on human opinion data. So the encoding pipeline isn't just guessing — it's optimizing against a metric that correlates with what viewers actually see. Luna: That's a smart move. But moving from cpu based encoding to GPU — that must have been a huge infrastructure shift. Weren't they heavily invested in CPU farms for encoding? Lucas: They were. Historically, encoding was done on CPU clusters using software encoders like x264. But as they moved to 4K and more complex codecs like HEVC and AV1, CPU encoding became too slow and expensive. Netflix started experimenting with gpu based encoding around 2018, using NVIDIA's NVENC and later their own custom hardware. Luna: So what's the performance difference? Like, how much faster is GPU encoding for a typical 4K title? Lucas: Netflix reported that GPU encoding can be up to ten times faster than CPU for the same quality target. That means a 90-minute 4K movie that might take 20 hours to encode on CPUs can be done in about two hours on GPUs. And that's not just speed — it's also cost, because you need fewer servers. Luna: Ten times faster is massive. But I imagine the challenge isn't just speed — it's also maintaining quality across different GPU generations and drivers. How do they ensure consistency? Lucas: They run extensive validation tests. Every time a new GPU generation comes out, they re-encode a reference set of clips and compare VMAF scores. They also have a canary deployment process where they test new GPU firmware on a small percentage of titles before rolling out globally. It's rigorous. Luna: Makes sense. And this is all for their own content, but Netflix also supports user-generated uploads for some features. Does the same pipeline apply? Lucas: Good question. For user-generated content — like the Netflix Party feature or short clips — they use a simplified version of the pipeline. It's still per-title, but with fewer encoding recipes and a faster analysis pass. The full pipeline with per-scene optimization is reserved for licensed and original content where quality really matters. Luna: So there's a tiered approach. That makes sense operationally. But let's talk about codecs. Netflix was an early adopter of AV1, the open-source codec. How does that fit into the pipeline? Lucas: Netflix started streaming AV1 in 2020, initially for 2K content on TV. AV1 is about 30 percent more efficient than HEVC, meaning you can stream the same quality at a lower bitrate. But it's also computationally expensive to encode — like, ten to twenty times slower than HEVC. So Netflix uses AV1 selectively: only for high-traffic titles and only for devices that support hardware decoding. Luna: That selective approach is interesting. They're not replacing HEVC wholesale — they're layering AV1 on top, optimizing for the biggest impact. Lucas: Exactly. And because they have per-title encoding, they can decide per scene which codec to use. For a simple scene, HEVC might be good enough. For a complex one, AV1 might be worth the extra encoding time. The pipeline makes that call automatically. Luna: So the real architecture here is a decision engine that's aware of content complexity, target quality, device capabilities, and encoding cost. That's a multi-variable optimization problem. Lucas: Exactly. And they've open-sourced a lot of the tooling around it. Besides VMAF, they released a tool called 'Content Aware Encoding' — which is basically the decision engine logic. It's used by other streaming providers and even broadcasters. Luna: That's a great example of engineering leadership — not just building something for yourself, but contributing it back. It elevates the whole industry. Lucas: Yeah. And it's also good for Netflix — if everyone uses VMAF, then every encoder optimizes for the same metric, which means Netflix's content looks better on competing platforms too. But that also pushes the industry forward. Luna: Before we wrap, I want to ask about operations. How does Netflix encode ten thousand titles a day without backlog? That's a huge pipeline management challenge. Lucas: They run a continuous encoding pipeline — it's not batch-based. When a new title is ingested, it goes into a priority queue. High-demand titles get encoded first, with more encoding recipes. Lower-demand titles might get a quicker pass with fewer recipes. They also have a system that re-encodes existing titles when new codecs or encoding improvements come out — they call it 'continuous quality improvement'. Luna: So they're constantly iterating on their library, not just encoding once and forgetting it. Lucas: Exactly. And they measure everything. They track encoding time, cost, and quality per title, and they have dashboards that show the overall encoding efficiency. If a new GPU driver causes a quality regression, they catch it within hours and roll back. Luna: That level of observability is impressive. It's a lot like how they run their CDN — it's all data-driven. Lucas: Speaking of observability — and I'm just going to say this because it's real — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making deep dives like this possible. No ads, no sponsors, just listener support. If these conversations are useful for what you're building or running, that's the only thing that keeps us going. Luna: Yeah, it's a small group, but it makes a huge difference. We appreciate every single person who does that. Lucas: So back to Netflix — one last thing I want to highlight is how their encoding architecture is evolving for the next generation of video: 8K, HDR10+, and even volumetric video. They're already experimenting with neural network-based encoding, which uses machine learning to predict optimal encoding parameters without the brute-force analysis. Luna: Neural encoding? So instead of running a hundred recipes, the model just predicts the best one based on scene content? Lucas: Exactly. They showed a demo at a conference where a neural network could predict the optimal bitrate and resolution for a scene with 95 percent accuracy compared to the full per-title pipeline. That could reduce encoding time by another order of magnitude. Luna: That's the kind of architecture shift that keeps Netflix ahead. They're not just optimizing for today's content — they're building for tomorrow's formats. Lucas: And that's the lesson for any engineering leader: build a pipeline that can evolve. Per-title encoding was a huge change from fixed ladders, and now they're adding ML on top. It's never finished. Luna: Great point. Thanks for breaking this down, Lucas. That was a fascinating look under the hood. Lucas: Thanks for the great questions. And to our listeners — if you enjoyed this, you know where to find us.