Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Pixar Rebuilt Its Render Farm for Real-Time
Transcript
- Lucas: If these conversations are useful for what you're building or running, I want to flag something quickly. Luna: Go for it. Lucas: This show stays ad-free because of a small group of listeners who toss in a couple of dollars a month at buy me a coffee dot com slash fexingo. It genuinely makes the difference. Luna: Yeah, and it's not about gatekeeping — it's about keeping this a resource for anyone who wants it. Lucas: Exactly. If you've gotten something out of the show, that link matters more than you'd think. Okay — back to architecture. Lucas: So Pixar's render farm — for decades it was the gold standard of batch rendering. You'd submit a job, wait hours or days, get back frames. That worked fine for theatrical releases where you have months of lead time. Luna: But then Disney Plus happened, and suddenly Pixar needed to produce episodic content and interactive experiences on tighter deadlines. Lucas: Right. The old model couldn't sustain that. So around 2020, Pixar Engineering started a multi-year rebuild of their core rendering infrastructure. Not just scaling up the farm — rethinking the entire pipeline. Luna: What was the fundamental tension they had to resolve? Lucas: Determinism versus speed. In batch rendering, you can afford to be slow as long as every frame is pixel-perfect and reproducible. Real-time rendering introduces non-determinism — frames may look slightly different each pass because of GPU scheduling, memory access patterns, things like that. Luna: And for Pixar, the brand is built on that deterministic quality. You can't have 'Soul' shifting hues between renders. Lucas: Exactly. So the engineering team took a layered approach. First, they decoupled the scene graph — the logical description of a scene — from the rendering engine. That's not new conceptually, but Pixar did it in a way that let them swap between batch and real-time renderers without changing the authoring tools. Luna: So an artist working in Presto — Pixar's internal animation tool — doesn't even know which renderer will process their work. Lucas: Correct. The scene graph is compiled into an intermediate representation that both RenderMan — the batch engine — and a new real-time engine called Hydra can consume. Hydra was actually open-sourced as part of the USD framework, but Pixar's internal version is heavily customized. Luna: And Hydra runs on GPU clusters, not the traditional CPU farm. Lucas: Yes. That's the second layer. They moved from a homogeneous cpu based render farm to a heterogeneous cluster with NVIDIA GPUs, AMD GPUs, and even some Intel ARC cards for testing. The big challenge was making sure the same scene graph produced the same result across different hardware. Luna: That sounds like a nightmare for driver-level bugs. Lucas: It was. The team built a continuous integration pipeline that renders a reference set of frames on every hardware combination every night. If a single pixel differs beyond a tolerance — usually one one-hundredth of a percent — they flag it and investigate. That caught several driver regressions early. Luna: And how did the storage architecture change? Because batch rendering can stream data in a predictable pattern, but real-time needs random access. Lucas: They moved from a san based system — storage area network — to a unified object store called PixarFS, built on top of Ceph. Every frame, every texture, every geometry cache is stored as objects, and the renderers pull what they need via a content-addressable hash. Luna: So no more 'mount this volume to render node seventeen'. Lucas: Right. The farm became stateless from the renderer's perspective. You spin up a container, it authenticates to PixarFS, pulls the assets it needs, renders, and pushes the output back. That made it trivial to scale out — Pixar can now burst into AWS for peak loads during a crunch period. Luna: But burst to the cloud introduces latency. How do they handle that for real-time work where you need frames in milliseconds? Lucas: They don't burst to cloud for real-time rendering. The cloud is only for batch jobs that can tolerate a few seconds of additional latency. Real-time rendering stays on-prem, close to the storage, with InfiniBand interconnects. They use a technique called 'render affinity' — the scheduler tries to keep a scene's data cached on the same nodes across multiple frames. Luna: So it's a hybrid model: on-prem for latency-sensitive work, cloud for burst capacity. Lucas: Exactly. And that hybrid model is now powering new workflows. For example, during the production of 'Elemental', the team used real-time previews in the director's review sessions. The director could tweak lighting and see results in seconds instead of overnight. Luna: That must have changed the creative process significantly. Lucas: It did. Pete Docter famously said in an internal talk that real-time rendering let them iterate on visual jokes that would have been cut for time before. The cost structure also shifted — Pixar now spends about sixty percent of its rendering budget on real-time infrastructure, forty percent on batch. That's inverted from five years ago. Luna: And what about the reliability side? If a real-time renderer crashes during a director review, you lose the creative moment. Lucas: They built a checkpointing system that saves the render state every few seconds. If a node fails, another node picks up from the last checkpoint within two seconds. That's transparent to the artist — they just see a slight hitch. The system is designed for five nines availability during review sessions, which is remarkable for a gpu bound workload. Luna: How did they achieve that level of reliability on consumer-grade GPUs? Lucas: They don't use consumer cards. Pixar uses datacenter GPUs — NVIDIA A100s and H100s — with ECC memory and enhanced thermal management. But even then, they saw higher failure rates than CPUs. The team implemented predictive failure detection using GPU temperature trends and memory error rates. If a card starts showing correction errors, it's proactively taken offline. Luna: So they treat GPUs as cattle, not pets. Lucas: Exactly. And that's a lesson for any engineering org moving to gpu heavy workloads: expect higher failure rates and build your orchestration accordingly. Luna: One thing I find interesting is that Pixar published very little of this publicly. Most of what we know comes from a few conference talks and job postings. Lucas: That's typical for a studio that treats its technology as a competitive advantage. But the patterns they used — scene-graph abstraction, unified storage, hybrid cloud, proactive fault detection — are transferable to any organization with a legacy system that needs to support new real-time demands. Luna: So if you're a CTO looking at your own render farm — or any batch processing pipeline — the question is: can you decouple the data model from the execution engine? Lucas: Yeah. And can you afford to let your artists — or your users — see results in seconds instead of tomorrow? Because that's where the creative leverage really is. Luna: And the cost is not just infrastructure — it's the engineering investment to rebuild those abstractions. Lucas: Right. Pixar spent about three years and an estimated two hundred million dollars on this transformation. But for them, the return is faster iteration, higher quality, and new revenue streams from streaming and interactive. For smaller teams, the same principles apply at a different scale. Luna: What's the one takeaway you'd offer a CTO listening? Lucas: Start with the interface between your data and your compute. If that interface is tightly coupled, you're locked into your current hardware and speed. Pixar showed that investing in a clean abstraction — the scene graph — unlocked a whole new set of capabilities without rewriting the authoring tools. That's the kind of architectural leverage worth pursuing. Luna: And maybe worth a couple of dollars a month at buy me a coffee dot com slash fexingo. Lucas: I knew you'd circle back to that. Alright, that's our time. Thanks for listening.