Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Scaled Engineering Without Burning Out
Transcript
- Lucas: There is a specific kind of chaos that happens when a software company doubles its headcount in eighteen months. You stop being a group of friends building something cool, and you become a machine that needs gears to stop grinding each other to dust. Luna: Exactly. And usually the first thing to break isn't the product, it's the trust between the teams shipping features and the teams trying to keep the lights on. Lucas: Right. So let's talk about Datadog. They crossed the fifty billion dollar market cap mark last year, but the real story here is what happened internally around two thousand twenty-two. That was the period where they went from roughly two thousand engineers to nearly four thousand in under two years. Most companies would have imploded by then. Luna: I remember reading their internal blog posts from that time. They weren't talking about revenue anymore; they were talking about latency in their own pull request reviews. Lucas: That's the anchor. Pull request latency. It sounds like a minor operational metric, but it's actually a proxy for organizational health. When your average wait time for a code review spikes past forty-eight hours, developers start bypassing checks, merging risky code, or just quitting. Datadog noticed their cycle times were stretching into weeks. Luna: So they didn't just hire more reviewers. They changed the actual structure of how work flowed through the company. Lucas: They implemented what they called Platform Engineering as a service. But I want to be careful with that buzzword. It wasn't about building a shiny new tool for fun. It was about taking the repetitive, draining infrastructure tasks away from feature teams so they could focus on business logic. Luna: Which means the engineers writing the billing code didn't have to worry about configuring the Kubernetes clusters underneath it? Lucas: Precisely. Before this shift, every single squad had to understand the minutiae of their own deployment infrastructure. That created massive cognitive load. By centralizing that into a dedicated platform team, they reduced the context switching for the average engineer by nearly thirty percent. Luna: Thirty percent seems high for just offloading infrastructure work. Was there a productivity gain elsewhere? Lucas: The gain was in consistency. When everyone manages their own infra, you get thirty different ways to do the same thing. With the platform abstraction, they standardized on one path. That meant fewer outages caused by human error during deployments. Luna: And fewer outages means less fire-fighting, which means happier engineers who aren't getting woken up at three in the morning. Lucas: You hit the nail on the head. Burnout in tech is rarely about working hard. It's about working on things that don't matter or fixing problems caused by yesterday's shortcuts. Datadog realized they were accumulating invisible debt in their processes. Luna: So they paid it down. But did the platform team become a bottleneck themselves? Centralization often creates that exact problem. Lucas: It's the classic paradox. If the platform team is too slow, the whole company stalls. Datadog solved this by treating the platform like a product. They gave internal customers, meaning the feature teams, a voice in the roadmap. If a feature team needed a new capability, they had to justify it with data. Luna: That shifts the dynamic from a gatekeeper to a service provider. The platform team has to earn their keep. Lucas: Exactly. And they measured success not by how many tickets they closed, but by how fast the feature teams could ship. It was a feedback loop designed to align incentives. The platform only wins if the rest of the company ships faster. Luna: That's a really smart way to measure internal efficiency. Usually, internal tools are judged on uptime alone. Lucas: Uptime is table stakes. Speed to value is the real metric. This approach allowed them to scale without adding layers of middle management. They didn't need more VPs to coordinate the chaos because the platform handled the coordination automatically. Luna: It feels like they turned their internal complexity into a competitive advantage rather than letting it become a liability. Lucas: Completely. Most companies let complexity win. Datadog engineered against it. Now, before we go deeper into the metrics, I want to pause for a second. Luna: Sure, what's on your mind? Lucas: If these conversations are useful for what you're building or running, consider supporting the show directly. A handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these episodes possible. No ads, just us talking shop. Luna: Yeah, it keeps the lights on and lets us dig into these deep dives without chasing advertisers. Lucas: Back to Datadog. Once they stabilized the platform, they looked at the data coming out of their systems. They found that a significant portion of their engineering capacity was spent on debugging observability issues within their own tools. Luna: Irony much? They build monitoring tools, but their own monitoring was noisy? Lucas: Terrible irony. They realized their alerting thresholds were set too aggressively. Engineers were getting pinged for non-critical events so often that they started ignoring alerts entirely. This is known as alert fatigue, and it's a massive safety risk. Luna: So they audited their own alerts. That must have been a painful process for the teams used to reacting instantly. Lucas: They reduced their total alert volume by sixty percent. How? By introducing severity levels that actually mattered. Instead of every warning being equal, they grouped signals. If five servers had a slight CPU spike, it was one ticket, not five panic-inducing pages. Luna: That requires discipline. You have to trust the system to catch the critical stuff even if you miss some noise. Lucas: It does. But the data showed that zero critical incidents were missed during the filtering period. In fact, response times improved because engineers were focusing only on true emergencies. It restored sanity to the on-call rotation. Luna: On-call rotations are usually the number one complaint among senior engineers. Fixing that probably boosted retention significantly. Lucas: Retention data supports that. Their voluntary turnover rate dropped by twelve percent in the quarters following the alert overhaul. That saves millions in recruiting and training costs, especially at their scale. Luna: Twelve percent is huge. You lose key institutional knowledge when people leave, and replacing them takes months. Lucas: Right. So we have two major interventions: Platform Engineering to reduce cognitive load, and Alert Hygiene to reduce stress. Both were internal process fixes, not product features. Luna: But they drove external results. Faster shipping, fewer outages, happier staff. It’s a virtuous cycle. Lucas: A virtuous cycle driven by unsexy operational details. Most investors look at the product roadmap. Smart operators know the engine room matters more. Luna: Do you think smaller companies can replicate this without the budget for a dedicated platform team? Lucas: They can adopt the philosophy. Even if you don't have a separate team, you can standardize your CI/CD pipeline. You can audit your alerts. The principle is the same: reduce friction and respect the engineer's attention. Luna: Standardization over customization. That's a hard sell when you're in startup mode, but essential for scale. Lucas: Exactly. Startup mode is about doing anything to survive. Scale mode is about doing one thing consistently. Datadog made the pivot deliberately. Luna: It’s interesting how they used their own product to fix their own problems. Did they sell those improvements to customers later? Lucas: They absolutely did. The alerting logic they refined became a core feature of their Intelligence Platform. They ate their own dog food and then sold the leftovers at a premium. Luna: That’s the best kind of R&D. Solving your own pain points and packaging the solution. Lucas: It turns internal suffering into external product value. It’s a model worth watching as more companies hit that four-thousand engineer mark. Luna: Or wondering why their own teams are burning out right now. Lucas: Look at your own pull request latency next week. It might tell you more about your future than your revenue projections do. Luna: I’ll check mine. Thanks, Lucas.