Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Solved the On-Call Burnout Crisis
Transcript
- Lucas: We have been tracking the stability metrics of major platform companies all quarter, but there is one specific operational shift that has caught my eye more than any other in the last six months. It is not about AI inference costs or API latency tuning. It is about how Datadog fundamentally restructured their own on-call rotations to stop the bleeding of senior engineering talent. Luna: You mean they actually capped the hours? Because most companies I talk to treat on-call as a badge of honor rather than a liability. Lucas: Exactly. Datadog implemented a hard cap of thirty hours per week per engineer for primary on-call duties. Before this policy, some senior staff were logging over fifty hours a month responding to alerts that often turned out to be false positives or minor degradations. The decision was driven by data showing that exhausted engineers made slower decisions during actual critical incidents. Luna: That is a huge jump from standard practice. Did they face pushback from the reliability teams who worried about coverage gaps? Lucas: They did, initially. The VP of Infrastructure argued that thirty hours would leave blind spots during high-traffic periods like Black Friday or major product launches. But Lucas, the counter-argument they built was that fatigue causes more downtime than absence. An engineer who is awake but mentally fried will miss a subtle anomaly in the logs that a well-rested engineer would catch instantly. Luna: So they traded raw headcount availability for cognitive sharpness. That sounds like a bet on quality over quantity. Lucas: Precisely. And here is where it gets interesting for listeners managing their own ops teams. Datadog did not just cut hours; they redesigned the alerting pipeline. They introduced a tiered severity system where only P-zero incidents—complete service outage—triggered an immediate page. Everything else went into a digest queue with a four-hour resolution window. Luna: A four-hour window for non-critical issues? That seems risky if you are running a SaaS platform. Lucas: It felt risky until they looked at the historical data. Eighty percent of what used to be considered urgent pages were actually performance degradations that self-healed within twenty minutes. By letting those sit in the digest queue, they eliminated thousands of unnecessary interruptions per month. The result was a forty percent reduction in total page volume. Luna: And yet their mean time to resolution for actual outages dropped. That implies the remaining pages were handled much faster. Lucas: It did drop. Because when the phone rang, the engineer knew it mattered. The psychological signal changed from noise to signal. We see this pattern repeatedly in healthy orgs: clarity of communication reduces anxiety, which improves performance under pressure. Luna: I wonder if smaller startups can afford to make this switch. They usually don't have the redundancy to let tickets sit for four hours. Lucas: That is a fair concern. Startups often operate with thinner margins on both infrastructure and personnel. However, the principle applies regardless of scale. You do not need a massive team to implement tiered alerting. You just need to audit your current pages and ask which ones actually require immediate human intervention versus which ones could wait. Luna: Right. So it is less about having enough people and more about filtering the signal. If everyone is getting pinged for everything, nobody knows what is important. Lucas: Spot on. Noise dilutes importance. When every alert feels like an emergency, no alert feels like one. Datadog’s move was essentially a signal to noise optimization problem disguised as an HR policy change. Luna: I love that framing. signal to noise optimization. It makes the case for better tooling rather than just more bodies. Lucas: It does. And speaking of tools, let us pause for a moment to acknowledge something real. These conversations about sustainable engineering practices are only possible because our listeners keep the network independent. If today’s breakdown of how Datadog managed pager fatigue gave you a new angle on your own team’s workflow, consider supporting us directly. It keeps these deep dives ad-free and ensures we can keep digging into the operational details that matter. You can support the show at buy me a coffee dot com slash fexingo. Luna: Yeah, really appreciate that. It helps us keep doing this research without corporate sponsorship dictating the topics. Lucas: Back to the mechanics of this shift though. There was a second layer to Datadog’s strategy that I think is even more impactful than the hour cap. They completely removed the concept of individual ownership for secondary on-call roles. Luna: What do you mean by removing individual ownership? Usually, teams rotate who is on standby each week. Lucas: Traditionally, Engineering Lead A is on call Monday through Wednesday, and Lead B takes Thursday through Sunday. This creates a 'hero culture' where individuals feel personally responsible for every blip in their window. Datadog switched to a shared pool model. Anyone from the rotation can pick up any ticket, regardless of who is officially scheduled. Luna: So it becomes a collective responsibility rather than a personal burden. That changes the dynamic entirely. Lucas: It does. It forces collaboration. Instead of one person staring at dashboards alone at three in the morning, you get cross-functional input. A backend engineer might solve a database lock issue that a frontend specialist would never have considered. The knowledge sharing accelerates across the whole team. Luna: That also mitigates the risk of key-person dependency. If the lead goes on vacation, the system doesn't break. Lucas: Exactly. Resilience isn't just about servers; it's about people. When knowledge is siloed in one person's head, that is a single point of failure. Shared on-call pools distribute that risk naturally. Luna: I have seen that happen in reverse too. Where the lead hoards knowledge so they remain indispensable. This policy actively prevents that behavior. Lucas: It does. And it rewards mentors. Senior engineers who explain solutions clearly get recognized not just for fixing bugs, but for elevating the entire rotation's capability. It shifts the incentive structure from heroic firefighting to systematic improvement. Luna: So the metric they track now must be different. They probably stopped measuring number of pages resolved and started measuring something else. Lucas: They shifted to measuring 'page inflation rate.' That is the percentage of pages that escalate to higher severity after initial triage. By improving the first responder's context and reducing fatigue, that rate dropped significantly. Fewer mistakes in the first five minutes meant fewer escalations later. Luna: Page inflation rate. That is a brilliant metric. It captures the cost of confusion rather than just the speed of reaction. Lucas: It does. And it aligns with the goal of sustainable growth. You want your systems to stabilize, not just react. If you are constantly fighting fires, you are never building the firewalls. Luna: Which brings us back to the business impact. How did this affect Datadog’s bottom line or their stock performance post-announcement? Lucas: Interestingly, the stock didn't jump immediately. Investors were skeptical. They thought cutting on-call capacity would hurt reliability scores. But over the next two quarters, their customer satisfaction scores regarding uptime actually improved. Clients noticed fewer false alarms and faster resolutions when things truly broke. Luna: So the external perception matched the internal reality. That is rare. Usually, internal efficiency gains take years to translate to customer trust. Lucas: It translates quickly when the product itself is observability. Datadog sells monitoring tools. Their own internal use of those tools to optimize their ops team serves as a live case study. It builds credibility with prospective customers who are struggling with the same chaos. Luna: Eating your own dog food in the most literal way. If you can fix your own pager, you can definitely fix mine. Lucas: Exactly. It is a powerful marketing narrative that happens organically. No press release needed. Just happy engineers shipping code instead of answering pagers at midnight. Luna: Do you think this model scales to hyper-growth companies? Like those doing IPOs right now? Lucas: I think it scales better than traditional models. Hyper-growth companies suffer most from burnout because they hire fast and expect instant results. The chaotic on-call structures typical of scaling firms are a major driver of attrition. Standardizing this approach early saves them millions in recruitment costs later. Luna: Recruitment costs are huge. Replacing a senior engineer can cost over one hundred thousand dollars when you factor in lost productivity and hiring fees. Lucas: And that is before you account for the institutional knowledge that walks out the door. Retention is cheaper than replacement. Datadog’s thirty-hour cap is essentially a retention strategy wrapped in an operational policy. Luna: It reframes the entire conversation. It is not about being nice to employees. It is about protecting the asset. Lucas: Precisely. Talent is the asset. Systems are just the vehicle. If you break the people driving the vehicle, the vehicle crashes regardless of how good the engine is. Luna: So for CTOs listening, the takeaway is to audit your pages and your people. Cut the noise, share the load, and measure the calm. Lucas: Audit your pages, yes. But also measure the calm. Look at your team’s engagement surveys. If people are dreading the weekend rotation, you have a structural problem that no amount of automation can fully fix. Luna: Automation handles the detection. Culture handles the response. You need both. Lucas: You need both. And as we look toward the rest of the year, with potential regulatory changes in data privacy and continued volatility in cloud pricing, stable engineering teams will be the ones who can adapt fastest. Luna: Stability breeds agility. A burned-out team is rigid. A rested team is flexible. Lucas: I think that is the core insight. Flexibility comes from bandwidth. And bandwidth comes from boundaries. Datadog drew a boundary at thirty hours, and gained flexibility in return. Luna: Boundaries create freedom. That is a lesson that applies far beyond just server maintenance. Lucas: It does. Whether you are managing code reviews, design sprints, or quarterly planning, defining what is off-limits protects what is essential. Luna: So the next time someone asks why you are limiting on-call hours, tell them you are buying freedom. Lucas: Tell them you are investing in resilience. And maybe mention that Datadog’s stock has been very resilient lately. Luna: Or just mention that your engineers are sleeping through the night. That sells itself. Lucas: It does. Let us see if more leaders adopt this thirty-hour standard by Q4. If they do, we might see a broader shift in how the industry values operational sanity over heroic effort. Luna: I hope so. The industry needs to normalize rest as a professional competency, not a luxury. Lucas: Agreed. Rest is a competency. And Datadog proved it drives better outcomes. Thanks for tuning in to The CTO Podcast.