Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Built Developer Trust Through Incident Transparency
Transcript
- Lucas: If these conversations are useful for what you're building or running, we appreciate you tuning in. But let's get straight into the mechanics of how Datadog handles the inevitable moment when their own platform goes dark. Luna: Right, because as a monitoring company, their credibility is literally tied to uptime. So when the lights go out on themselves, the pressure isn't just technical—it's existential. Lucas: Exactly. And most companies handle that panic by going silent. They lock down communications, issue vague status page updates, and wait until things are fixed to say anything. Datadog does the opposite. They treat their internal incident review process like a public product feature. Luna: That sounds counterintuitive. Admitting your system failed seems like it would hurt stock prices or churn customers. Lucas: It does in the short term, but long-term data suggests otherwise. The key is the speed and specificity of their post-mortem. We’re looking at a model where they publish detailed root cause analyses within forty-eight hours of resolution. Not weeks later, not buried in an appendix. Front and center. Luna: Forty-eight hours is incredibly fast for enterprise-grade engineering teams. What kind of details do they actually include? Lucas: They don't just say 'server error.' They show the timeline. Minute-by-minute breakdowns of when the anomaly was detected, when alerts fired, and crucially, when human intervention happened. They map the gap between automated detection and manual action. Luna: So they’re auditing their own reaction time. That’s a bold move. Do they assign blame, or is it purely systemic? Lucas: Strictly systemic. This is the 'blameless' philosophy taken to its logical extreme. If a developer pushed bad code, the report doesn't name them. It names the lack of guardrails that allowed that code to reach production without sufficient testing. Luna: I’ve seen how hard it is to get engineering leads to agree on blameless reviews. Is that cultural or structural? Lucas: Structural, mostly. They have clear thresholds for what constitutes a major incident. If it impacts more than five percent of users for ten minutes, it triggers an automatic review workflow. It’s not discretionary. The system forces the transparency. Luna: Automatic workflows remove the ego element. That makes sense. But does this actually help them retain customers during the outage? Lucas: Yes, because it gives customers something to hold onto: predictability. When clients see a Datadog post-mortem, they realize the failure was isolated, understood, and addressed with concrete architectural changes. It turns fear into confidence. Luna: Confidence built on evidence rather than marketing speak. I like that. What about the follow-through? Do they actually fix the problems they identify? Lucas: This is where many companies fail. They write the report and file it away. Datadog ties every post-mortem finding directly to a ticket in their project management system. Those tickets are visible internally and often referenced in subsequent releases notes. Luna: Visible internally is one thing. Referencing them publicly is another. Does that risk exposing too much internal chaos? Lucas: Not if you frame it right. They focus on the solution, not the mess. For example, after a recent latency spike, they didn't just say 'we added caching.' They explained why the previous caching strategy failed under specific load conditions. Luna: Educating the customer base while apologizing. That’s a sophisticated dual-purpose communication strategy. Lucas: It is. And it reinforces their brand as experts. If you can explain your own failure better than your competitors can explain their successes, you win trust. Luna: But not all companies have the luxury of being a monitoring tool provider. Most businesses are opaque by nature. Can smaller tech firms replicate this? Lucas: They can adopt the rhythm, if not the scale. Start with internal post-mortems that are rigorous. Then, pick one minor incident per quarter to share externally. Just a simple email to subscribers explaining what went wrong and what changed. Luna: A quarterly transparency update. That feels manageable. Does it require legal clearance? Lucas: Usually yes, but the template is consistent. Stick to facts: what happened, impact scope, resolution steps, preventive measures. Avoid speculative language. Legal teams prefer boring reports over dramatic ones. Luna: Boring is good. Boring is safe. What role does leadership play in this process? Do CEOs usually get involved in writing these? Lucas: The CTO or VP of Engineering signs off. Sometimes the CEO adds a brief note, but the heavy lifting is technical. Leadership provides the cover to ensure the team isn't punished for speaking up. Luna: That protection is vital. Without it, engineers will sanitize reports to protect themselves. Lucas: Absolutely. Sanitized reports are useless. You need raw data to improve systems. Datadog incentivizes honesty through promotions and recognition, not punishment. Luna: Incentivizing honesty is almost harder than managing code quality. How do they measure success in this area? Lucas: They track Mean Time to Resolution, obviously. But also Mean Time to Acknowledge and Mean Time to Publish the Post-Mortem. Speed of communication becomes a KPI alongside uptime. Luna: Speed of communication as a KPI. That shifts the entire operational mindset. It stops people from hiding issues until they're solved. Lucas: It forces proactive engagement. If you know you’ll be scrutinized for delays in reporting, you prioritize clarity immediately. Luna: Does this create more work for the engineering team? Writing detailed reports takes time away from coding. Lucas: Initially, yes. But over time, templates streamline the process. The real cost is in the initial cultural shift. Once everyone accepts that transparency is part of the job description, the friction disappears. Luna: Friction is always highest at the start. What about competitive advantage? Does this actually drive revenue? Lucas: Indirectly, yes. Enterprise buyers evaluate risk heavily. A vendor with transparent, rapid recovery processes is lower risk than one with black-box operations. It becomes a differentiator in RFPs. Luna: Lower risk equals higher conversion. That’s a solid business case for openness. Are there any downsides to this model? Lucas: Only if you lie. If you promise transparency but deliver vague statements, you lose twice: once for the outage, twice for the dishonesty. The bar is high. Luna: High bar, high reward. It’s essentially a trust compound interest strategy. Lucas: Precisely. Every honest report compounds the trust capital. Over years, that creates a moat that features alone cannot build. Luna: Before we dive deeper into the metrics of those reports, I want to pause for a second. If these conversations are useful for what you're building or running, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo — that's F. E. X. I. N. G. O. Lucas: Yeah, and no pressure at all. Just if you found value in how we broke down Datadog's transparency model, that support helps us keep producing deep dives like this week in and week out. Luna: Thanks for supporting independent analysis of how tech companies operate behind the scenes. Lucas: Let's get back to the numbers. When Datadog publishes these reports, what is the average length? Are they dense technical documents? Luna: I checked a few. They vary, but typically around fifteen hundred words. Concise enough to read in ten minutes, detailed enough for engineers to verify. Lucas: That length is strategic. Long enough to demonstrate rigor, short enough to respect the reader's time. They use headers, bullet points, and links to relevant documentation. Luna: Scannability is key. Executives skim for impact, engineers drill down for detail. The format accommodates both audiences simultaneously. Lucas: Exactly. And they link to the actual pull requests or commit hashes where fixes were applied. Full traceability. Luna: Linking directly to code commits is rare. Most companies stop at the summary level. Why go that far? Lucas: Because it proves they didn't just talk about fixing it—they actually did it. It closes the loop completely. No ambiguity. Luna: Ambiguity is where distrust lives. Closing the loop removes the doubt. Does this influence how other vendors structure their own responses? Lucas: It sets a benchmark. Competitors are watching. If Datadog is publishing detailed timelines, others look sloppy by comparison. It raises the industry standard. Luna: Raising the floor for everyone. That benefits the whole ecosystem, even if it puts pressure on rivals. Lucas: It forces innovation in communication, not just code. Reliability isn't just about uptime anymore; it's about perceived reliability. Luna: Perceived reliability driven by radical honesty. That’s a powerful insight for any leader managing technical teams today. Lucas: Agreed. As we look toward Q4 of 2026, with increased scrutiny on AI infrastructure stability, this model might become even more critical for cloud providers. Luna: Especially with AI models hallucinating or failing unpredictably, showing exactly how you detect and contain those errors could be the ultimate selling point. Lucas: Controlling the narrative of failure is becoming as important as preventing the failure itself. It’s a subtle shift, but a profound one. Luna: Do you think we'll see more standardized formats for incident reports across the industry, similar to financial disclosures? Lucas: Possible. Regulatory bodies are starting to ask about digital resilience. Standardization might follow regulation. Luna: Standardized incident reports. Imagine the efficiency gains if every company used the same template for transparency. Lucas: It would make comparing vendor reliability much easier for buyers. Until then, Datadog’s ad-hoc but rigorous approach remains the gold standard. Luna: Gold standard earned through sweat equity and radical candor. Hard to replicate, easy to admire. Lucas: Hard to replicate, essential to emulate. For anyone leading a tech team, the question isn't whether you can afford to be transparent—it's whether you can afford not to be. Luna: A compelling way to close it out. Transparency as a survival strategy, not just a nice to have policy. Lucas: Exactly. As systems grow more complex, our ability to communicate complexity honestly will define our longevity.