Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Prevented AI Hallucinations in Production
Transcript
- Lucas: We have spent years talking about how companies build observability tools, but today I want to flip the script. Datadog is not just selling monitors; they are deploying large language models to help engineers debug systems in real time. The problem is that when an AI suggests a configuration change or writes a fix for a production error, it can hallucinate. Luna: That sounds terrifying for a company whose entire value proposition is reliability. If their AI breaks the very thing it is supposed to be fixing, what is the fallback? Lucas: The fallback is a strict guardrail system that requires human approval for any write operation. Lucas here, and we are looking at how Datadog manages the risk of generative AI in a high-stakes environment. They treat every ai generated suggestion as a draft, never a command. Luna: So they are essentially using AI to propose solutions while keeping humans firmly in control of the execution. That makes sense for trust, but does it slow down the debugging process too much? Lucas: It does add latency, but the alternative is catastrophic failure. Let us look at the mechanics. When an engineer asks the AI why a service is failing, the model might suggest rolling back a deployment or changing an environment variable. Datadog’s system validates these suggestions against a sandbox before presenting them to the user. Luna: I assume this sandboxing has to be incredibly fast. Engineers do not want to wait ten minutes for a simulation to prove a fix works. Lucas: Exactly. They use lightweight containerization to run these simulations in seconds. However, the real innovation is not speed; it is accuracy. They have trained their models on decades of actual incident data, so the AI knows which suggestions are likely to cause more problems than they solve. Luna: Training on historical incident data seems like a smart way to ground the model. Does that mean the AI learns from past mistakes rather than just general internet knowledge? Lucas: Precisely. General LLMs pull from public repositories, which often contain outdated or insecure code. Datadog’s models are fine-tuned on anonymized customer data, ensuring they understand the nuances of enterprise architecture. This reduces the chance of suggesting a deprecated library or a risky patch. Luna: That is a significant competitive moat. Most competitors are still wrestling with generic models that don’t understand corporate infrastructure constraints. Lucas: Right. And this brings us to the core issue: hallucination detection. Even with fine-tuning, LLMs can drift. Datadog implemented a secondary layer of validation called intent verification. Before any suggestion is shown, a smaller, faster model checks if the intent matches the context of the alert. Luna: So you have a big model generating ideas and a small model acting as a gatekeeper. Is that efficient enough for thousands of simultaneous alerts? Lucas: It is designed to be. The small model acts as a filter, rejecting nonsensical suggestions before they ever reach the user interface. This keeps the experience clean and prevents engineers from being bombarded with irrelevant or dangerous advice. It is a two-stage filter that prioritizes safety over volume. Luna: I wonder if this approach changes how engineering teams trust their tools. Do people start relying on the AI too quickly? Lucas: That is a valid concern. Datadog actively discourages blind trust. Their UI clearly labels ai generated content and encourages users to verify the logic. They even provide links to the relevant logs and metrics that support the suggestion, so the engineer can see the evidence themselves. Luna: Transparency is key. If the AI just says 'fix this' without showing why, no one will use it. Showing the reasoning builds confidence. Lucas: Absolutely. But there is another layer: feedback loops. Every time an engineer accepts or rejects an AI suggestion, that data is fed back into the model. This continuous learning process helps the AI improve over time, becoming more accurate with each interaction. Luna: So the more you use it, the better it gets, provided you keep correcting it when it goes wrong. That feels like a standard machine learning cycle, but applied to operational safety. Lucas: It is, but with higher stakes. In a web app, a bad recommendation might just annoy a user. In observability, it could take down a payment gateway. Datadog treats these interactions as high-value training data, not just engagement metrics. Luna: This raises the question of data privacy. You mentioned anonymized customer data earlier. How strictly do they enforce that boundary? Lucas: Very strictly. Customer data is isolated by tenant ID. No cross-tenant leakage is allowed. They undergo regular third-party audits to ensure that one company’s insights never leak into another’s model updates. Compliance is baked into the architecture, not bolted on. Luna: That level of isolation must be expensive to maintain. Are they passing those costs on to enterprises? Lucas: They are, but enterprises are willing to pay for it. Trust is the currency of the observability market. If Datadog loses that trust through a data breach or a harmful AI suggestion, they lose everything. The cost of security is an investment in retention. Luna: It reminds me of how financial institutions handle trading algorithms. Speed matters, but integrity matters more. One glitch can trigger a flash crash. Lucas: Good analogy. And just like in finance, the regulations are tightening. As AI becomes more embedded in critical infrastructure, regulators will likely demand more transparency and accountability from vendors like Datadog. Luna: Do you think current frameworks are ready for this? Or are we going to see a wave of new compliance rules specifically for AI in IT operations? Lucas: I think we are already seeing it. The EU AI Act and similar US guidelines are starting to classify certain AI uses as high-risk. Datadog is positioning itself ahead of these curves by building compliance into their product design rather than reacting to it later. Luna: Proactive compliance is always smarter than reactive panic. It saves legal headaches and builds brand equity simultaneously. Lucas: Exactly. Now, before we dive deeper into the technical specifics of their sandboxing architecture, I want to take a brief moment. If these conversations are useful for what you are building or running, and you find value in the deep dives we do here, consider supporting the show directly. Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. It helps keep the research sharp and the episodes ad-free. Luna: Thanks for the support, everyone. It really does make a difference in keeping the quality high. Now, back to the sandboxing. Lucas: Let us talk about the technical implementation of that sandbox. Datadog uses ephemeral environments that spin up only when needed. These environments mimic the customer’s actual stack but with synthetic traffic. This allows the AI to test fixes without risking real users. Luna: Ephemeral environments sound costly in terms of compute resources. How do they manage the overhead of spinning these up frequently? Lucas: They optimize heavily. Instead of full VMs, they use lightweight containers that share kernel resources. The startup time is measured in milliseconds, not minutes. This efficiency makes it feasible to run hundreds of simulations concurrently during peak incident times. Luna: Millisecond startup times are impressive. That means the AI can iterate rapidly, trying multiple hypotheses in parallel. Does it prioritize which hypotheses to test first? Lucas: Yes, it uses a scoring system based on probability. Suggestions that align with common failure patterns are tested first. Less likely scenarios are queued behind them. This ensures that the most probable fixes are presented to the engineer almost immediately. Luna: Prioritization by probability is a classic heuristic. It balances speed with coverage. What happens if the top-ranked hypothesis fails in the sandbox? Lucas: The system moves to the next candidate. It does not give up until it finds a viable solution or exhausts the list. If no solution is found, it returns a detailed analysis of the logs, asking the engineer to provide more context. It admits when it doesn’t know. Luna: Admitting ignorance is a huge virtue in AI systems. It prevents the AI from making things up just to please the user. Honesty builds better long-term relationships. Lucas: Agreed. And this honesty extends to explaining limitations. If the AI detects that the issue might be hardware-related rather than software-related, it flags that distinction clearly. It guides the engineer toward the right team or tool. Luna: So it acts less like a magic box and more like a senior engineer who knows when to call in a specialist. That is a very helpful mental model. Lucas: That is exactly the goal. To augment human expertise, not replace it. The AI handles the pattern matching and data retrieval, while the human provides the judgment and context. It is a partnership. Luna: Partnerships require clear roles. How do they define the boundary between what the AI should do autonomously versus what requires manual intervention? Lucas: The boundary is defined by impact. Read-only actions, like summarizing logs or suggesting queries, can be autonomous. Write actions, like restarting services or changing configs, always require approval. This policy is hard-coded and cannot be overridden by the AI. Luna: Hard-coded policies are the only way to ensure consistency. Software can evolve, but safety rules need to be immutable. Lucas: Immutable safety rules are the foundation of trust. Without them, the entire system collapses under the weight of uncertainty. Datadog understands that in observability, certainty is the product. Luna: Certainty in a chaotic world. That is a powerful promise. As we wrap up, what is the one takeaway for CTOs watching this? Lucas: Takeaway number one: Never let AI act without guardrails. Takeaway number two: Invest in human-in-the-loop workflows. And takeaway number three: Measure your AI’s accuracy as rigorously as you measure your uptime. Luna: Accuracy metrics for AI are just as critical as performance metrics for servers. We need to track false positives and user satisfaction alongside traditional KPIs. Lucas: Spot on. As AI becomes more embedded in our tools, we must treat it with the same rigor we apply to our core infrastructure. The future of tech leadership is managing these hybrid human ai systems effectively. Luna: Managing the hybrid is the new skill set. I wonder how many organizations are actually ready for that transition yet. Lucas: Fewer than we would like to admit. But the ones who adapt early will have a massive advantage in efficiency and resilience. The question is not if AI will change operations, but how well we prepare for it.