Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Stabilized AI Observability
Transcript
- Lucas: If you are building or running an ai enabled product right now, you probably have a new kind of technical debt that doesn't show up in your standard dashboards. It is not about CPU usage or memory leaks. It is about unpredictability in the black box. Luna: Right. Traditional monitoring tells you if the server is alive. It does not tell you if the model just started hallucinating a completely different answer than it gave five minutes ago. Lucas: Exactly. And that is the crisis I want to drill into today. We looked at how Datadog approached this problem, not by trying to fix the models themselves, but by fixing the way they observe them. They built a specialized observability framework for generative AI workloads that shifts the focus from simple request-response metrics to trace-based context tracking. Luna: So instead of just measuring how long a query took, they are tracking the entire chain of reasoning steps within the system? Lucas: Precisely. For years, engineering teams treated large language models like sophisticated database lookups. You send a prompt, you get a response, you log the duration. But when you are dealing with non-deterministic outputs, that binary success metric breaks down. A response can be technically fast and correct, or slow and wrong, or fast and dangerously confident. Datadog realized that to stabilize these systems, you need visibility into the tokens as they are generated, not just the final result. Luna: That makes sense because the cost and the risk are often distributed across those intermediate steps, especially in complex agentic workflows. Lucas: Spot on. The specific architectural shift here is what they call semantic drift detection. In their internal rollout, which stabilized before we even got to mid-2026, they found that without granular trace data, they were missing subtle degradations in output quality. They started seeing patterns where the model would slowly drift away from factual accuracy over a long conversation thread, something no standard latency alert could catch. Luna: It feels like we are moving from monitoring infrastructure to monitoring intelligence, which is a much harder line to draw. Lucas: It is harder, but necessary. The core insight from the Datadog approach was treating the AI inference pipeline as a distributed system with its own unique failure modes. They introduced token-level latency tracking. Most platforms just measure time-to-first-token or total completion time. Datadog’s framework breaks that down further, allowing engineers to see exactly where the bottleneck is—is it the retrieval step, the reasoning step, or the generation step? Luna: And does that granularity actually help with the business side, or is it just for engineers who want more pretty graphs? Lucas: It hits the bottom line directly. When you can isolate the retrieval step as the source of latency, you can optimize your vector database independently of your model provider. That decoupling is huge for cost control. If you don’t have that visibility, you are just throwing money at the model provider hoping they will speed up. With trace-based tracking, you know exactly which part of your stack needs investment. Luna: I like that. It turns a vague complaint about slowness into a specific engineering ticket. Lucas: Yes, and it also changes how you handle errors. In traditional software, an error is usually a crash or a timeout. In AI, an error might be a plausible-sounding lie. Datadog’s framework allows for automated evaluation pipelines that run against every outgoing response. They compare the output against a set of ground-truth criteria using a smaller, faster model to judge the larger one. It is a meta-monitoring layer. Luna: Using a smaller model to grade the bigger one sounds efficient, but isn’t there a risk of bias in the grading logic? Lucas: That is a fair concern, but it is manageable. The key is that the grader is deterministic. You are not asking the small model to be creative; you are asking it to check against a rubric. Did the response contain the required facts? Was the tone appropriate? Is the format correct? These are pass-fail conditions, not subjective judgments. This creates a consistent baseline for quality that doesn’t fluctuate with the underlying model’s temperature setting. Luna: So you are essentially creating a service level objective for something that has never had one before. Lucas: You are hitting the nail on the head. We used to have SLOs for uptime and latency. Now we are building SLOs for accuracy and relevance. Datadog’s work has pushed the industry toward defining what a ninety-ninth percentile accurate response actually looks like. It is a massive cultural shift for engineering organizations that are used to binary metrics. Luna: It must be tough for CTOs to sell that to finance teams who only understand server costs. Lucas: The bridge is cost per successful outcome. Once you have the trace data, you can calculate the true cost of a bad answer. If a customer gets a wrong shipping estimate because the model hallucinated, that is a refund plus a support call. Datadog’s framework helps quantify that leakage. It moves the conversation from 'AI is expensive' to 'Our current observability gap is costing us fifteen percent in operational waste.' Luna: That is a compelling argument for budget approval. But implementing all of this tracing adds overhead to the system itself. Lucas: It does, but the overhead is relatively low compared to the cost of undetected failures. The instrumentation is mostly done at the application layer, wrapping the API calls. It does not require deep kernel modifications. The trade-off is worth it because you gain the ability to reproduce issues. In traditional debugging, you guess what went wrong. In AI observability, you replay the exact input sequence and see the exact output path. Luna: Replayability is a game changer for debugging non-deterministic systems. Lucas: Absolutely. And this brings us to the broader implication for technical leadership. As we move through the rest of 2026, the companies that win will not be the ones with the best models, since most access the same foundation models. They will be the ones with the best feedback loops. Datadog’s approach shows that robustness comes from visibility, not just raw compute power. Luna: So the moat is shifting from the model to the middleware that manages it. Lucas: Exactly. The value proposition is moving upstream to the orchestration layer. If you can prove that your system consistently delivers high-quality outputs with lower latency than your competitor’s, you have a competitive advantage. That requires the kind of detailed trace data that Datadog helped popularize. Luna: It seems like a lot of this relies on standardized logging formats. How universal is this solution across different providers? Lucas: Standardization is the next big hurdle. Currently, each major model provider has its own telemetry format. Datadog acts as a normalization layer, ingesting logs from OpenAI, Anthropic, Google, and open-source models alike, and converting them into a unified schema. This prevents vendor lock-in at the observability level, which is crucial for enterprises that might swap model providers seasonally to get the best price-performance ratio. Luna: That flexibility is vital. You don't want to be stuck with a slow model because your monitoring tools are tied to it. Lucas: Right. So, looking ahead, the question for engineering leaders is whether they invest in this foundational layer now or pay for it later in outages and reputation damage. The trend is clearly toward deeper integration of ai specific observability into the core CI/CD pipelines. Luna: We will likely see this become a standard requirement for any serious AI deployment within the year. Lucas: I agree. It is no longer optional. If you are running AI in production, you need to see inside the box. Otherwise, you are just flying blind. Luna: And that visibility is what separates a prototype from a product. Lucas: Well said. Before we dive back into the specifics of implementation strategies, I want to pause for a moment about how we keep this content going. We deliberately do not run ads on these shows. If you find the analysis useful for what you are building or running, and you want to support that choice, the link is buy me a coffee dot com slash fexingo. Luna: It really helps keep the research deep and the editing tight without needing corporate sponsors dictating the topics. Lucas: Thank you to everyone who supports us directly. It allows us to stay independent. Now, let's get back to the technical details of how to actually implement these trace-based systems in your own codebase. Luna: Let's start with the practical side. What is the first step a team should take when adopting this kind of granular monitoring? Lucas: The first step is instrumentation hygiene. Many teams have messy logs that mix user data with internal debug info. Before you add AI tracing, you need to ensure your existing application traces are clean and correlated. You cannot build a reliable semantic drift detector on top of noisy, unstructured logs. Start by standardizing your span naming conventions across your microservices. Luna: So structure first, then intelligence. That seems logical. Lucas: Correct. Once your base layers are solid, you can wrap your LLM calls with dedicated spans. Tag them with the model version, the prompt template ID, and the user segment. This metadata is essential for filtering noise later. You want to be able to ask, 'Show me all requests to Model X that resulted in low confidence scores last Tuesday.' Luna: Filtering by confidence score is smart. It lets you focus on the edge cases rather than the happy path. Lucas: Exactly. The happy path is easy. The edge cases are where your revenue leaks happen. By tagging those requests, you can run targeted evaluations. You don't need to evaluate every single request in real-time, which would be too costly. You can sample the low-confidence ones for deep analysis. Luna: Sampling reduces the computational overhead significantly while still catching the problems. Lucas: It does. And it gives you a representative dataset for improving your prompts. Over time, you will see patterns in the failures. Maybe certain keywords trigger hallucinations. Maybe certain user segments require simpler explanations. That feedback loop closes the gap between development and production. Luna: It turns operations into R&D. That is a powerful way to think about it. Lucas: It is. And as we look at the coming months, expect more tools to emerge that automate this sampling and evaluation process. The manual work of writing evaluators will shrink, but the strategic decision of what to measure will remain critical. Luna: So the tooling will get easier, but the judgment call stays with the leader. Lucas: Precisely. The technology handles the scale; the human handles the nuance. That partnership is the future of stable AI systems. Luna: I think that is a great place to leave it for now. It really highlights how much the role of the engineer is evolving. Lucas: It is. And staying curious about these shifts is the best way to prepare for them. Thanks for listening.