Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Monitors Its Own Infrastructure
Transcript
- Lucas: So there's a classic tech saying — 'who watches the watchers?' And for a company like Datadog, that's not just a philosophical question. It's a daily engineering reality. Luna: Right — they build the monitoring platform that thousands of companies rely on. But they also have to run that platform for themselves. Lucas: Exactly. And today we're going to look at how Datadog monitors its own infrastructure. I mean, they process something like 30 trillion time-series data points per day across their customers — but they also ingest all of their own metrics, traces, and logs. It's a fascinating dogfooding exercise. Luna: And it's not just about eating their own dog food — it's about scaling that dog food to handle their own massive environment. Do we know how big that environment is? Lucas: According to their engineering blog and some talks from their SRE team, they run about 8,000 hosts across multiple cloud providers and their own bare-metal infrastructure. That's a lot of signals. And they target 99.99 percent uptime for their own platform — same as they promise customers. Luna: So they're essentially running a mini version of their entire customer-facing product internally, but with the added twist that if their monitoring breaks, they can't fix it fast without monitoring. It's a bootstrap problem. Lucas: It really is. They have to maintain a 'monitor of monitors' — a separate lightweight system that watches the main Datadog instance. Typically that's a smaller deployment running on different infrastructure, so if the primary goes down, they still have visibility. Luna: That's smart. But I imagine the main challenge isn't just redundancy — it's the sheer volume of data they generate internally. How do they avoid drowning in their own alerts? Lucas: That's a huge part of the story. Alert fatigue is real, and at Datadog's scale, false positives can actually cause engineers to ignore real issues. So they've built a sophisticated alert correlation and deduplication system. For example, if a host goes down, they don't want 50 alerts for each service running on that host — they want one alert that says 'host X is down' and the rest are suppressed. Luna: I remember reading about their 'alert aggregator' service. It basically groups incoming alerts by root cause using a dynamic windowing algorithm. If multiple services start failing within a short time window, it assumes a common dependency and rolls them up. Lucas: Right. And that's on top of their existing monitoring products. Internally, they use Datadog's own APM, log management, and infrastructure monitoring — all dogfooded. But they've also built custom dashboards for their SRE team that aggregate across all these data sources. Luna: It's like they're running a meta-Datadog. But let's talk about the time-series database. They built a custom one, right? For storing metrics. Lucas: Yes. They have a proprietary time-series database that they originally built to handle their customers' data, and they use it internally too. It's designed for high cardinality — meaning you can attach many unique tags to each metric — and it can ingest millions of data points per second. Internally, they're pushing about 30 trillion time-series points daily, and that's just their own metrics. Luna: Thirty trillion. That number is hard to get your head around. And they need to query that data in real-time for dashboards and alerts. Lucas: Exactly. The query latency requirements are sub-second for most dashboards. So they use a tiered storage approach: recent data is kept in memory, older data is compressed and stored on SSDs, and very old data is archived to object storage. But all of it is queryable through a unified API. Luna: Before we go deeper — I want to mention something that makes this show possible. If these kinds of deep-dive conversations are useful for what you're building or running, a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo. That's literally what funds making this many of these episodes. Lucas: Yeah, it's real. We don't run ads, and the listener support keeps it going. So if you find value in it, that's the way to help. Luna: Alright, back to Datadog. So they have this custom time-series DB — how does it handle the cardinality issue? Because modern microservices generate a ton of unique metric tags. Lucas: Great question. The database uses a technique called 'time-series compression' that exploits the fact that many metrics share the same tags. Instead of storing the full tag set with every data point, it stores a mapping of tag combinations to a unique ID, and then the data points only reference that ID. That reduces storage by something like 10x. Luna: And that's crucial for cost. They also have a tiered retention policy — fine-grained data for the first 30 days, then aggregated rollups for longer periods. Lucas: Right. They also use what they call 'monitor as code' — internal teams define their alerts and dashboards in YAML files that are version-controlled and deployed through CI/CD pipelines. That way, changes are reviewed and tested before they hit production. Luna: That's a practice that's becoming standard in the industry, but it's interesting to see how Dogfooding enforces it internally. If your dashboards break, you feel it immediately. Lucas: Exactly. And because they're a monitoring company, any internal outage is a PR nightmare. So they have a rigorous incident management process. They use a version of the 'Google SRE' model — incidents have a commander, communication lead, and subject matter experts. Postmortems are blameless and mandatory. Luna: And they also run regular 'game days' where they simulate failures — like killing a database server or dropping network packets — to see how their monitoring and incident response holds up. Lucas: Yes. That's a great practice that many companies should adopt. But one of the most interesting aspects to me is the trace-agent. Datadog's distributed tracing product is a big differentiator, and they use it internally to map service dependencies and find latency hotspots. Luna: So they run the trace-agent on every host, and it sends traces to the backend. But at their scale, even tracing themselves generates huge volumes of data. Lucas: They have a sampling strategy. They capture 100 percent of traces for error spans, but for healthy requests, they sample adaptively — adjusting the sampling rate based on the overall request rate. That way, they keep the most informative data without overwhelming storage. Luna: And they also use trace analytics to drive capacity planning. For example, if a new deployment increases latency for a particular endpoint, they can spot it in the trace data and roll back quickly. Lucas: Exactly. So the takeaway here is that Datadog's internal monitoring isn't just a copy of what they sell. It's a hardened, battle-tested version that has to survive their own scale. They've had to build custom solutions for alert deduplication, time-series compression, and sampling that then often make it into their product. Luna: So the dogfooding loop is actually a product development strategy. They solve their own hard problems, and then those solutions become features for customers. Lucas: That's the beauty of it. And it's something every company building developer tools should think about — if you don't use your own product in production, you're missing the biggest source of real-world feedback. Luna: Final thought — what's the one thing you'd take away from this for an engineering leader? Lucas: I'd say: invest in a 'monitor of monitors' early. Even if you're not at Datadog scale, having a lightweight, independent watchdog can save you when your main observability stack goes down. And second: embrace dogfooding not as a checkbox, but as a core part of your engineering culture.