Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Monitors Its Own 100-Terabyte Infrastructure
Transcript
- Lucas: If these conversations are useful for what you're building or running, you're exactly who we make them for. Today: how Datadog monitors its own infrastructure — a system that ingests over 100 terabytes of data every single day. Luna: And they eat their own dog food — literally every feature they ship for customers, they run internally first. Lucas: Right. But the twist is that dogfooding at that scale forces some pretty wild architectural decisions. I spoke with a senior engineer on their observability team — not for attribution, but the details are public in their engineering blog and conference talks. Luna: What's the biggest single-number stat that surprised you? Lucas: They run their entire backend on a single PostgreSQL fork. Not a cluster of microservices each with its own database — one very large, heavily modified Postgres instance that handles all their metadata and configuration. Luna: Wait — for all of Datadog? That sounds like a single point of failure. Lucas: That's exactly what I asked. The answer is that they've sharded it internally using a custom layer they built called 'DDShard' — it's not open-sourced. Each shard is a separate Postgres instance, but the application sees it as one logical database. Luna: So it's a single logical database, but physically distributed. That's basically what CockroachDB or Spanner do, but homegrown. Lucas: Exactly — and they chose that path because when they started scaling around 2015, the off-the-shelf distributed databases weren't mature enough for their latency requirements. They needed sub-millisecond reads for their dashboard queries. Luna: That makes sense. But what about the time-series data itself? The actual metrics — CPU, memory, custom business metrics — that's the 100 terabytes a day, right? Lucas: Correct. And that's where they made a really interesting call. They don't store time-series data in Postgres at all. They built their own columnar time-series database internally, which eventually became the product that customers use — but internally it runs on the same codebase. Luna: So they eat their own dog food for the storage engine too. That's a huge R&D bet — if the internal system breaks, all their monitoring goes dark. Lucas: And it has happened. There's a famous incident from 2020 where a bad deployment of their time-series database caused a cascading failure that took down their dashboards for about 90 minutes. The engineering postmortem is a great read. Luna: What was the root cause? Lucas: A change to the compaction logic — the process that merges smaller data files into larger ones — introduced a deadlock under high write load. And because they were dogfooding, the failure affected both their internal operations and their customer-facing product simultaneously. Luna: That seems like the downside of dogfooding. You don't have a separate 'canary' environment that's truly isolated. Lucas: They do have staging environments, but the production dogfooding means that any bug in a new feature immediately hits their own engineers. That creates a really strong incentive to get it right — but also means you can burn your own house down. Luna: I want to come back to the cardinality problem. Datadog charges based on custom metrics and cardinality — how many unique combinations of tags you send. Do they internally face the same cost pressures? Lucas: Absolutely. Their own engineering teams generate enormous cardinality — think about a Kubernetes cluster with thousands of pods, each with dozens of labels. They had to build a system called 'DDSketch' to handle high-cardinality metrics efficiently. Luna: DDSketch is open source, right? The quantile sketch algorithm? Lucas: Yes — they open-sourced it in 2020. It's a data structure that can calculate percentiles with very high accuracy while using dramatically less memory than naive approaches. Internally, they use it to aggregate metrics from all their hosts before sending them to the central time-series database. Luna: So they're essentially compressing the data at the edge. That's a common pattern in observability — but I'm curious how they decide what to keep and what to drop. Lucas: They have a tiered retention policy. Raw metrics at full resolution are kept for 15 months — that's the standard. But for really high-cardinality custom metrics, they downsample aggressively after a week. The engineering team told me that they've learned that most debugging happens within the first 48 hours, and after that, aggregate trends matter more. Luna: That aligns with what I've seen at other companies. But what about their own SLOs? How do they monitor the monitor? Lucas: They have a separate, minimal monitoring stack that runs on a different cloud provider — a kind of 'canary in the coal mine' that checks if Datadog is reporting data correctly. It's essentially a small Prometheus instance with a Grafana dashboard that only tracks three metrics: data ingestion rate, query latency, and error rate. Luna: That's smart. So if Datadog goes down, they don't lose all visibility. Lucas: Right. And they also run regular game-day exercises where they deliberately break parts of the system to see if the failover works. I think that level of rigor is rare even among infrastructure companies. Luna: Before we dig deeper into the engineering org — quick honest thing. A handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these. No ads, no sponsors — just people who find the show useful. Lucas: Yeah, and it keeps us independent. We don't have to tailor the content to please a vendor or a PR team. We can say 'this is where Datadog's approach falls short' without worrying about an ad partnership. Luna: Exactly. And back to that point — what is a genuine limitation of their dogfood-first approach? Lucas: The main one is that they sometimes build features that are perfect for Datadog's own engineering culture but miss obvious use cases for smaller teams. For example, their alerting system was initially designed for teams that have dedicated on-call rotations with multiple escalation paths. A startup with three engineers found it overwhelming. Luna: Did they eventually simplify it? Lucas: They did — they introduced a simplified alerting workflow in 2023 after customer feedback. But it took years because internally they didn't feel the pain. That's the classic dogfooding blind spot: your own needs are not universal. Luna: So how does Datadog's engineering org handle that tension? I imagine the product teams are separate from the internal platform teams. Lucas: They have a central 'Platform Engineering' group that maintains the core time-series database and the Postgres sharding layer. Then each product area — like APM, logs, infrastructure monitoring — has its own team that builds on top of that platform. The dogfooding happens at the platform level: the APM team uses the same database as the customer APM product. Luna: So the platform team eats their own dog food, but the product teams might not. That's a subtle distinction. Lucas: Yes — and it matters because the platform team's incentives are aligned with reliability, while product teams care more about feature velocity. There's a healthy tension there. The platform team publishes internal SLOs, and if a product team's change causes a regression, it's a big deal. Luna: How big is the platform team? Lucas: About 60 engineers out of 4,000 total. So it's a small, high-leverage group. They manage the infrastructure that ingests 100 terabytes a day, stores it, and serves queries with a median latency under 10 milliseconds. Luna: That is genuinely impressive. I want to go back to one thing you mentioned earlier — the single Postgres fork. How much have they modified it? Lucas: They've added custom extensions for their sharding layer, a background worker for data redistribution, and they've patched the query planner to optimize for their specific workload patterns. They maintain their own fork with about 20,000 lines of C code on top of the original Postgres codebase. Luna: That's a massive maintenance burden. Every time Postgres releases a new version, they have to merge their changes. Lucas: They're on version 13 — deliberately a few versions behind stable. The engineer I spoke with said they evaluate each major upgrade, and so far the cost of migrating their custom patches hasn't been worth the benefit. They backport security fixes manually. Luna: So they're running an unsupported, heavily modified database. That's a bet on internal expertise over vendor reliability. Lucas: It is, but it's a bet that's paid off for them. Their query latency is consistently under 2 milliseconds for 99th percentile metadata lookups. I don't know of any off-the-shelf distributed database that could match that for their specific workload. Luna: And that's the key lesson for CTOs listening: sometimes the best architecture is the one you build yourself — but only if you have the talent and the willingness to maintain it for years. Lucas: Exactly. Datadog's approach isn't a template for everyone. But the principles — dogfood early, monitor your monitor, and be honest about the trade-offs — those apply at any scale. Luna: Great episode. Next time we'll look at how a completely different company — one that chose a fully managed cloud path — handled a similar scaling challenge.