Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Rebuilt Its Observability Platform for 100 Petabytes Daily
Transcript
- Lucas: So Datadog is ingesting over a hundred petabytes of telemetry data every single day. And if you've ever used their dashboards, you know the queries are supposed to come back in seconds. But the architecture that got them to, say, ten petabytes a day was not going to survive a hundred. Luna: Right — because a tenfold increase in data volume doesn't just mean ten times the storage cost. It changes the query patterns, the indexing, the whole retrieval path. Lucas: Exactly. The original system was built on a custom time-series database — let's call it TSDB v1 — that stored metrics in a row-oriented format. It worked beautifully when cardinality — the number of unique metric name-tag combinations — was in the millions. But by 2022, customers were sending metrics with dozens of tags, and cardinality was exploding into the billions. Luna: So the old index was basically O scan on every query? That sounds painful. Lucas: Worse. The B-tree index that powered tag lookups started fragmenting under concurrent writes. Query latency went from sub-second to three, four seconds for moderately complex dashboards. And because the write path was synchronous to that same index, ingestion throughput also plateaued. Luna: So they had a dual problem: queries were slowing down, and they couldn't scale writes without degrading reads even further. Lucas: Right. The engineering team, led by their VP of Platform, made a call in early 2023: rebuild the storage layer from scratch. But with a hard constraint — existing customer dashboards and alerts must continue to work without any code changes on the customer side. Luna: That's the toughest kind of migration. You can't break the API contract, so you have to build a compatibility layer that mimics the old query semantics while the new engine runs underneath. Lucas: They called the new engine the Columnar Metrics Store. Instead of row-oriented storage, they switched to a columnar format inspired by Apache Parquet, but custom-built for time-series. Each metric's time series became a column group, compressed with delta of delta encoding for timestamps and xor based compression for floating-point values. Luna: That's the Facebook Gorilla paper approach, right? The delta of delta compression for timestamps and XOR for values? I remember reading that it can compress a typical time series to under a byte per data point. Lucas: That's exactly the inspiration. Datadog took that and added a few twists. They partitioned data by time into chunks — say, two-hour windows — and within each chunk, they built a bloom filter per tag value. So when a query comes in filtering on a specific tag, the system can skip entire chunks that don't contain that tag, without scanning the index. Luna: Bloom filters are probabilistic, though. There's a false-positive rate. How did they tune that to avoid excessive disk reads? Lucas: They tuned the filter size to keep false positives under one percent for the most common tag cardinalities. And they layered an LRU cache on top — the hot chunks, which are the most recent two hours, stay in memory anyway. So the bloom filter is mostly useful for historical data that lives on SSD or object storage. Luna: Speaking of storage tiers — they moved to a three-tier model, right? Hot, warm, cold? Lucas: Yes. Hot tier: data less than four hours old, stored on local NVMe SSDs, replicated across three availability zones. Warm tier: data up to thirty days old, stored on provisioned SSD volumes in a separate cluster. Cold tier: anything older than thirty days, stored in S3 with the columnar format, but still queryable. The trick is that the query engine transparently routes to the right tier based on the time range of the query. Luna: And the cold tier queries — do they actually run in seconds, or is there a noticeable delay? Lucas: Datadog published benchmarks showing that p99 query latency for cold data is under three seconds for dashboards spanning thirty days. That's impressive because S3 has much higher latency than local SSDs. They achieved it by pre-computing min, max, and count aggregates for each chunk at write time, so many queries can be satisfied from those pre-computed values without scanning the raw data. Luna: So they essentially built a small aggregate layer on top of the columnar store. That's clever — it means the most common dashboard queries — 'what was the average CPU usage over the last week' — hit the pre-computed stats, not the raw series. Lucas: Exactly. The raw data is only decompressed when the user drills into a specific time window or applies a complex aggregation that can't be served from the pre-computed stats. And that's relatively rare. Luna: I want to talk about the human side of this. Three years of rebuilding a core storage engine while keeping the old system running — that must have been a huge organizational challenge. Lucas: It was. The team split into two tracks. One track, called 'legacy stability', owned the existing TSDB v1 and kept it running, fixing bugs, handling capacity. The other track, 'next-gen storage', built the new engine. They had a weekly sync where the next-gen team demoed progress and the legacy team flagged any customer issues that might be related to the new system. Luna: Did they do a big-bang migration, or gradually move customers over? Lucas: Gradually. They first enabled the new engine for a small percentage of new data — say, one percent of accounts — and compared query results against the old system for weeks. Then they ramped to ten percent, then fifty. But they always kept the old system as a fallback. If a query against the new engine timed out or returned an error, it would automatically retry against the old TSDB. Luna: That's a lot of infrastructure to run simultaneously. Double the storage cost, double the compute for queries. Lucas: It was expensive, but it gave them a safety net. And they used that dual-run period to iterate on the new engine's performance. For example, they discovered that certain high-cardinality queries — say, a tag with ten million unique values — caused the bloom filter to be too large to fit in memory, so they had to switch to a different filter variant: a blocked bloom filter that compresses better. Luna: And this kind of real-world tuning is exactly why you need that dual-run period. You can simulate all you want in staging, but production traffic has a way of finding edge cases. Lucas: Right. By mid-2025, they had migrated all active customers to the new engine. The old TSDB v1 was finally decommissioned in December 2025. And the result? Query latency for the p99 dropped from over three seconds to under five hundred milliseconds. And ingestion throughput scaled linearly: they can now handle over a hundred petabytes a day with room to grow. Luna: That's a massive improvement. But it also means they've set a new baseline for what customers expect. Now every future feature has to meet that sub-second bar. Lucas: Absolutely. And the team is already working on the next bottleneck: the query engine itself. Because now that storage is fast, the CPU cost of parsing and planning complex queries is becoming the new limit. They're experimenting with a just-in-time compilation approach for aggregation functions. Luna: So it never ends. You fix one bottleneck, and another appears. That's the nature of building at this scale. Lucas: It really is. And honestly, if this kind of deep-dive into engineering architecture is useful for what you're building or running, that's exactly why we do these episodes. We keep the show ad-free and listener-supported, so if today's episode was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Luna: Yeah, it genuinely helps us keep digging into stories like this. Every bit of support goes straight into research time and hosting costs. Lucas: Now, one thing I didn't mention about the Datadog migration is how they handled alerting during the cutover. Because alerts are even more sensitive to latency than dashboards. Luna: Oh, right — if an alert fires five seconds late, that could mean a missed incident window. Lucas: Exactly. So they built a separate alert evaluation pipeline that bypassed the new columnar store initially and stayed on the old TSDB. Only after the new engine had proven itself for three months on dashboard queries did they start routing alert evaluations through it. And even then, they kept a shadow evaluation running — the same alert evaluated against both systems — to compare results. Luna: That shadow evaluation probably caught a few discrepancies, I'm guessing. Lucas: It did. They found cases where the new engine's floating-point aggregation produced slightly different results due to different summation order. IEEE 754 rounding means that summing numbers in a different order can change the least significant bits. For most dashboards, that's invisible. But for alerts with very tight thresholds, it could cause false positives. Luna: So they had to standardize the summation order across both engines — likely by sorting the data points by timestamp before summing. Lucas: That's exactly what they did. They made both engines use a deterministic sort on before aggregation. It added a few milliseconds to query time, but it eliminated the discrepancy. And that's the kind of detail that only emerges when you run two systems side by side in production for months. Luna: It's also a good reminder that even with a well-designed new system, you can't skip the validation phase. Every edge case you find in production saves you from a bigger incident later. Lucas: Yeah. And Datadog published a detailed postmortem on their engineering blog earlier this year — I'll link it in the show notes. They were surprisingly transparent about the mistakes, too: like the time they accidentally deployed a config change that caused the new engine to fall back to the old one for all queries, doubling the load on the legacy system. Luna: That must have been a stressful day for the on-call team. But it's also the kind of failure that builds institutional knowledge. Next time, they'll have a canary deployment for config changes. Lucas: They do now. And that's the final takeaway: rebuilding a platform at this scale isn't just about the clever compression or the columnar format. It's about the operational maturity to manage the migration without breaking trust with your customers. Luna: Well said. And I think that's a good note to end on. Next episode, we're looking at how Snowflake rebuilt its query optimizer for semi-structured data. Lucas: Looking forward to it. See you next time.