Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Monitors Its Own Monolith at Scale
Transcript
- Lucas: So Datadog sells observability — monitoring, tracing, logs — to basically every tech company you've heard of. But here's the thing: Datadog itself runs on a single monolith for its core metrics pipeline. Luna: Wait. The company that literally tells everyone else to monitor their microservices... runs a monolith? Lucas: Exactly. And they're open about it. In fact, their CTO has said the decision to keep a monolith was deliberate. The argument is that for a system ingesting millions of metrics per second, a monolith reduces latency and avoids the complexity of distributed tracing for internal debugging. Luna: But doesn't that create a single point of failure? If that monolith goes down, everything stops. Lucas: It does — but they've built redundancy into the monolith itself. They run multiple instances across regions, and the key insight is that the monolith's state is mostly ephemeral. Metrics flow in, get processed, and are fanned out to storage. If one instance dies, another picks up. The real risk isn't the monolith itself — it's the database underneath. Luna: So how do they dogfood their own product? They're using Datadog to monitor Datadog? Lucas: Yes — but with a twist. They built an internal tool called Watchtower specifically for their own monolith. Watchtower aggregates health signals from every host running the metrics pipeline and compares them against a baseline of the last hour. If a host starts dropping events — say, below 1.2 million events per second — Watchtower pages the on-call engineer. Luna: 1.2 million events per second per host — that's a specific number. Is that their advertised throughput? Lucas: It's the internal threshold they've tuned over years. Their public benchmarks are higher because they include compression and batching. But for internal monitoring, they use that number because it's the point where customer-facing latency starts to degrade. Below that, they know they have headroom. Luna: That's fascinating — they're essentially using a known good threshold as a canary. Lucas: Right. And they don't just measure throughput. Watchtower also tracks error rates per metric type, memory usage per pipeline stage, and even the time it takes to flush a batch to their storage layer. The goal is to catch regressions before they impact customer dashboards. Luna: How big is the engineering team behind Watchtower? Lucas: Surprisingly small — about five engineers, plus part-time contributions from the SRE team. It's not a separate product; it's a set of scripts and dashboards that sit on top of their own monitoring stack. They treat it as a first-class internal customer. Luna: One quick thing — and this is totally unrelated to Datadog, but it ties into how we keep making episodes like this. A handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these. No ads, no sponsors — just listeners who find these conversations useful. Lucas: Yeah, that's right. It's a small group, but it makes a big difference. If these conversations are useful for what you're building or running, that's the only reason we keep going. Luna: So back to Datadog — you mentioned Watchtower catches regressions. Can you give an example of a regression they actually caught? Lucas: Sure. About a year ago, a change to the way they serialized metric tags accidentally introduced a lock contention in the serialization layer. The monolith didn't crash, but throughput dropped from 1.4 million events per second per host to about 900,000. Watchtower flagged it within two minutes because the baseline deviation crossed five percent. The on-call engineer rolled back the change in under ten minutes. Luna: So the monolith actually made debugging faster — because everything is in one process, you don't have to trace across services. Lucas: Exactly. That's the tradeoff the CTO emphasizes: a monolith is harder to deploy safely, but easier to debug when something goes wrong. Every function call is local. You don't need distributed traces to figure out where a bottleneck is. Luna: But what about scaling? Don't they eventually hit a wall where one instance can't handle the load? Lucas: They do scale horizontally — they run multiple instances of the monolith, each handling a shard of the metric space. The sharding is based on the metric name hash, so each instance owns a subset. When they need more capacity, they add more instances and rebalance the shards. It's not that different from how you'd scale a microservice, except the codebase is the same everywhere. Luna: So the monolith is really a 'distributed monolith' — multiple copies of the same binary. Lucas: That's the term they use internally. And they've been running this way for over five years. The CTO has said publicly that they've considered splitting the pipeline into microservices multiple times, but each time the analysis showed the latency cost of network hops would outweigh the benefits. Luna: What about deployment risk? A bad deploy to a monolith can take down everything. Lucas: They mitigate that with canary deployments. A new version goes to one shard first, monitored by Watchtower for the full baseline window — usually 15 minutes. If throughput stays above 1.2 million and error rates don't spike, they roll to the next shard. If anything looks off, they automatically roll back that shard. Luna: That sounds like a microservices deployment pattern applied to a monolith. Lucas: Exactly. They borrowed the best practices from microservices without breaking the codebase. The key enabler is that their internal tooling — Watchtower, their CI/CD pipeline, their feature flags — treats each shard as an independent unit. So they get the isolation of microservices with the simplicity of a monolith. Luna: I have to ask — do they ever regret not switching? Are there any pain points that microservices would solve? Lucas: The main pain point is developer onboarding. New engineers have to understand the entire metric pipeline because it's one codebase. With microservices, you could isolate a team to own just the ingestion layer. Datadog compensates with very detailed internal documentation and a week-long bootcamp that walks new hires through the monolith's architecture. Luna: So the tradeoff is: operational simplicity and debugging speed versus team autonomy and onboarding friction. Lucas: Right. And for Datadog, the math has favored the monolith so far. But they're not dogmatic about it. The CTO has said they continuously evaluate, and if the friction ever outweighs the benefits, they'll split. But as of today, their internal metrics show the monolith handles over 99.99 percent uptime for their own dashboards. Luna: One metric I'd love to see: how many engineers does it take to keep that monolith running? Lucas: The core pipeline team is about 12 engineers, including the Watchtower team. That's out of a total engineering org of roughly 1,800. So less than one percent of engineers own the entire metrics pipeline. That's remarkably lean for a company that processes over 100 trillion data points per day. Luna: That's the kind of efficiency that makes a CTO think twice before splitting. Lucas: Absolutely. And it's a great reminder that architecture decisions aren't about what's trendy — they're about what works for your specific constraints. Datadog's monolith works because they've invested in the tooling to make it safe and observable. Luna: So the next time someone says monoliths are dead, you can point to Datadog. Lucas: Exactly. It's not that monoliths are better — it's that the right architecture depends on your team, your scale, and your willingness to build the internal tools that make a monolith viable. Luna: And in Datadog's case, those tools are literally the same product they sell. Lucas: Which is the ultimate dogfooding story.