Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Built Observability at Massive Scale
Transcript
- Lucas: So, Datadog. They process on the order of a trillion events a day, and their entire business is built on making that data useful in real time. But what I find really interesting is how they got there — it wasn't one big bet, it was a series of architectural decisions that compounded. Luna: A trillion events a day — that's a number that breaks my brain. How do you even start to architect for that? Lucas: Right, and the honest answer is they didn't start there. When Datadog launched in 2010, it was just a SaaS dashboard for server metrics. The early architecture was pretty conventional — a relational database, a polling agent, and a web front end. But as they added more customers, the volume of time-series data exploded, and they had to move to a custom time-series database pretty quickly. Luna: So they built their own database? That's a big leap for a startup. Lucas: It was, but they had no choice. The off-the-shelf options at the time — like Graphite or RRDtool — couldn't handle the cardinality and the write throughput. So they built something internally, which eventually became what they call the Metrics Database. That was the first big architectural bet. And it paid off, because it gave them the ability to handle high-cardinality tags, which became a huge differentiator. Luna: High cardinality — for listeners, that means you can slice and dice metrics by any tag, like host, service, or even user ID. That's what makes modern monitoring so powerful. Lucas: Exactly. And that's the foundation. But the next big shift came when they realized metrics alone weren't enough. Customers wanted logs and traces too. So they had to bring those together on a unified platform, which meant building — or acquiring — new data stores for each type. Luna: And that's when they started acquiring companies, right? Logz.io or something? Lucas: They did acquire a few, but the key move was building their own log management and APM — application performance monitoring — from scratch, rather than bolting on open-source tools. That was a deliberate decision to keep everything integrated. And that integration is the real story, because it's what lets Datadog correlate metrics, logs, and traces in a single query. Luna: So the architecture became a sea of different data stores, but with a common query layer on top. That sounds like a massive engineering challenge. Lucas: It is. And Datadog's CTO, Alexis Lê-Quôc, has talked about how they standardize on a few core primitives — like a time-series engine, a log store, and a search index — and then build features on top of those. That's what lets them ship new products quickly. They don't rewrite the plumbing every time. Luna: So they've essentially built their own internal data platform, and that's the moat. Anyone can build a dashboard, but handling that scale with that kind of flexibility is hard. Lucas: Right. And that's why they're still growing. But scaling the platform is only half the story. The other half is scaling the engineering org to support it. Datadog went from a few hundred engineers to over five thousand in a few years. That kind of growth is brutal if you don't have a clear team topology. Luna: What did they do differently? I feel like most companies just pile on more people and hope for the best. Lucas: So they've talked about adopting a 'two-pizza team' model, where teams are small, cross-functional, and own a specific service or product area end to end. That's not unique, but they were disciplined about it. They also invested heavily in internal developer tooling and documentation, so that new engineers could become productive quickly. Luna: That sounds like the kind of stuff that doesn't show up on a balance sheet but is actually the hardest part of scaling. You can't just buy your way to good internal tooling. Lucas: Definitely not. And there's another cultural piece: they have a strong 'output not input' philosophy. They measure success by customer impact, not by lines of code or number of features shipped. That keeps teams focused. Luna: But even with good culture, you have to make hard architectural trade-offs. Like, how do you balance real-time ingestion with long-term storage? That's a classic tension. Lucas: And that's where their tiered storage strategy comes in. They use hot storage for recent data, which is fast and expensive, and then move data to cheaper, colder storage as it ages. Queries automatically span both tiers, so users don't have to think about it. That's a huge engineering effort, but it lets them offer unlimited retention at a reasonable cost. Luna: So the user sees one unified view, but under the hood it's a multi-tiered beast. That's elegant. Lucas: It is. And now they're layering AI on top of that. Datadog has been rolling out anomaly detection and predictive alerts that use machine learning to spot issues before they happen. That's a natural next step, but it also creates new challenges — like how do you train models on that much data without blowing up your compute costs? Luna: And that's where I imagine the cost optimization story comes in. You can't just throw GPUs at everything. Lucas: Right, and they've been smart about using smaller, purpose-built models rather than one giant LLM for everything. They also use a lot of pre-computation and caching to reduce inference load. So the AI features are targeted — like detecting a spike in error rates or forecasting capacity needs — rather than trying to do everything at once. Luna: That's a good lesson for any CTO: don't use a sledgehammer when a scalpel will do. And it's also a lesson about staying focused on your core value proposition. Lucas: Exactly. And I think that's why Datadog has been able to fend off competitors like Grafana Labs and New Relic. They've built a platform that's not just a tool, but a system that engineers rely on daily. And that's the kind of product that people defend. Luna: You know, this conversation reminds me why we do this show ad-free. We want to bring you these deep dives without the noise of sponsor interruptions. If you find value in that, a small contribution at buy me a coffee dot com slash fexingo goes a long way to keep it going. No pressure, but we appreciate the support. Lucas: Yeah, it's a deliberate choice, and we're grateful for the listeners who make it possible. But back to Datadog — one question I've been thinking about is whether their architecture is a model for other companies or if it's too unique to replicate. Luna: What's your take? Can a smaller company adopt the same principles without the scale? Lucas: I think you can adopt the principles, but not the specifics. For most companies, buying a commercial observability tool is the right call. But the idea of having a unified data platform, with common primitives and a consistent query experience, is something you can apply even at a smaller scale. It's about avoiding silos, even if you're using managed services. Luna: So it's less about building your own database and more about designing your architecture for integration and flexibility. That's a more accessible lesson. Lucas: And also the cultural piece. The 'output not input' philosophy, the two-pizza teams, the investment in internal tooling — those are things any CTO can implement, regardless of company size. That might be the real takeaway from Datadog's journey. Luna: Well, and I think the other takeaway is that observability is not just a nice to have — it's a business enabler. Datadog's own success proves that companies will pay for tools that help them move faster and sleep better at night. Lucas: Yeah, and that's why we'll keep seeing more players in this space, and more innovation. The question is, who will be the next Datadog? Or will Datadog continue to dominate? I think they're well-positioned, but the tech landscape is never static. Luna: Well, and for listeners, if you're thinking about your own observability strategy, the key is to think about it as a platform, not a collection of tools. That's the Datadog lesson in a nutshell. Lucas: And with that, I think we've covered a lot. Next time, we might dive into the nitty-gritty of how they actually implement their tiered storage, but for now, thanks for joining us.