Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Splunk Rebuilt Its Observability Platform for 10 Petabytes Daily
Transcript
- Lucas: Splunk ingests about ten petabytes of new data every single day. That's roughly the equivalent of the entire written works of humanity coming in every forty-eight hours. And until recently, it was doing that on an architecture designed when the internet was a fraction of that size. Luna: So they had to rebuild the engine while the plane was flying. I imagine you can't just take the system down and rewrite it from scratch when you have thousands of enterprise customers running security operations on your platform. Lucas: Exactly. And that's really the fascinating part of this story. Splunk's CTO, who came on board in 2022, made a bet: instead of trying to optimise the old monolithic indexing layer, they would re-architect the entire ingest pipeline around streaming, using Apache Kafka as the backbone. But they had to do it without ever breaking the existing search experience for customers. Luna: Right, because for a lot of those customers, Splunk is the central nervous system of their security operations centre. If search goes down, they're flying blind. Lucas: Precisely. So the team started by identifying the biggest bottleneck: the indexing layer. In the old system, when data came in, it had to be indexed immediately before it could be searched. That created a tight coupling between ingest and search. If ingest spiked, search slowed down. And with ten petabytes a day, that spike was constant. Luna: It reminds me of something we've talked about before on this show — the idea that the hardest part of scaling isn't the volume, it's the coupling between components. Lucas: That's exactly the insight. So they decoupled ingest from search by introducing a streaming layer. Data now lands in Kafka topics, and then it's processed independently for indexing and for real-time search. That means a surge in ingest doesn't directly impact query performance. Luna: But Kafka itself has challenges at that scale. Ten petabytes a day means something like a hundred gigabytes per second sustained. Did they have to customise Kafka? Lucas: They did. They worked closely with the Kafka community and made some significant modifications. One of the key changes was around partitioning. Standard Kafka has a practical limit on the number of partitions per broker. Splunk needed hundreds of thousands of partitions to handle the data diversity from different customers and data sources. So they re-engineered the partition management layer to be more dynamic. Luna: Hundreds of thousands of partitions. That's not off-the-shelf Kafka anymore. Lucas: No, it's a heavily forked version. But the bet was that Kafka's core strengths — durability, replayability, ordering guarantees — were worth building on. And it allowed them to introduce a concept called tiered storage. In the old system, all data lived on expensive SSDs because it had to be searchable instantly. In the new system, hot data stays on SSDs for the first few hours, then gets moved to cheaper object storage like S3, with intelligent caching that brings frequently accessed older data back to fast storage on demand. Luna: That must have changed the economics significantly. I recall that storage was one of the biggest cost drivers for Splunk customers. Lucas: It was. And this is where the engineering team had to make a really tough trade-off. Because moving data to object storage means search latency for older data goes up. It's not instant anymore. So they had to be very transparent with customers: 'For data older than thirty days, expect search results in seconds, not milliseconds.' Most enterprise customers accepted that because the cost savings were dramatic. Some didn't, and Splunk maintained a premium tier that keeps everything on fast storage. Luna: That's a classic product segmentation move. But it also required changes to the query engine itself, right? Because the old query language assumed all data was local and fast. Lucas: Right. And here's where the story gets really interesting. The CTO decided that instead of patching the old query language — which was called SPL, Search Processing Language — they would build a new one, designed from the ground up for a distributed, tiered storage world. The new language is called SPL2. It's backwards compatible with SPL, but it adds constructs for explicitly referencing storage tiers, for streaming results, and for handling incomplete data sets. Luna: Backwards compatible is key. Because if you have years of saved searches and dashboards, you can't just tell customers to rewrite them all. Lucas: Exactly. So they built a compatibility layer that translates old SPL queries into SPL2 at runtime. It's not perfect — some edge cases behave differently — but they ran millions of customer queries through it during the beta and achieved over ninety-nine percent compatibility. Luna: That's impressive. But it also means the engineering team had to maintain two query engines in parallel for a while. That's a huge overhead. Lucas: It is. And they did it for about eighteen months. The old engine kept running for customers who hadn't migrated, while the new engine powered the streaming ingest path and new features. The migration strategy was gradual: new customers only saw the new engine, existing customers could opt in, and eventually the old engine was turned off for everyone. Luna: I imagine the biggest challenge wasn't technical, but cultural. Getting the whole engineering org to buy into a multi-year rebuild while still shipping features. Lucas: That's where the CTO's leadership really mattered. He framed it not as a rewrite, but as a 'platform evolution'. Every quarter, teams had to deliver something that improved the customer experience, not just the architecture. For example, the first deliverable was a new real-time alerting feature that used the streaming pipeline — customers saw immediate value. That built trust. Luna: Smart. And that's a lesson that applies beyond Splunk: don't ask teams to invest in infrastructure that doesn't show benefits for years. Find the quick wins that also move you toward the long-term goal. Lucas: If these kinds of deep dives into engineering strategy are useful for what you're building or running, it's worth mentioning that this show is entirely listener-supported. No ads, no sponsors. The reason we can keep doing episodes like this — with the time and research they require — is because a growing number of listeners chip in at buy me a coffee dot com slash fexingo. That's it. Just a coffee's worth to help cover hosting, research, and editing. And it makes a real difference. Luna: Yeah, it genuinely does. And we really appreciate those who have already contributed. It keeps the show focused on the technical details, not on selling things. Lucas: So back to Splunk. Once the streaming pipeline was live and the new query engine was in production, the team turned their attention to the search experience itself. One of the most requested features was the ability to run historical searches across months of data without waiting forever. The old system would essentially scan all the data sequentially. The new system uses a combination of pre-computed summaries and a distributed query planner that splits work across multiple nodes. Luna: That sounds like a query planner problem. How do you decide which parts of a query can run in parallel and which need sequential processing? Lucas: Exactly. They built a cost-based optimizer that evaluates the query structure, the data distribution, and the storage tier of each shard. For example, if you're searching for a specific error code across the last six months, the planner knows that data older than thirty days is on object storage and has higher latency. So it might prioritise fetching recent data first and return partial results quickly, while background workers complete the older scans. Luna: That's similar to what Google does with BigQuery, but applied to machine data. It's fascinating how these ideas cross-pollinate. Lucas: And Splunk's team was very deliberate about that. They studied how other hyperscalers handled similar problems. But they also had to deal with the unique nature of machine data: it's incredibly diverse, with different schemas, timestamps, and formats. A log from a web server looks nothing like a log from a network router. Luna: So the schema-on-read approach that Splunk pioneered becomes even more important at scale. You can't force everything into a predefined schema upfront. Lucas: Exactly. And the new architecture actually improved schema-on-read. Because the streaming layer can apply transformations on the fly, the system can extract fields and create indexes dynamically based on query patterns. If a particular field is frequently searched, the system automatically creates an index for it. If it's rarely used, it stays unindexed, saving storage. Luna: That's adaptive indexing. I've seen that in some modern databases, but not in an observability platform at this scale. Lucas: It's one of the innovations that came out of this rebuild. And it's a good example of how rethinking the architecture from first principles can lead to features that weren't possible before. The CTO told me that if they had tried to bolt adaptive indexing onto the old system, it would have been a nightmare of spaghetti code. The clean break with the streaming pipeline made it natural. Luna: So what's next for Splunk's platform? They've rebuilt the ingest layer, the query engine, the storage tiers. Is there anything left? Lucas: Actually, the next frontier is the machine learning layer. Splunk has a product called Machine Learning Toolkit, but it's still largely batch-oriented. The vision is to bring ML inference into the streaming pipeline, so that anomaly detection and predictive alerts happen in real time, as data flows through Kafka. That's a significant engineering challenge because it requires low-latency model serving at massive throughput. Luna: That would be a game-changer for security operations. Imagine detecting a zero-day exploit not after the fact, but as it's happening, based on deviations from learned patterns. Lucas: Exactly. And it's the logical next step. But for now, the team is still consolidating the streaming architecture. They migrated the last of their internal data centers to the new pipeline just last quarter. So the rebuild is essentially done, and now it's about optimisation and new capabilities. Luna: It's a great case study in how to modernise a platform without losing your customer base. The key lessons: decouple tightly coupled systems, use streaming as a buffer, invest in backward compatibility, and find quick wins to fund the long-term vision. Lucas: And never underestimate the cultural challenge. The Splunk CTO spent as much time on internal communication as on architecture. He held weekly engineering town halls, wrote detailed RFCs, and made sure every team understood how their work fit into the bigger picture. That's probably the most underrated part of this whole story. Luna: Agreed. The technology is impressive, but the leadership is what made it possible.