Latest / Cybersecurity Business with Fexingo: Security Companies, Breaches, and Enterprise Defense / How Enterprises Are Using Security Data Lakes
Transcript
- Lucas: So there's a quiet revolution happening in how enterprises store and analyze security data, and it's not a new detection tool or a new AI model. It's the data lake. Luna: Wait — are we talking about moving security logs out of the SIEM into something like Snowflake or Databricks? Lucas: Exactly. More and more security teams are realizing that traditional SIEMs — platforms like Splunk or the legacy on-prem guys — were built for a world where you ingested maybe a few terabytes a day. Now, with cloud sprawl and SaaS, a mid-size enterprise can generate 50 terabytes of security telemetry daily. Luna: And storing all that in a SIEM gets prohibitively expensive. I've heard of companies paying millions just to retain logs for compliance. Lucas: Right. So a growing number of CISOs are saying: why not store the raw data in a cheap object store — S3, Azure Blob, whatever — and use a query engine on top that's built for scale? That's the security data lake. And the economics are stark. Luna: Give me a number. Lucas: I talked to a global retailer — name off the record — that was spending $12 million a year on a SIEM license. They moved their security telemetry to a data lake built on Snowflake, and their total cost dropped to about $2.5 million. That's over 80 percent savings. Luna: And they kept all the data? No truncation? Lucas: They kept everything. Full-fidelity, raw logs going back two years — about 1.2 petabytes. Before, they could only retain 90 days of full logs and the rest was sampled. That's a game changer for threat hunting and forensic investigations. Luna: But a data lake is just storage and compute. What about the detection logic? The correlation rules? The alerting? Lucas: Great question. They didn't throw away the SIEM entirely. They kept a lightweight SIEM for real-time alerting — think a few hundred critical rules — but the heavy analytics, the historical searches, the compliance reporting, all moved to the data lake. Luna: So the SIEM becomes like a tripwire, and the data lake is the forensic lab. Lucas: Perfect analogy. And the net effect is that analysts can now query across months or years of data in seconds, instead of waiting hours for a search to time out. That changes how you approach threat hunting. Luna: I want to dig into a specific case. You mentioned a Fortune 500 financial services firm earlier? Lucas: Yeah, a top-ten bank — again, not naming. They had a classic Splunk deployment, ingesting about 30 terabytes a day. Their mean time to detect was 14 days. After migrating to a Databricks-based security data lake, they got it down to under 4 hours. Luna: That's a massive improvement. Was it purely the data architecture, or did they change processes too? Lucas: Both. But the architecture enabled the process change. With the legacy SIEM, they couldn't run broad, exploratory queries across historical data — it was too slow and too expensive. So analysts stopped trying. In the data lake, every analyst could write a SQL query against the entire dataset and get results in a minute. Luna: So you democratized data access. That's a cultural shift. Lucas: It is. And that's actually one of the bigger challenges I hear from practitioners. You need analysts who are comfortable with SQL — or with tools like PySpark. Not every security person comes from a data engineering background. Luna: But isn't that where the cloud providers are stepping in? I've seen AWS and Azure offer managed services that abstract away some of the complexity. Lucas: They are. AWS Security Lake launched a couple years ago, and Azure has something similar. They handle the ingestion and the schema normalization, so security teams don't have to build pipelines from scratch. But you still need someone to define the schema, manage partitions, optimize queries. Luna: It sounds like the role of the security data engineer is becoming critical. I've seen job postings for exactly that title — 'Security Data Engineer' — six-figure salaries, fully remote. Lucas: And that's a new role. Five years ago, that didn't exist. Now it's one of the hardest positions to fill because you need someone who understands both security operations and big data infrastructure. Luna: Right, and those two skill sets rarely overlap. Most good data engineers don't want to deal with threat intel and incident response. Lucas: Exactly. So some companies are splitting it: a data engineering team builds and maintains the lake, and the security team consumes it. That can work, but you need strong SLAs on data freshness and query performance. Luna: One concern I've heard is about query latency. If you're trying to do real-time detection, a data lake with object storage might not cut it. Lucas: That's fair. Object storage like S3 has high latency for small, random reads. So for real-time alerting, you still want a stream processing engine — like Kafka plus Flink — that triggers rules in milliseconds. The data lake is for the stuff that doesn't need sub-second response. Luna: So the architecture becomes a hybrid: streaming for real-time, batch for historical. Lucas: Exactly. And the smart teams are building that into their design from day one. They don't try to make the data lake do everything. Luna: What about compliance? Regulators want to know you're monitoring and retaining logs appropriately. Does a data lake make that easier or harder? Lucas: Easier in some ways, harder in others. Easier because you can keep everything — no more 'we only keep 90 days' arguments. Harder because you have to prove that the data hasn't been tampered with. That's where immutable storage and write once read many policies come in. Luna: And you need to be able to say: 'Yes, we have every DNS log from the past two years.' That's a powerful statement in a regulatory audit. Lucas: It is. And some firms are using the data lake to run continuous compliance checks — like, every hour, a query checks whether any user has been granted admin access without a ticket. That's proactive compliance, not reactive. Luna: Before we go further, Lucas — I want to mention something. We've been talking about a topic that's deeply technical and incredibly practical, and that's exactly the kind of conversation we want to keep bringing you. These episodes are ad-free because we believe in focusing on the substance, not the sponsor breaks. Lucas: Yeah, and that's a choice we've made from the start. If you find this valuable and want to support that independence, the simplest way is to buy us a coffee — literally. The link is buy me a coffee dot com slash fexingo. Luna: It's a small gesture that keeps this show going without any ads, ever. And we really appreciate everyone who's done that. Now, back to data lakes — one more thing about cloud cost. Lucas: Right, because the elephant in the room is that while data lakes are cheaper than SIEMs, they're not free. You still pay for S3 storage and compute time on Snowflake or Databricks. And if you write inefficient queries, costs can balloon. Luna: Would you say the total cost of ownership is still lower than a traditional SIEM? Lucas: For most organizations, yes — especially at scale. But you have to manage it actively. We've seen teams that set up a data lake, then six months later get a shock because their query costs are higher than expected. The solution is to partition your data smartly — by date, by log type — and use things like S3 Intelligent Tiering. Luna: So it's not a set-it-and-forget-it solution. It requires ongoing attention. Lucas: Exactly. But the payoff is huge. I think we're going to see a majority of large enterprises move to a security data lake model within the next three years. Luna: That aligns with what I'm hearing from CISOs. They're tired of being locked into proprietary log storage with egregious egress fees. Lucas: And that's the other big driver — vendor lock-in. With a data lake on open formats like Parquet, you can switch query engines, move to a different cloud, or even run on-prem. That flexibility is worth a lot to security leaders. Luna: So for someone listening who's considering this shift — what's the first step? Lucas: Start with a single log source. Pick something high-volume but low-risk — like web proxy logs or DNS logs. Ingest those into a data lake and run your existing SIEM queries against both systems in parallel. Prove that the data lake is faster and cheaper, then expand. Luna: Proof of concept, not a rip and replace. That makes sense. Lucas: And don't forget to involve your data engineering team early. They'll have opinions on partitioning, compression, and schema design that can save you months of headache. Luna: One quick thing — what about open-source options like ClickHouse or Druid? Are those viable? Lucas: Absolutely. ClickHouse, in particular, is gaining traction in security for its speed on large datasets. Some teams are using it as the query layer on top of S3. The trade-off is you have to manage the infrastructure yourself, versus Snowflake or Databricks which are managed. Luna: So it's a classic build-versus-buy decision. Lucas: Exactly. And it depends on your team's expertise. If you have strong DevOps and data engineering, ClickHouse can be incredibly cost-effective. If you'd rather focus on security outcomes, a managed service might be worth the premium. Luna: I want to close on one big-picture thought. We've talked about cost, speed, and flexibility — but for me, the most compelling argument is that a security data lake lets you keep all your data. In a world where threats are becoming more sophisticated and dwell times are still measured in weeks for many organizations, having that full history is a strategic advantage. Lucas: I completely agree. And I think as ai driven threat detection matures, having a rich, full-fidelity dataset to train models on will become table stakes. The teams that start building their data lakes now will be way ahead. Luna: Alright, that's a solid note to end on. Thanks for listening — we'll be back next episode. Lucas: Take care, everyone.