Latest / Cybersecurity Business with Fexingo: Security Companies, Breaches, and Enterprise Defense / How AI Security Starts With Data Provenance
Transcript
- Lucas: So last week I saw a demo that genuinely gave me a chill. A researcher showed how you could slip a single malicious image into a model's training data — just one image out of millions — and that model would then misclassify every stop sign as a yield sign when deployed in a self-driving car. And the model didn't show any sign of tampering in standard validation tests. Luna: That's terrifying. And it's the kind of attack that doesn't even need to touch the model's weights after deployment. It's baked in during training. Lucas: Right. And that's exactly what we're going to dig into today — the business case for AI security, specifically the piece that nobody's talking about enough: data provenance. We're at a moment where every enterprise is rushing to deploy large language models and custom machine learning systems, but very few have stopped to ask a basic question: where did the training data actually come from, and can we prove it wasn't tampered with? Luna: So it's not about the model itself being attacked in production, but about the data that shapes it. Which feels like a supply chain problem wearing a machine learning costume. Lucas: That's exactly the frame. And the supply chain analogy is spot on. You know how after the SolarWinds breach, we all became obsessed with software bill of materials — SBOMs — to track every component in your software. The AI world needs the same thing, but for data. A data bill of materials, if you will. Luna: I've heard that term floating around. But how would that even work in practice? You're talking about datasets that could be terabytes of images or text scraped from the web. Lucas: The short answer is: you start with cryptographic hashes. You fingerprint every piece of data at ingestion — every image, every text snippet — and you store that hash in a tamper-evident ledger. Then when you train a model, you can verify that each batch of data hasn't changed since you recorded it. If an attacker swaps a single image, the hash changes, and you know something's wrong. Luna: But that assumes you trust the source of the data in the first place. If you're pulling from a public dataset on the internet, how do you know it wasn't poisoned before you even touched it? Lucas: That's the hard part. And that's where the concept of provenance gets really interesting. It's not just about hashing your own copies — it's about tracking the entire lineage of the data. Where did it originate? Who's touched it? What transformations has it gone through? And that's a problem that security teams haven't had to solve before, because data was never this valuable. Luna: So what happens when a company gets it wrong? Is there a real-world example that shows the stakes? Lucas: There is, and it's called PoisonGPT. In 2025, researchers demonstrated a proof of concept where they uploaded a poisoned model to the Hugging Face hub — the popular repository for open-source AI models. They named it something innocuous, and it got downloaded a bunch of times before anyone noticed. The model was fine on standard benchmarks, but it had been subtly trained to output false information when prompted about a specific topic. Luna: So it's like a sleeper agent model. It passes all the obvious tests, but it's waiting for a trigger phrase to go off. Lucas: Exactly. And while that was a proof of concept, the technique is real. The researchers showed that with access to the training pipeline, you can inject a backdoor that's nearly impossible to detect with standard evaluation. And the scary part is, many enterprises are fine-tuning open-source models — and they're pulling them from exactly these kinds of public repositories without any provenance checks. Luna: Okay, so that's the threat. But what's the business case for actually investing in this? I mean, for a CISO, this is one more thing on an already overloaded plate. Lucas: The business case is actually pretty compelling, and it comes down to the cost of a failure. If a model you deploy makes a biased decision — say, it's used in hiring and it's been poisoned to discriminate against a certain demographic — that's not just a technical bug. That's a regulatory nightmare, a PR disaster, and potentially a lawsuit. And the worst part is, you might not even know the model is compromised until it's too late. Luna: So it's an existential risk for the AI initiatives that companies are betting their futures on. If the model goes rogue, the whole project goes down with it. Lucas: And that's why I think we're going to see AI security budgets grow dramatically over the next few quarters. We're already seeing it in the conversations I have with CISOs. They're starting to ask about model provenance before they approve any new AI project. They're demanding that their data science teams document the origins of every training dataset. And they're starting to put controls in place — not just technical, but also process controls. Luna: Like what? What does a first step actually look like for a company that's serious about this? Lucas: The first step is inventory. You can't protect what you don't know you have. So you create a catalog of every AI model in your organization, every dataset that was used to train or fine-tune it, and every pipeline that touches that data. Then you start applying the provenance framework — hashing, logging, and versioning. It's a lot like how we handle software dependencies, but for data. Luna: So it's really an extension of the software supply chain discipline. Which means the tools and practices that the security community has been building for the last few years are going to be repurposed for this new challenge. Lucas: Exactly. And that's actually an opportunity for the vendors who specialize in supply chain security. They've already solved the component tracking problem, and they can adapt those solutions for AI data. But there's also a gap — no one has built the definitive tool for AI provenance yet. The market is wide open. I wouldn't be surprised if we see a wave of startups in this space over the next year. Luna: Speaking of the business side of things, I want to be honest about something. A lot of what we discuss on this show — the frameworks, the controls, the technologies — it can feel pretty far from your day-to-day if you're not in a security role. But the reason we keep doing this is because these conversations matter for anyone who's building or running a business in the digital age. And that's exactly why listener support matters too. Lucas: You're right. And this is the part where I'll say — if you're getting value from these conversations, and you want to keep them coming without ads, the simplest way to help is to buy me a coffee dot com slash fexingo. It's a small gesture that goes a long way in keeping this show going. Luna: Yeah, it's the easiest way to support the work we do. Every little bit helps. Lucas: So back to data provenance. One thing I didn't mention is the regulatory angle. The EU's AI Act is already requiring more transparency about training data, and I think we're going to see similar pressure in other jurisdictions. So even if you're not motivated by security, compliance is going to force the issue. Luna: That's a good point. So there's a convergence — the security imperative and the compliance imperative are pointing in the same direction. That makes it easier to get budget approval. Lucas: Absolutely. And the timing is interesting. We're at the end of August 2026, and the AI security market feels like where cloud security was in 2015 — everyone knows it's important, but there's still a lot of confusion about what it actually means and who's accountable. That's a massive opportunity for the people who figure it out early. Luna: So what should our listeners do tomorrow morning? What's the one actionable thing? Lucas: Start the conversation. If you're in any kind of leadership role, ask your data science team this question: 'Can you prove where our training data came from?' If they can't answer it, that's your first red flag. And then start building that inventory. It's the same advice we'd give for any other supply chain risk — know what you're running, and know where it came from. Luna: That's a great place to end. It's one of those questions that seems simple, but the answer is probably going to be complicated. And that complication is where the risk lives. Lucas: It is. And it's also where the opportunities are — for the security pros who can solve it, and for the vendors who can build the tools. We'll be keeping an eye on this space, and I'd bet we'll have more episodes on this topic in the coming months. Until then, keep asking the hard questions.