Latest / Dark Perimeter: Real Breaches, Real Stakes / The State of AI: Where It Is, Where It's Going, and What Tomorrow Looks Like
Transcript
- 0:00SPEAKER_00: Two years ago, the conversation was about whether AI was real, whether the benchmarks meant anything, whether the hype cycle would collapse the way every previous one did.
- 0:09SPEAKER_01: It didn't collapse.
- 0:11SPEAKER_00: It didn't collapse.
- 0:12SPEAKER_00: But it also didn't deliver what people said it would, at least not on the timeline they said it would.
- 0:17SPEAKER_00: And I think we're at a point now where it's worth stepping back and asking the harder question.
- 0:22SPEAKER_00: Not is AI real, but which parts of it are real and which parts are still a story we're telling ourselves.
- 0:28SPEAKER_01: The story's doing a lot of work right now, especially in security.
- 0:32SPEAKER_00: Let's start with what's actually working because I think the skeptics sometimes overcorrect.
- 0:38SPEAKER_00: The Stanford AI Index came out last month, and the numbers are genuinely interesting.
- 0:42SPEAKER_00: Top models are now clearing 50% on expert-level benchmark questions across medicine, law, mathematics.
- 0:49SPEAKER_00: Three years ago that number was closer to 15.
- 0:52SPEAKER_01: Benchmarks aren't operations.
- 0:54SPEAKER_00: No, they're not.
- 0:56SPEAKER_00: But they're not nothing either.
- 0:57SPEAKER_00: And the adoption numbers tell a similar story.
- 1:00SPEAKER_00: Somewhere around 88% of organizations are using AI in some form.
- 1:05SPEAKER_00: That's not pilot programs.
- 1:06SPEAKER_00: That's deployed in workflow.
- 1:09SPEAKER_01: Using it and relying on it are different things.
- 1:12SPEAKER_01: I can use a compass and still get lost.
- 1:15SPEAKER_00: That's exactly the distinction I want to push on, because the productivity gains are real in narrow domains.
- 1:20SPEAKER_00: Code generation, alert triage, first pass document review, customer-facing automation.
- 1:26SPEAKER_00: The data is consistent across those areas.
- 1:29SPEAKER_00: But the moment you move outside the narrow domain, the performance curve drops off a cliff.
- 1:34SPEAKER_01: And nobody tells you where the cliff is.
- 1:37SPEAKER_00: Which is one of the central problems.
- 1:39SPEAKER_00: The model doesn't know what it doesn't know.
- 1:41SPEAKER_00: It doesn't experience uncertainty the way a human analyst does.
- 1:44SPEAKER_00: It produces output with the same confident tone, whether it's right or it's completely fabricated.
- 1:50SPEAKER_01: Hallucination rates?
- 1:52SPEAKER_00: Still significant.
- 1:53SPEAKER_00: 15 to 50 percent on general tasks, depending on the model and the methodology.
- 1:58SPEAKER_00: And in high-stakes domains, legal, medical, anything requiring precise citation, the numbers climb.
- 2:05SPEAKER_00: Some studies are putting global hallucination rates above 70% for certain legal query types.
- 2:11SPEAKER_01: Which means if you're using AI to research case law or interpret a contract without a human reviewing the output, you are operating on fiction some percentage of the time.
- 2:20SPEAKER_00: And the percentage is not small enough to ignore.
- 2:23SPEAKER_00: The underlying architecture hasn't solved this.
- 2:26SPEAKER_00: These models are trained to produce the most statistically likely next token.
- 2:31SPEAKER_00: They're not assessing their own confidence.
- 2:33SPEAKER_00: They're not flagging uncertainty.
- 2:35SPEAKER_00: They're completing the pattern.
- 3:04SPEAKER_00: And the organizations that are doing this well have built review workflows that treat AI output as a draft, not a deliverable.
- 3:11SPEAKER_01: Which slows things down, which is the opposite of the pitch.
- 3:15SPEAKER_00: Right.
- 3:16SPEAKER_00: The pitch is speed.
- 3:17SPEAKER_00: The reality is that you need a human in the loop to catch the errors, which puts you back on the human's timeline.
- 3:24SPEAKER_00: The value is still real.
- 3:26SPEAKER_00: You're getting a faster first pass, better coverage across large data volumes.
- 3:31SPEAKER_00: But it's not autonomous, not reliably.
- 3:35SPEAKER_01: What about agents?
- 3:36SPEAKER_01: Because that's where the hype is right now.
- 3:38SPEAKER_00: AI agents are at what Gartner is calling the peak of inflated expectations.
- 3:43SPEAKER_00: Something like 60% of organizations are planning to deploy them within two years.
- 3:48SPEAKER_00: The number that have actually deployed them to date is around 17%.
- 3:53SPEAKER_00: And of that 17% narrow scope, supervised, nothing like the demos.
- 3:59SPEAKER_00: Most of them are doing single-task automation in a constrained environment.
- 4:03SPEAKER_00: The fully autonomous agent that plans, executes, adapts, and closes a loop without human authorization, that's not what's shipping.
- 4:11SPEAKER_01: I've seen the demos.
- 4:12SPEAKER_01: They're impressive.
- 4:13SPEAKER_01: They're also staging environments.
- 4:15SPEAKER_00: The benchmark versus real-world gap is one of the most important things to understand right now.
- 4:20SPEAKER_00: AI is rarely tested the way it's actually used.
- 4:23SPEAKER_00: The conditions are cleaner, the tasks are better defined, the failure modes aren't present.
- 4:28SPEAKER_00: You put it in a production environment with messy data and ambiguous inputs, and the performance degrades in ways that don't show up in the literature.
- 4:37SPEAKER_01: Alright, let's talk about offense because that's where I think the gap between hype and reality is narrowest.
- 4:42SPEAKER_01: The attackers are not waiting for the enterprise to figure this out.
- 4:46SPEAKER_00: No, and the numbers are striking.
- 4:48SPEAKER_00: Average e-crime breakout time, the window from initial access to lateral movement across the network dropped to 29 minutes last year.
- 4:56SPEAKER_00: That's a 65% increase in speed from the year before.
- 5:00SPEAKER_01: 29 minutes is nothing.
- 5:02SPEAKER_01: If you don't have automated detection and you're relying on a human to catch the initial alert and escalate it, they're already inside before you've opened your ticketing system.
- 5:11SPEAKER_00: And AI is compressing that timeline on the attacker side.
- 5:14SPEAKER_00: Automated reconnaissance, rapid vulnerability identification.
- 5:18SPEAKER_00: Exploit chain construction that would have taken a skilled human operator hours is now being done programmatically.
- 5:29SPEAKER_01: We're not talking about mass blast campaigns with bad grammar anymore.
- 5:33SPEAKER_01: The new generation is behavioral.
- 5:35SPEAKER_01: They're pulling data from LinkedIn, from public social media, from breach databases, from company websites.
- 5:41SPEAKER_01: They know who you report to, what projects you're working on, what your communication style looks like.
- 5:45SPEAKER_01: The email arrives and it reads like something your actual manager would write.
- 5:49SPEAKER_00: And increasingly it's not email, it's voice, it's video.
- 5:53SPEAKER_01: The RUP incident is the one I keep coming back to.
- 5:56SPEAKER_01: A finance employee at a major engineering firm was on a video call.
- 6:00SPEAKER_01: The call appeared to include the CFO and several other company executives.
- 6:05SPEAKER_01: Everyone on the call was AI generated.
- 6:07SPEAKER_01: Every participant, not just the CFO, every other face in the room, their voices, their mannerisms.
- 6:14SPEAKER_01: The employee wired$25.5 million.
- 6:17SPEAKER_00: That's the thing about deepfakes that gets underappreciated.
- 6:20SPEAKER_00: It's not just the impersonation of the primary target, it's the social proof around them.
- 6:26SPEAKER_00: Humans are wired to trust consensus.
- 6:28SPEAKER_00: If ten people on a call are all telling you the same thing, the social pressure to comply is enormous.
- 6:34SPEAKER_00: And if all ten are fabricated, that pressure is manufactured from nothing.
- 6:39SPEAKER_01: 85% of organizations reported at least one deepfake-related incident in the past year.
- 6:45SPEAKER_01: That's not a niche threat anymore.
- 6:47SPEAKER_00: The infrastructure for this is now a service.
- 6:49SPEAKER_00: Deepfake is a service platform that lower the technical bar to near zero.
- 6:53SPEAKER_00: You don't need a machine learning background.
- 6:55SPEAKER_00: You need a credit card.
- 6:57SPEAKER_01: Nation state actors are combining this with long-term infrastructure targeting energy grids, healthcare systems, financial networks, and the timing isn't random.
- 7:06SPEAKER_01: There are specific windows, FIFA this year, the Winter Olympics, the U.S.
- 7:09SPEAKER_01: midterms, where the noise floor is higher and the targeting opportunities increase, defenders are stretched, attention is split, and the attackers know the calendar.
- 7:17SPEAKER_00: The new frontier that I find particularly troubling is data poisoning.
- 7:22SPEAKER_00: Instead of attacking the AI system's outputs, you attack the training data upstream.
- 7:26SPEAKER_00: You corrupt the data set that the model is trained on, which means the model learns the wrong things, and you don't necessarily know what's happened until the model's making decisions in production.
- 7:36SPEAKER_01: And that's a hard problem because by the time you see the anomalous behavior, the model has been deployed, it's been trusted, it may have been making decisions for months.
- 7:45SPEAKER_00: The integrity of training data is a security problem that most organizations haven't operationalized yet.
- 7:51SPEAKER_00: They're thinking about model output security.
- 7:54SPEAKER_00: They're not thinking about supply chain integrity for the training pipeline.
- 7:58SPEAKER_01: Let's flip to defense because I don't want to leave the impression that it's all running in one direction.
- 8:03SPEAKER_00: The adoption numbers on the defensive side are actually higher than I expected.
- 8:08SPEAKER_00: Around 95% of organizations say they're deploying AI in some security workflow.
- 8:14SPEAKER_00: Threat detection, alert triage, incident response.
- 8:18SPEAKER_01: The triage use case is real.
- 8:20SPEAKER_01: The volume of alerts that a modern SOC has to process is not humanly manageable.
- 8:25SPEAKER_01: If AI can close 60 or 70% of the noise, the obvious false positives, the known good signatures, the duplicate alerts, you're freeing analysts for the investigations that actually require judgment.
- 8:36SPEAKER_00: The agentic SOC platforms are starting to push beyond triage into active response.
- 8:42SPEAKER_00: Quarantining hosts, isolating sessions, executing containment actions based on real-time risk assessment.
- 8:49SPEAKER_00: Some of them are doing this without waiting for human authorization.
- 8:53SPEAKER_01: Which makes me nervous in a different way.
- 8:56SPEAKER_00: Tell me.
- 8:57SPEAKER_01: An autonomous system that can quarantine infrastructure is also an autonomous system that an attacker can manipulate into quarantining the wrong things.
- 9:05SPEAKER_01: If I can feed false signals to your detection engine, I can make your AI take your own systems down for you.
- 9:11SPEAKER_01: That's not hypothetical.
- 9:12SPEAKER_01: That's a logical extension of how these systems work.
- 9:15SPEAKER_00: Adversarial inputs.
- 9:17SPEAKER_00: You're essentially prompt injecting the security platform.
- 9:20SPEAKER_01: And the platforms aren't uniformly robust against that.
- 9:23SPEAKER_01: The ones that are doing this well have kept humans in the authorization loop for high consequence actions.
- 9:28SPEAKER_01: The ones that are fully autonomous for cost reduction reasons are carrying a risk they may not have fully modeled.
- 9:34SPEAKER_00: The asymmetry problem on the defensive side is real, and I don't think it's solvable in the near term.
- 9:40SPEAKER_00: Attackers iterate fast.
- 9:42SPEAKER_00: They have no compliance requirements, no change management process, no liability for false positives.
- 9:48SPEAKER_00: Defenders are operating under all of those constraints simultaneously.
- 9:53SPEAKER_01: The organizations I've seen navigate this best, and there are a handful, they're not the ones with the most sophisticated AI.
- 10:00SPEAKER_01: They're the ones who have done the harder work of mapping which problems AI is actually solving versus which problems it's performing the appearance of solving.
- 10:09SPEAKER_00: That distinction matters more than the tooling.
- 10:12SPEAKER_01: It does, because if you buy an AI security platform and you don't change the underlying process, you've just added complexity.
- 10:19SPEAKER_01: The AI is doing something, but it's not integrated into how decisions get made.
- 10:23SPEAKER_01: It's decorative.
- 10:24SPEAKER_00: There's a version of this conversation we haven't had yet, which is where all of this is heading.
- 10:29SPEAKER_00: And I want to be careful about how I frame this because the expert disagreement here is wider than most people realize.
- 10:51SPEAKER_01: And then you have Jan Lakun, who arguably knows as much about this as anyone alive, saying current architectures fundamentally cannot get there, that large language models are a dead end for general intelligence, that we need something structurally different.
- 11:05SPEAKER_00: Jeffrey Hinton is the one I find most interesting to watch.
- 11:08SPEAKER_00: A few years ago he was saying 30 to 50 years to AGI.
- 11:12SPEAKER_00: He has since revised that to five to twenty years.
- 11:15SPEAKER_00: The fact that someone with his depth of understanding moved that much in that direction is worth paying attention to.
- 11:48SPEAKER_01: And on the defensive side, presumably the same capability is available.
- 11:53SPEAKER_00: Presumably.
- 11:54SPEAKER_00: But there's the asymmetry again.
- 11:55SPEAKER_00: Attack has the initiative, defense has to cover everything, and if the system on the offensive side is more capable, even marginally, the defender's position degrades over time.
- 12:06SPEAKER_01: What does the skeptical camp say?
- 12:08SPEAKER_01: If Lacun is right, if we hit architectural limits before we get there?
- 12:12SPEAKER_00: We're still looking at a sustained period of AI systems that are capable enough to transform threat actors' operations without being capable enough to fully transform defense.
- 12:24SPEAKER_00: The incremental capability gains keep going.
- 12:27SPEAKER_00: The reliability gaps keep being exploited.
- 12:29SPEAKER_00: And we're in this extended middle period where AI is powerful enough to change everything about how attacks are executed, but not reliable enough to be fully trusted in automated defense.
- 12:40SPEAKER_01: Which is actually the harder scenario to plan for.
- 12:43SPEAKER_00: Why do you say that?
- 12:45SPEAKER_01: Because if AGI arrives, everything changes and everyone knows it.
- 12:49SPEAKER_01: You adapt or you don't.
- 12:50SPEAKER_01: But the slow grind, capability keeps increasing, reliability keeps lagging, the threat surface keeps expanding, and you never get a clean moment where you can say this is the new normal.
- 13:00SPEAKER_01: That's operationally exhausting.
- 13:01SPEAKER_01: You never get to stop adjusting.
- 13:04SPEAKER_00: The organizations that are positioning well for that scenario are building institutional knowledge about AI's failure modes rather than just its capabilities.
- 13:13SPEAKER_00: They know where it hallucinates.
- 13:15SPEAKER_00: They know what inputs make it unreliable.
- 13:17SPEAKER_00: They know which decisions to hand off and which to keep human-centered.
- 13:22SPEAKER_01: They're treating it like a junior analyst.
- 13:25SPEAKER_00: That's exactly the right frame.
- 13:26SPEAKER_00: Fast, useful, broad coverage, occasionally wrong, sometimes confidently wrong, and never the last word on anything that matters.
- 13:35SPEAKER_01: The problem is that the vendors aren't selling it that way.
- 13:38SPEAKER_00: No, they're selling certainty.
- 13:40SPEAKER_00: And certainty is the wrong relationship to have with a system that produces statistically likely outputs and calls them facts.
- 13:48SPEAKER_01: If you're running a security program right now and someone asks you what to do with all of this, what's the actual answer?
- 13:54SPEAKER_00: Narrow your use cases to where the model's failure modes don't matter.
- 13:58SPEAKER_00: Alert triage.
- 13:59SPEAKER_00: A false positive from the AI costs you time, not data.
- 14:02SPEAKER_00: Threat intelligence summarization, first pass only, human reviews, the output, code review assistance, useful, but the human signs off.
- 14:10SPEAKER_00: Don't automate decisions where the cost of a hallucination is high.
- 14:14SPEAKER_01: And on the threat side.
- 14:16SPEAKER_00: Assume deepfakes.
- 14:17SPEAKER_00: Assume that voice and video are no longer reliable identity signals.
- 14:21SPEAKER_00: Build verification into your processes that doesn't depend on recognizing someone.
- 14:26SPEAKER_00: Out-of-band confirmation for anything involving money, credentials, or access.
- 14:31SPEAKER_01: Two-person integrity for high-value transactions through channels that weren't introduced in the session you're trying to verify.
- 14:38SPEAKER_00: And watch your training data if you're using AI in production.
- 14:42SPEAKER_00: The supply chain for AI is a supply chain.
- 14:44SPEAKER_00: It has the same integrity requirements as everything else.
- 14:47SPEAKER_01: Tomorrow doesn't look like science fiction.
- 14:49SPEAKER_01: It looks like this, but more so, the capability curve keeps climbing, the reliability gap stays uncomfortable, the attackers keep moving faster than the defenders can institutionalize.
- 14:59SPEAKER_01: And the question isn't whether AI is real.
- 15:01SPEAKER_01: That argument is over.
- 15:03SPEAKER_01: The question is whether the organizations defending against it are developing the discipline to know when to trust it and when not to.
- 15:10SPEAKER_00: And the answer to that question right now is that most of them are not.
- 15:14SPEAKER_00: Not because they can't, because it's harder than buying the platform.
- 15:18SPEAKER_01: It's always harder than buying the platform.
- 15:21SPEAKER_00: It is.
- 15:22SPEAKER_00: Good night.