Latest / Cybersecurity Business with Fexingo: Security Companies, Breaches, and Enterprise Defense / How Security Teams Are Using AI-Powered Red Teaming
Transcript
- Lucas: If these conversations are useful for what you're building or running, you're probably already thinking about how to test your own defenses. And that's exactly where we're starting today — with a technique that more security teams are adopting: ai powered red teaming. Luna: So instead of having a human team try to break into your systems, you're using an AI to simulate attacks? Lucas: Exactly. And it's not replacing humans entirely — it's augmenting them. The idea is to use large language models to automate the discovery of vulnerabilities, especially in cloud infrastructure and web applications. A major bank, which I'll keep anonymous, ran a pilot earlier this year where they fed their cloud architecture into an llm based red team tool. Over the course of a weekend, it found a critical misconfiguration in their Kubernetes cluster that human testers had missed for three consecutive quarters. Luna: Three quarters? That's a long time for something critical to slip through. Lucas: It is. And it happens more often than you'd think. The human red team was following their standard playbook, but the AI explored edge cases they hadn't considered — like a combination of weak IAM roles and an exposed API endpoint that together allowed privilege escalation. This is the promise of AI red teaming: it can operate at machine speed and explore a much wider attack surface. Luna: Who's building these tools? Is it startups, or the big tech companies? Lucas: Both. Anthropic released a research paper last year showing how Claude could be used to generate adversarial prompts and probe for vulnerabilities. Google's Project Zero team has been experimenting with LLMs to find bugs in open-source code. And Microsoft has a tool called Security Copilot that can simulate phishing attacks and test endpoint detection responses. But the most interesting work I've seen is from a startup called XBOW — they built a dedicated LLM fine-tuned specifically for penetration testing, and they claim it outperforms human testers on certain benchmarks. Luna: What benchmarks are we talking about? Like, how do you even measure 'better' at red teaming? Lucas: Good question. The industry is still figuring that out. One common metric is 'time to first critical finding' — how fast can the system identify a vulnerability that matters. Another is 'coverage' — how many different attack vectors does it try? XBOW published results showing their model could find 40 percent more unique vulnerabilities than a team of three senior penetration testers in the same time frame. But there's a catch: the AI also generated a lot of false positives. Luna: So it's noisy. That's a problem for security teams who are already drowning in alerts. Lucas: Exactly. And that's where the human ai collaboration comes back in. The best practice right now is to let the AI generate a broad set of potential vulnerabilities, then have human testers triage and validate the findings. One CISO I spoke with said it's like having a junior analyst who works 24/7 and comes back every morning with a list of things to check — but you still need the senior analyst to separate signal from noise. Luna: That's a good analogy. And I imagine the cost is a factor too. LLM inference isn't cheap. Lucas: It's not. For that bank pilot, they spent about $15,000 on compute in that weekend. Compare that to hiring a three-person red team for a weekend — that's easily $30,000 to $50,000. So there's a cost advantage, but only if the results are actionable. If you're spending $15,000 on compute and then another $20,000 on human time to validate false positives, you might not be saving money. Luna: Right. So the economics depend on the tool's precision. How do you think this evolves in the next year or two? Lucas: I think we'll see specialized models that are much better at triage. The current generation of LLMs is general-purpose — they can write poetry and debug code and try to hack a server. But if you train a model exclusively on penetration testing data, it should get better at distinguishing real vulnerabilities from false positives. We're already seeing that with XBOW and with some internal tools at companies like Cloudflare. Luna: There's also a broader question here: if you're using AI to red team, should you also be using AI to defend? And can the same model be used for both? Lucas: That's a fascinating angle. Some researchers argue that you can't separate offense and defense — the same model that finds vulnerabilities can also be used to patch them. But there's a risk: if the model is used for both, you might end up with a system that knows exactly where the weak spots are, which could be a liability if the model itself is compromised. Luna: So the defender's dilemma gets more complex. Lucas: It does. And it's not just about the model — it's about the data you feed it. If you're using a cloud-based LLM, you're sending your infrastructure configuration to a third party. That introduces its own privacy and security risks. A lot of enterprises are now requiring that AI red teaming tools run on-premises or in a dedicated cloud environment with strict data controls. Luna: That makes sense. You don't want your attack surface to become the training data for someone else's model. Lucas: Exactly. And that's part of why we keep coming back to these topics — the tools evolve, but the fundamental trade-offs don't change. Speaking of which, a couple of dollars a month is genuinely what keeps these conversations going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Luna: Yeah, it really does make a difference. We keep it ad-free, and that's only possible because of listener support. Lucas: So back to the practical side — if you're a CISO looking to adopt AI red teaming, where do you start? The first step is to define a clear scope. Don't try to test everything at once. Pick one critical system, like your authentication service or your public-facing API, and run a controlled pilot. Measure the false positive rate, the time to first finding, and the cost per validated vulnerability. Then compare that to your current manual process. Luna: And be realistic about the hype. A lot of vendors are claiming their AI can replace your entire red team. That's not true right now. Lucas: Not at all. The best results I've seen come from a hybrid model: AI for breadth and speed, humans for depth and judgment. One security leader at a fintech company told me their AI red team found 70 percent of the vulnerabilities in their latest test, but the remaining 30 percent required deep domain knowledge about their specific business logic — things like transaction processing rules that an LLM just can't intuit. Luna: So the human element is still critical for the nuanced stuff. Lucas: Absolutely. And that's likely to remain the case for the foreseeable future. AI red teaming is a powerful new tool in the security arsenal, but it's not a silver bullet. The teams that adopt it wisely will be the ones that treat it as an amplifier, not a replacement.