Latest / Cybersecurity Business with Fexingo: Security Companies, Breaches, and Enterprise Defense / How CISOs Use AI to Triage Alerts and Beat Alert Fatigue
Transcript
- Lucas: So there's this number that's been rattling around my head for weeks. A mid-sized fintech, about two thousand employees, told me their security operations team processes over fifteen thousand alerts every day. And if you've ever worked in a security operations center, you know maybe a handful of those are worth a human's time. Luna: Fifteen thousand. That's not a team, that's a fire hose. And I'm guessing the response isn't 'let's hire more analysts,' because nobody can hire that fast. Lucas: Exactly. So what they did instead — and what a growing number of CISOs are doing — is they brought in an AI triage layer. It's not replacing the analysts. It's the first pass that says, 'this alert is probably noise, this one is worth looking at.' Luna: And the results? I mean, did it actually cut the noise, or is this another vendor demo that looks great on a slide? Lucas: In their case, it cut the daily alert volume by about seventy percent. So we're talking from fifteen thousand down to forty-five hundred. And of those, the AI flagged maybe two hundred as high priority. That's a workload a human team can actually handle. Luna: Okay, so the math is compelling on the surface. But I've heard the argument that AI triage just shifts the problem — it doesn't solve it, it moves the misses somewhere else. Lucas: That's the honest critique, and it's worth digging into. Because the way these systems work, they're trained on historical alerts and known attack patterns. So they're really good at catching the things they've seen before. The risk is that a novel attack — something that doesn't look like anything in the training data — could slip through without a human ever laying eyes on it. Luna: So you're trading a known problem — too many false positives — for a less visible one, the unknown unknowns. How do CISOs square that? Lucas: The ones I've talked to are pretty upfront about it. They keep a human in the loop for the high-priority alerts, and they also run periodic 'red team' exercises where they deliberately feed the AI attacks it hasn't seen before, to see if it catches them or kicks them up to a human. Luna: So the AI becomes more of a triage nurse than a doctor. It's not making the final call, it's just making sure the doctor sees the right patients. Lucas: Right, and that's actually a useful way to think about it, because the alternative — going back to rules-based systems — is kind of untenable. The volume of alerts has just exploded. Some of that is because we've got more tools generating more telemetry, and some of it is because attackers are churning out more noise to hide their real attempts. Luna: Yeah, I mean I remember reading that the average false positive rate in a SOC is somewhere around ninety percent. So you've got analysts spending their whole day clicking 'dismiss' on things that are not actually threats. Lucas: And that's where the burnout comes from. There was a study last year — I think it was from a big cybersecurity vendor — that said something like two-thirds of SOC analysts feel overwhelmed by alert volume. That's not a sustainable way to run security operations. Luna: And we haven't even talked about the cost. Hiring more analysts is expensive, and they're hard to find. So AI triage is almost a necessity from a budget standpoint, too. Lucas: That's a big part of why CISOs are pushing this. It's not just about efficiency, it's about retention. If you can keep your best analysts from burning out, you're not constantly recruiting and training new ones. Luna: Before we go further, I want to acknowledge something. If this episode is giving you a practical angle on AI in security operations, that's exactly what we're going for. And the way we keep these conversations ad-free is listener support — buy me a coffee dot com slash fexingo. It's a simple way to keep the lights on, so to speak. Lucas: Yeah, we really appreciate that. Even a small contribution helps us cover the time we put into research and talking to CISOs and practitioners. And it lets us keep sharing concrete examples like this one. Luna: So back to the fintech. What did the AI actually catch that the old system missed? Lucas: That's the part that made me a believer, honestly. A few months in, the AI flagged a series of login attempts that looked scattered and innocuous — different IPs, different times of day, different browsers. But the model recognized a pattern: the usernames were being pulled from a list that had been circulating on a dark web forum. It was a credential stuffing attack, and it had been slipping past the rules-based system because no single login looked suspicious. Luna: So the AI found the needle in the haystack by looking at the whole haystack at once. That's something a human analyst might eventually have caught, but probably after real damage was done. Lucas: Exactly. And that's the promise — AI can see correlations across thousands of events that a human brain just can't hold in working memory. But here's the catch: the model only knew about that dark web list because it was in the training data. If the attackers had used a fresh list, not yet indexed anywhere, the AI might have missed it too. Luna: So it's not magic. It's pattern recognition, which means it's only as good as the data it's trained on. And that's a point I think a lot of vendors gloss over. Lucas: Right. And that's why the human-in-the-loop model still matters. The AI is great at finding known patterns at scale, but it's not great at reasoning about novel situations. So CISOs are building processes where the AI escalates anything with even a hint of uncertainty, and analysts do deeper investigations. Luna: What about the analysts themselves? Do they feel threatened by this, or is it more of a relief? Lucas: From what I've heard, it's mostly relief. The analysts I talked to say it lets them focus on the interesting work — actually chasing down threats — instead of clicking through false positives. One SOC lead told me it's the first time in years his team feels like they're doing security instead of clerical work. Luna: That's a nice reframing. It's not about replacing people, it's about letting them do the higher-value work. But I've also heard the flip side — that over-reliance on AI can make analysts worse at spotting things on their own. Skill atrophy. Lucas: That's a real concern, and some CISOs are addressing it by keeping a percentage of alerts going to humans even if the AI is confident they're benign. It's like keeping your pilots in the cockpit during autopilot — you want them to stay sharp for when the automation fails. Luna: So how do CISOs decide which alerts to keep in the human loop? Is there a rule of thumb? Lucas: A common approach is to keep at least ten percent of the low-priority alerts flowing to humans for review. That way analysts stay familiar with the baseline noise, and they can catch things the AI might have misclassified. It's a small cost compared to the risk of missing a novel attack. Luna: That makes sense. What about the tools themselves — are these bespoke builds, or are CISOs buying off the shelf? Lucas: Mostly off the shelf, but with a lot of tuning. The vendors have gotten pretty good at pre-training models on common attack patterns, but each organization's environment is different. So there's a period of calibration where you feed the AI your historical alerts and let it learn what 'normal' looks like for your network. Luna: And how long does that calibration take? Weeks? Months? Lucas: Usually a few weeks to a couple of months, depending on how much historical data you have. The fintech I mentioned had about two years of alert logs, which was enough to get the model to a place where it was catching real threats without overwhelming the analysts with new false positives. Luna: So what's the bottom line for a CISO who's listening? Is this something they should be exploring now, or is it still early days? Lucas: I'd say it's past early days. The technology is mature enough that you can pilot it with a manageable scope, and the pressure on SOC teams is only going to grow. But I'd caution against thinking of AI triage as a silver bullet. It's a tool that works best when you combine it with a culture of continuous testing and a commitment to keeping your analysts engaged. Luna: So it's not 'set it and forget it.' It's more like a partnership where the AI handles the noise and the humans handle the judgment calls. Lucas: Exactly. And that partnership is what's going to define security operations for the next few years. The teams that figure it out will be the ones that don't just survive the alert flood — they'll actually catch the attacks that matter. Luna: I think that's a great place to leave it. Thanks for walking us through the specifics, Lucas. Lucas: Anytime. And if you're a listener who's been on the fence about AI triage, I hope this gives you a few concrete questions to ask your team.