Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Insurance Adjuster Denies Your Claim
Transcript
- Lucas: So there's this story out of Ohio — a woman named Patricia whose basement flooded after a record rainfall. She filed a claim with her homeowners insurance, and within 48 hours she got a letter saying the claim was denied. The reason? An AI model had determined that the damage was caused by 'gradual seepage,' which her policy explicitly excluded. Luna: But she had photos showing standing water from the storm. How does an AI decide it's seepage? Lucas: That's the million-dollar question. The model was trained on thousands of past claims, and it learned to associate certain keywords in the adjuster's notes — words like 'damp,' 'mold,' 'slow leak' — with the seepage exclusion. Patricia's claim form used the word 'water damage,' but the transcript of her phone call with the adjuster mentioned that the basement had been 'damp for a few days' before she called. The AI latched onto that single phrase and overrode everything else. Luna: So a human adjuster's offhand remark becomes the deciding factor, and the homeowner has no idea that's what triggered the denial? Lucas: Exactly. And this isn't some obscure startup. This is a top-five insurer — I won't name them because they all do some version of this — using a model built by a major AI vendor. The system scans the entire claim file, including phone transcripts, adjuster notes, even the metadata on photos, and assigns a 'denial probability score.' If that score crosses a threshold, the claim is automatically rejected without a human ever looking at it. Luna: Which means the human appeal process is already stacked. You're fighting a machine that made a decision in seconds, and you have to prove a negative — that the water didn't seep in gradually. Lucas: Right. And the model doesn't explain itself. It just spits out a score. So the insurer's response to Patricia's appeal was essentially a form letter saying 'our system has reviewed your claim and the decision stands.' No reasoning, no transparency. She eventually hired a public adjuster — someone who works for the policyholder, not the insurance company — and after two months of back and forth, they got the denial overturned. But by then, she'd already paid for emergency water extraction out of pocket. Luna: How common is this? I mean, we hear about AI in underwriting, but claims is a whole different ballgame. That's real money for real people. Lucas: It's growing fast. A 2025 survey by the National Association of Insurance Commissioners found that about 60% of large property insurers now use AI in some part of the claims process, up from 35% just three years earlier. And the most common use case is exactly this — automating the initial denial decision for low-to-medium dollar claims, typically under fifty thousand dollars. The logic is that it saves money and speeds things up for straightforward claims. Luna: But straightforward is in the eye of the algorithm. If the model is trained on historical data that reflects past biases, doesn't it just encode those biases into the denial rates? Lucas: That's exactly what a study from the Consumer Federation of America found last fall. They looked at denial rates across zip codes in three states — Texas, Florida, and California — and compared claims processed by AI versus claims processed entirely by humans. In majority-white neighborhoods, the AI denial rate was roughly the same as the human rate, around 12 percent. In predominantly Black and Hispanic neighborhoods, the AI denial rate was 19 percent — 40 percent higher. And here's the kicker: even when the AI didn't deny the claim outright, it flagged those claims for 'manual review' at a much higher rate. And manual review almost always resulted in a lower payout. Luna: So the AI is essentially creating a two-tier system. If you live in a certain neighborhood, your claim is more likely to be automatically denied or underpaid. Lucas: And that's not because the AI was explicitly told to discriminate. It's because the training data — historical claims — already reflected patterns of lower payouts and higher scrutiny in those neighborhoods, probably due to a mix of redlining history, lower property values, and maybe even subconscious bias from human adjusters. The AI just learned those patterns as 'normal' and reproduced them at scale. Luna: This is the classic garbage-in, garbage-out problem, but with real-world consequences that ripple through families. A denied claim can mean a family can't afford to repair their home, leading to mold, health issues, even displacement. Lucas: And there's another layer. The models are often trained on policy language, not just claims data. But policy language is notoriously ambiguous — it varies by state, by insurer, even by policy year. So the AI has to interpret phrases like 'sudden and accidental' versus 'gradual deterioration.' That's a task that even experienced adjusters disagree on. But the AI makes a binary decision: covered or not covered. Luna: And there's no human judgment in the loop? No one reviewing the borderline cases? Lucas: Some insurers claim they have a 'human-in-the-loop' for claims above a certain dollar threshold, but for smaller claims — which is the vast majority — the AI decision is final unless the policyholder appeals. And the appeals process itself is often opaque. A 2026 report from the Government Accountability Office found that fewer than 10 percent of policyholders whose claims were denied by an AI system ever file a formal appeal. Most just accept the decision, not realizing they have recourse. Luna: That's bleak. It also feels like a massive regulatory gap. Insurance is heavily regulated at the state level, but most state insurance departments don't have the expertise or the resources to audit AI models. They're still using actuarial tables from the 1990s. Lucas: There are a few states trying to catch up. Colorado passed a law in 2024 requiring insurers to disclose when they use AI in claims decisions and to provide a meaningful explanation for any denial. And California's insurance commissioner issued a bulletin last year saying that ai driven denials must be based on 'accurate and complete data' and must not discriminate on the basis of race or geography. But enforcement is spotty. The insurers often argue that the models are proprietary trade secrets, so they can't share the details. Luna: Trade secrets versus consumer rights — that's a tension we've seen in other areas, like credit scoring and hiring. But with insurance, you're talking about people's homes, their health, their ability to recover from a disaster. Lucas: And that's why this episode matters. If today's conversation gave you something useful — maybe you're a policyholder wondering if your claim was fairly evaluated, or you work in insurance and you're thinking about how these tools are deployed — this is the kind of content that stays ad-free because of listener support. You can help keep it going at buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to make sure conversations like this — that dig into the real-world impact of AI — remain accessible to everyone. Lucas: So back to Patricia's case — after the public adjuster got involved, they discovered something interesting. The AI model had been trained on a dataset that overrepresented 'seepage' denials in certain zip codes. Basically, the training data contained more examples of seepage-related denials from Patricia's area than from other areas, so the model was more likely to classify any water damage claim from that zip code as seepage. It was a data skew that the vendor hadn't caught. Luna: So the model was essentially making a statistical guess based on her address, not on the specifics of her claim. That's geographic profiling. Lucas: Exactly. And the insurer's response when the public adjuster pointed this out? They said the model was 'validated by an independent third party' and stood by the decision — until the public adjuster threatened to take the case to the state insurance department. Then they suddenly settled for the full claim amount plus interest. So the system works if you have the resources to fight it. Luna: But most people don't. And they shouldn't have to. The burden should be on the insurer to prove the model is fair and accurate, not on the policyholder to prove it's wrong. Lucas: Some researchers are proposing a kind of 'algorithmic impact assessment' for insurance models, similar to what the EU's AI Act requires for high-risk systems. The idea is that before an insurer deploys a claims AI, they'd have to test it for disparate impact across protected classes and geographic areas, and make the results public in a redacted form. A few consumer advocacy groups are pushing for this at the federal level, but so far, the industry has lobbied hard against it. Luna: What about the vendors themselves? The companies building these models — they have a responsibility too, right? They're the ones creating the black box. Lucas: They do, but they're also one step removed from the consumer. The vendor sells the model to the insurer, and then the insurer deploys it. The vendor might say 'we provide the tool, how you use it is your responsibility.' But if the tool is inherently biased because of the training data, that's a design flaw. A few vendors are starting to offer 'explainability modules' that show which factors drove a decision, but they're expensive and most insurers don't buy them. Luna: So we're in a situation where the technology is profitable, the regulation is weak, and the consumers are left holding the bag. That feels like a pattern we've seen before — in credit, in hiring, in criminal justice. Lucas: It is a pattern. And the insurance industry is particularly resistant to change because they operate on thin margins and they see AI as a way to cut costs. But the irony is that false denials — the ones that get overturned on appeal — actually cost them more in the long run, because of legal fees and settlement costs. If the models were more accurate, they'd save money and avoid the reputational damage. So there's a business case for fairness. Luna: But that business case only works if insurers are willing to invest in better data and more transparent models. And right now, the incentive structure doesn't push them that way. They're rewarded for processing claims fast, not fairly. Lucas: Which brings us back to the core ethical question: when an AI makes a decision that affects someone's home or livelihood, who is accountable? The vendor? The insurer? The data scientist who trained the model? Right now, the answer is basically no one. And until that changes, we'll keep hearing stories like Patricia's. Luna: And those stories are the canary in the coal mine. If we can't get insurance AI right, what hope is there for more complex applications like healthcare or criminal sentencing? Lucas: That's the big question. And it's one we'll keep coming back to on this show. For now, if you have a claim denial story — especially one involving AI — we'd love to hear it. You can reach us through the Fexingo website. And as a reminder, this show stays independent because of listeners who find it valuable. If that's you, consider supporting at buy me a coffee dot com slash fexingo. Thanks for listening.