Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Insurance Adjuster Denies the Claim
Transcript
- Lucas: So it's May 26th, 2026. Hurricane Delta hit the Florida Panhandle three weeks ago. And right now, tens of thousands of homeowners are dealing with the aftermath — not just the water damage, but the insurance claims process. Luna: And that process increasingly isn't handled by a human adjuster. It's handled by an AI system. Lucas: Exactly. According to a report that dropped last week from the Florida Office of Insurance Regulation, one of the state's largest homeowners insurers — I won't name them, they're currently under investigation — deployed an AI claims model that denied payouts at a rate of 34 percent for claims that were later overturned on appeal. Luna: Thirty-four percent error rate. That's not a rounding error. That's systematic. Lucas: It is. And it gets worse when you break it down by demographic. The same report found that homeowners over the age of 65 were 22 percent more likely to have their claim initially denied compared to younger applicants, even when the damage was comparable. Non-English-speaking policyholders saw denial rates nearly 30 percent higher. Luna: So the model isn't just wrong. It's wrong in a biased direction. That's textbook algorithmic fairness failure. Lucas: Right. And this is where AI ethics stops being abstract and starts hitting people's checking accounts. The insurer in question had been using this model since early 2025. They told regulators it was trained on historical claims data from the previous decade. But here's the thing — that data itself reflected patterns of discrimination. Older homeowners historically file fewer appeals. Non-English speakers historically accept denials more often. The model learned those patterns as signals of 'likely fraudulent' rather than historical inequity. Luna: So it's a feedback loop. The AI looks at the past, sees that certain groups rarely push back, and flags them as high-risk. Then it denies them. And because they don't appeal, the model gets reinforced. Lucas: Exactly. And what's interesting is that the insurance industry has argued that AI speeds up claims processing and reduces human bias. In theory, a machine doesn't get tired, doesn't have a bad day, doesn't hold grudges. But in practice, if the training data is contaminated with historical bias, you just automate that bias at scale. Luna: It reminds me of the COMPAS recidivism algorithm we talked about in episode one. Same core problem: biased data in, biased decisions out. But with insurance, there's an added layer — the company has a financial incentive to deny claims. Lucas: That's the key difference. COMPAS was a tool used by judges. The company that built it, Northpointe, wasn't directly profiting from longer sentences. But an insurance AI that denies more claims directly improves the bottom line. So there's a built-in conflict between accuracy and profitability. Luna: And the appeal process becomes the safety valve, right? If you know the AI denies a certain percentage, you just build a human review layer. But that only works if people actually appeal. Lucas: And if they can navigate the system. The Florida report noted that the appeal form was only available in English and Spanish. But the affected population included significant Haitian Creole and Vietnamese-speaking communities. So even when the denial was wrong, the path to correction was blocked by language. Luna: This is exactly the kind of deployment that regulators are starting to clamp down on. Florida's new AI transparency law, which went into effect in January, requires insurers to disclose any automated decision system used in claims. And California just passed something similar last month. Lucas: Right. But disclosure alone doesn't fix bias. You need auditing requirements. Florida's law doesn't mandate third-party audits — it just says the company has to tell the state which models they're using. California's law goes further, requiring an annual bias audit by an independent firm. But the question is enforcement. Luna: And that's where we start getting into the weeds of AI regulation. The EU AI Act has a whole risk-tier system, but the US is still a patchwork. Florida and California are basically beta testing different approaches. Lucas: And you can see the industry pushing back. The American Insurance Association has argued that AI models are trade secrets, and that forcing companies to open them up for audit could expose proprietary algorithms. But that argument only goes so far when your model is denying one in three legitimate claims. Luna: Speaking of trade secrets, there's another dimension here. The Florida insurer's model was built by a third-party vendor. So the insurer can say 'we didn't build it, we just used it.' That diffusion of responsibility makes accountability really hard. Lucas: That's a pattern we've seen across AI ethics cases — from hiring algorithms to child welfare risk scores. When the model is a black box from a vendor, the deploying company can claim ignorance. And the vendor says their model performed fine in testing. The gap between test conditions and real-world deployment is where the failures happen. Luna: So what would a responsible deployment look like? If you're an insurer and you want to use AI, what do you do differently? Lucas: First, you audit your training data for historical bias before you train the model. Second, you continuously monitor outcomes by demographic group — not just overall accuracy. Third, you design the appeal process to be as frictionless as possible, ideally in the applicant's preferred language. And fourth, you have a human-in-the-loop for high-dollar claims. Luna: That all sounds sensible. But it adds cost. And the whole point of using AI is to reduce cost. Lucas: Right. There's a tension between efficiency and equity. But the cost of getting it wrong — regulatory fines, lawsuits, reputational damage — can be higher. The insurer in Florida is now facing a class-action lawsuit. The plaintiff's attorneys are citing the state's own report as evidence of negligence. Luna: And that's going to be a pattern we'll see more of. ai related class actions are already picking up. If your system causes harm, you can't hide behind 'the algorithm made me do it.' Lucas: Exactly. And this brings us to something I want to mention. If today's conversation gave you something useful, a way to think differently about AI in your own industry, that's exactly why we do this show. And listener support is what keeps it ad-free and independent. If you'd like to help, you can find us at buy me a coffee dot com slash fexingo. No pressure at all — it's just a way for people who find value in these conversations to chip in. Luna: Yeah, it really does help. We put a lot of research into each episode, and knowing listeners appreciate it makes it worthwhile. Lucas: So back to the insurance model. One thing I found striking in the Florida report was that the vendor's pre-deployment testing showed a denial error rate of only 4 percent. But in the real world, it was 34 percent. That's a massive distribution shift. Luna: What caused the gap? Was the test data not representative? Lucas: Partially. The vendor tested on clean, standardized claims data. But real-world claims are messy — photos are blurry, descriptions are incomplete, adjusters make typos. The model wasn't robust to that noise. Also, Hurricane Delta was a catastrophic event. The volume of claims surged, and the model was operating under conditions it hadn't been trained for. Luna: So the system was brittle. It worked fine in a lab, but broke under stress. That's a classic failure mode for machine learning systems. Lucas: Exactly. And it's a reminder that AI ethics isn't just about bias — it's also about robustness. A system that fails unpredictably in high-stakes situations can do real harm. The insurance commissioner in Florida has now called for a moratorium on new AI claim systems until standards are developed. Luna: That's a big move. Usually regulators are slow to act. But when you have a clear failure with evidence, it creates political momentum. Lucas: And we're likely to see other states follow. New York and Texas are already signaling interest. The question is whether the industry will get ahead of it with voluntary standards, or wait to be forced. Luna: History suggests they'll wait. But maybe this case changes the calculus. Because when 34 percent of denials are wrong, that's not just a tech problem — it's a consumer protection crisis. Lucas: It is. And I think the broader lesson here is that AI deployment in regulated industries needs to be treated like a medical device, not a marketing tool. You wouldn't release a drug without clinical trials. Why should you release an AI that decides whether someone gets their home rebuilt? Luna: That's a good framing. And it raises the question of who's responsible for setting those standards. Is it the federal government, states, or industry self-regulation? Lucas: Right now it's a mix, but the Florida case might accelerate federal interest. There's a bill in Congress — the Algorithmic Accountability Act — that's been reintroduced a few times. It would require impact assessments for high-risk AI systems. It hasn't passed yet, but events like this build the case. Luna: So we're at a pivotal moment. The technology is already deployed, but the guardrails are still being built. And the people affected are the ones paying the price while we figure it out. Lucas: That's the uncomfortable truth. Every new technology goes through a period of harm before regulation catches up. The goal is to shorten that period. And episodes like this — where we dig into specific failures — are part of that process. Because if you don't understand what went wrong, you can't fix it. Luna: And we'll be watching how Florida's moratorium plays out, and whether other states follow. Thanks for listening. Lucas: Next time we'll look at AI in hiring again, but from a different angle — the rise of 'AI面试' in China and what it means for global talent markets. Until then.