Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When AI Decides Your Loan with No Human Appeal
Transcript
- Lucas: Luna, I want to talk about something that's happening right now in lending that most people don't even realize — fully automated loan underwriting systems that have no human appeal process at all. Luna: You mean where an AI says no and there's no way to talk to a person? That sounds like a nightmare for someone who actually deserved the loan. Lucas: Exactly. And it's more common than you think. A Consumer Financial Protection Bureau study from late last year found that 78 percent of people whose loan applications were denied by an AI system never receive any explanation for the denial beyond a generic letter. Luna: Seventy-eight percent? So the vast majority of borrowers are just left in the dark. Lucas: Right. And the reason matters because these systems are making decisions based on thousands of data points — some of which might be wrong. Take the case of a small business owner in Ohio named Maria. She runs a catering company. She applied for a business expansion loan through an online lender that uses a fully automated underwriting model. Luna: What happened? Lucas: She was denied. Then she applied again six months later — denied again. Over eighteen months, she applied four times and was denied every time. No explanation beyond 'your application does not meet our criteria.' Eventually she hired a lawyer who demanded the data the model used. Turned out there was a single incorrect entry in a commercial database — a misreported late payment on an unrelated account from five years earlier that wasn't even hers. Luna: And the AI just kept compounding that error. No one ever flagged it because there was no human underwriter to spot the inconsistency. Lucas: Precisely. And that kind of thing is happening at scale. The Consumer Financial Protection Bureau report also noted that error rates in automated underwriting models are still not systematically audited by many lenders. They deploy the model, monitor default rates, but rarely check whether the denials themselves are correct. Luna: So the feedback loop is broken. The model only learns about false positives — people who got a loan and then defaulted. But it never learns about false negatives — people who were wrongly denied. Lucas: That's the core problem. And it's particularly acute for small businesses and minority borrowers. A study from the Federal Reserve Bank of Atlanta found that Black-owned businesses were 1.8 times more likely to be denied by an automated system than by a human underwriter, even when controlling for credit score and revenue. Luna: Because the AI might be picking up on proxy variables — like the average income in a zip code or the type of business — that correlate with race but aren't legitimate risk factors. Lucas: Exactly. And without a human appeal process, those borrowers have no way to challenge the decision. Some states have started to act. Colorado passed a law last year that requires any fully automated lending decision to include a mechanism for human review upon request. California is considering similar legislation. Luna: But even with human review, if the system is a black box, what is the human supposed to review? They might just rubber-stamp the AI's decision because they don't understand how it got there. Lucas: That's a real concern. The Colorado law does require lenders to provide a 'meaningful explanation' — but nobody's quite sure what that means in practice. Some lenders are offering a summary of the top three factors that influenced the decision. But those factors are often generic — 'debt to income ratio,' 'payment history' — things the borrower already knows. Luna: So the explanation isn't actually helpful. It's like being told you failed a test because you didn't score high enough. Lucas: Right. And the deeper issue is that the models themselves are proprietary. Lenders argue that revealing too much about how they work would allow people to game the system. But there's a tension between protecting trade secrets and protecting consumers. Luna: I've also heard about a newer approach called 'adversarial explanations' — where the model generates a counterfactual: 'If your revenue had been five percent higher, you would have been approved.' Is that catching on? Lucas: It is, but mainly in academic research and a few fintech startups. The challenge is that counterfactuals can be misleading if the model is non-linear. A small change in one variable might flip a denial to an approval, but that doesn't mean the borrower can realistically make that change. Still, it's better than nothing. Luna: Let me ask you this — is there any scenario where fully automated lending with no human appeal is actually better than the alternative? I'm thinking about speed and cost. A human underwriter might take days and charge fees. Lucas: There are trade-offs. For very small loans — say a five hundred dollar microloan — the cost of a human review could exceed the profit margin. So some lenders argue that automation is the only way to serve that market. But the question is whether we should accept higher error rates in exchange for access. Luna: And whether the people who get those loans even know they're being evaluated by a machine with no recourse. Lucas: Exactly. A 2023 survey by the Pew Charitable Trusts found that only 34 percent of borrowers knew whether their lender used AI in underwriting. So most people are walking into this blind. Luna: That feels like a disclosure problem. If you're going to be judged by an algorithm, you should at least know that upfront. Lucas: I agree. And some advocates are pushing for a 'right to a human' — a legal requirement that any significant financial decision made by AI must be reviewable by a person upon request. The European Union's AI Act has something similar for high-risk systems, but the US doesn't have a federal equivalent yet. Luna: So where does this leave someone like Maria, the Ohio caterer? Did she ever get her loan? Lucas: Eventually, yes. After her lawyer got the data corrected, she reapplied and was approved within a week. But she lost eighteen months of potential growth. And the lender never acknowledged the error. They just said 'the new application met the criteria.' Luna: No apology, no systemic fix. Just 'move along.' Lucas: Right. And that's the part that worries me most. These systems scale not just good decisions, but bad ones. A human underwriter who makes a mistake affects one person at a time. An AI model that makes a systematic error can deny thousands of qualified applicants before anyone notices. Luna: And by the time someone does notice, the damage is done. You know, this kind of conversation makes me think about the value of independent, ad-free journalism that digs into these issues. If today's discussion gave you something to think about, and if it's worth the price of a coffee to you, you can support the show at buy me a coffee dot com slash fexingo. No pressure, just a way to keep these conversations going without ads or sponsors. Lucas: That's a good way to put it. And honestly, listener support is what lets us spend the time to track down these specific cases instead of just covering surface-level trends. So thank you to anyone who chips in. Luna: Alright, back to the topic. Lucas, you mentioned that some states are acting. What about the federal level? Is the CFPB planning to do more? Lucas: There are rumors that the CFPB is drafting a rule that would require lenders to conduct 'adverse action notice' testing — basically, they'd have to verify that the reasons given to denied applicants are actually the reasons the model used. Not just boilerplate. But nothing has been proposed formally yet. Luna: So for now, the burden is on the borrower to figure out if they've been wronged. And most don't have the resources to hire a lawyer and demand an audit. Lucas: Exactly. And that's why these issues matter beyond the tech world. They affect real people's ability to start a business, buy a home, or get an education. The AI is making a decision that changes someone's life, and if there's no human appeal, the system is effectively unaccountable. Luna: I think the key takeaway for me is that automation isn't inherently bad — but removing the human safety net turns efficiency into a liability. Thanks for unpacking this. Lucas: Glad we could. And hopefully, we'll see more regulation that requires a meaningful human review — not just a checkbox, but a real second look.