Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Credit Check Penalizes You for a Typo
Transcript
- Lucas: So there's a story out of Cleveland — a woman named Patricia Okonkwo runs a catering business called 'Golden Spoon Events.' She applied for a fifteen thousand dollar equipment loan through an online lender — think Kabbage, Fundbox, that universe — and got denied within thirty seconds. The reason: the AI flagged a 'high-risk name discrepancy.' Luna: Wait — she doesn't have a criminal record or anything like that, right? Lucas: Clean file. Perfect payment history on two existing credit cards. The discrepancy was that on her tax return — which she uploaded as a PDF — her business name was spelled 'Golden Spon Events.' One letter missing. The AI's natural language processing module treated 'Spon' as an alias, and the system's fraud model assigned a negative score. Luna: So a typo in a PDF cost her a loan. That's not a hypothetical — that actually happened. Lucas: It happened last November. I spoke with her for the episode. She eventually got the error fixed after three weeks of back and forth, but by then the equipment she needed — a commercial fridge — had been sold to someone else. The lender wouldn't re-run the application because their policy says decisions are final unless the error was on their side. They argued the error was on hers. Luna: And the AI just amplified a human mistake into a denial. This is the kind of thing that never makes the news because the dollar amount is small, but the cumulative effect is huge. Lucas: Exactly. And it's not isolated. A 2025 study from the Federal Reserve Bank of Philadelphia looked at credit denial disputes across five major online lenders. They found that twelve percent of denials involved a data entry error — a misspelled name, a wrong address number, a transposed Social Security digit — and of those, only three percent were ever corrected. The rest just became permanent records. Luna: Three percent correction rate. So ninety-seven percent of those borrowers just absorb the denial and move on? Lucas: They don't even know the error exists, because the lender doesn't tell them. The denial letter says 'our automated scoring system determined your application does not meet our underwriting criteria.' No mention of 'Spon' vs 'Spoon.' The FCRA — Fair Credit Reporting Act — requires lenders to disclose the source of adverse information if it's from a credit bureau. But if the error is inside the lender's own document parsing, that's not covered. Luna: So there's a regulatory blind spot. The law was written for a world where humans reviewed applications and could catch a typo. Now the machine reads it and the machine decides — and the human never looks. Lucas: That's the core issue. And it's not just lenders. Insurance companies use similar NLP to parse application forms. Some property insurers will quote you a higher premium if your address has a formatting inconsistency — 'Street' vs 'St.' — against a government database. The system treats them as different locations. Luna: Let's talk about the mechanics, because I think listeners might assume this is just a 'bad AI' problem. But actually, the NLP models doing this are state of the art. They're not broken. Lucas: Right. The models are working exactly as designed. They're trained to detect anomalies — anything that deviates from the pattern in a training set. If most applications have a consistent business name across documents, then a mismatch is statistically unusual. The model flags it as risk. The problem is that the training data doesn't include a label for 'benign human error.' So the model has no concept of a typo that is just a typo. Luna: It's not that the AI is biased against typos — it's that it's naive about them. It treats every discrepancy as equally meaningful. Lucas: And that's a design choice. Some fintechs do build in a 'fuzzy matching' layer — they use Levenshtein distance or Jaro-Winkler similarity to allow for minor spelling variations. But many don't, because fuzzy matching increases false positives: it lets through more cases that really are fraudulent. There's a trade-off. Patricia Okonkwo's case is an example of a lender that prioritized fraud detection over inclusion. Luna: So the question becomes: who bears the cost of that trade-off? The lender saves a few hundred dollars in potential fraud, but the borrower loses access to capital. And for a small business, that can be existential. Lucas: It gets worse when you look at the compounding effect. A single denial can lower your credit score if the lender does a hard pull. Then you apply elsewhere, and the next AI sees the denial on your record and that becomes another negative signal. A typo can trigger a downward spiral. Luna: You know, this reminds me of something we've talked about before — the idea that AI systems don't just reflect existing bias, they can create entirely new categories of bias that have no human analog. There's no human loan officer who would deny someone because of a typo in a PDF. But a machine will do it a thousand times a day. Lucas: And that's the ethical problem. The Equal Credit Opportunity Act prohibits discrimination on the basis of race, gender, age, and so on. But it doesn't say anything about 'string similarity threshold.' So a lender can legally deploy an NLP model that systematically disadvantages people with less standardized paperwork — which often correlates with lower income, less access to legal help, or English as a second language. Luna: Honestly, if this conversation gave you something to think about, that's exactly the kind of thing listener support helps us keep doing. We keep the show ad-free, and the only way that works is if folks who find value in it toss something in the hat. If today felt worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Lucas: Yeah. It's a small gesture but it genuinely makes a difference. Keeps us independent, keeps us digging into cases like this one. Luna: So back to the policy side — what's being done about it? Is there any regulatory movement? Lucas: A bit. The Consumer Financial Protection Bureau issued a guidance document in March 2026 that explicitly says lenders using AI must have a process for consumers to dispute 'algorithmic data errors' — not just credit bureau errors. But it's guidance, not a rule. No enforcement mechanism yet. Meanwhile, the Fed study I mentioned recommends something called 'error proportionality' — the idea that the materiality of an error should affect how the system handles it. A missing letter shouldn't carry the same weight as a mismatched Social Security number. Luna: That seems like common sense, but it's not how any of these systems are built today. Is there a technical fix that's straightforward? Lucas: The most practical fix is to add a human-in-the-loop for any denial triggered solely by a data inconsistency. If the NLP flag is the only reason for a low score, the application should be routed to a human reviewer before the denial is final. Some lenders are experimenting with that — including one of the big four banks, actually — but it adds cost and time, which is the opposite of what the AI is supposed to deliver. Luna: So the business incentive is to remove the human, not add one. That's the tension. Lucas: Exactly. And until the regulation catches up — or until the reputational risk of stories like Patricia's becomes big enough — most lenders will optimize for speed and fraud prevention. The typo is just collateral damage. Luna: What about the borrower side? Is there anything a small business owner can do proactively to protect themselves? Lucas: A few things. First, double-check every character in every uploaded document — sounds obvious, but most people don't. Second, if you get denied, ask for the specific reason in writing. Under ECOA you're entitled to an explanation, though in practice many lenders send vague boilerplate. Third, file a dispute with the lender's compliance officer — not just customer service. And if that fails, file a complaint with the CFPB. It won't fix your loan in time, but it creates a paper trail that regulators can use. Luna: That's a lot of burden on the borrower. The system is supposed to work for them, not require them to be data-entry experts. Lucas: Yeah, and that's the deeper point. We've built a credit infrastructure that assumes perfect data, but the data is never perfect. And the people who pay the price are the ones who can least afford to — the small businesses, the gig workers, the people without a lawyer to look over their application. The AI doesn't see that. It just sees 'Spon' and says 'no.' Luna: Alright, let's leave it there. Next time we'll look at what happens when the AI that's screening your rental application confuses your name with a flagged tenant from a different state. Lucas: That's a good one. See you next time.