Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / The AI System That Decides Your Credit Score
Transcript
- Lucas: So you apply for a store credit card at a mid-tier retail chain — maybe a furniture place, or an electronics outlet. You have decent credit, stable income, no late payments. The system says no. And when you call to ask why, the representative reads a script: 'Our proprietary algorithm determined your application did not meet our risk threshold.' Luna: They have no idea why it said no either. Lucas: Exactly. That's the core tension in ai driven credit scoring. On one side, lenders say machine learning models catch patterns humans miss — they reduce defaults, they expand access to credit for thin-file borrowers. On the other side, those same models can encode biases in ways that are nearly impossible to audit. And regulators are waking up to the fact that the 'why' matters as much as the outcome. Luna: We've talked a lot about AI in hiring, in healthcare, in criminal justice. But credit scoring touches almost every adult in the country. It feels like a blind spot. Lucas: It is. And there's a specific case from 2023 that really crystallizes the problem. A major credit bureau — one of the big three — was sued by the Consumer Financial Protection Bureau for using a machine learning model that allegedly penalized customers who used certain types of retail credit cards. The model had been trained on transaction-level data, and it learned that people who maxed out store-specific cards were a higher default risk. But here's the kicker: the model also flagged people who simply those cards, even if they carried zero balance. Luna: So it was basically punishing people for their choice of wallet, not their actual financial behavior. Lucas: Right. And the bureau's complaint pointed out that the bureau itself couldn't fully reverse-engineer the model's decision logic. It was a neural net with hundreds of features — some of them derived in ways the developers themselves struggled to explain. The case eventually settled, but it set a precedent: lenders can't just say 'the algorithm decided' and call it compliance. Luna: And the Federal Reserve released a study last year that quantified how often this happens. Lucas: Yes — the 2025 Federal Reserve note on machine learning in consumer credit. They looked at anonymized application data from a large online lender, and they found that 18 percent of credit denials from the ai driven model could not be explained by traditional risk factors: payment history, debt to income ratio, length of credit history. Those 18 percent were essentially 'algorithmic black boxes' — the model said no, but conventional underwriting would have said yes. The Fed study called it a 'significant transparency gap.' Luna: And that gap falls hardest on people who are already on the margins — thin credit files, young borrowers, recent immigrants who have no traditional history but might be perfectly creditworthy. Lucas: Exactly. The same study showed that applicants with what the model classified as 'non-traditional credit usage' — which included things like paying rent via a fintech app, or using a prepaid debit card with direct deposit — were disproportionately denied, even when their cash flow was solid. The model simply didn't have enough training examples for those patterns, so it classified them as high risk by default. Luna: So the bias isn't necessarily malicious — it's a data artifact. But the outcome is the same: people get locked out of credit, and they never know why. Lucas: And that's the part that's starting to shift legally. The 'right to explanation' isn't just an academic concept anymore. In the EU, the AI Act explicitly requires that high-risk AI systems — and credit scoring is classified as high-risk — provide meaningful explanations for their decisions. In the US, it's more fragmented, but we're seeing state-level bills. California's proposed AI Accountability Act would force lenders to publish annual audits of their credit models, including the features that most influence denials. Luna: But is a published audit enough? If the model is a deep neural net, even the audit might just list 'feature X had weight Y' — not a reason a human can act on. Lucas: That's the hard part. There's a whole field called 'explainable AI' — XAI — that tries to produce human-readable explanations. But the most common technique, SHAP values, essentially tells you which input features pushed the decision in which direction. It doesn't tell you the model learned that relationship. And sometimes, the explanation itself can be misleading. For example, a model might learn that people who fill out the application on a desktop are lower risk than those on mobile — because historically, desktop users had higher incomes. The SHAP value will say 'device type — mobile,' but the real driver is income proxy. So the explanation is technically accurate but substantively wrong. Luna: That feels like a loophole lenders could exploit — 'our model uses these 200 features, and we can explain each one,' but the explanation is meaningless. Lucas: And some do. But the regulators are catching on. The CFPB has started requiring lenders to submit not just their model's outputs, but their 'explainability reports' — and they're comparing those reports against actual denial patterns. If the explanation says 'income was the primary factor,' but the data shows the model is denying people at every income level who use a certain type of bank account, that's a red flag. Luna: So the conversation is shifting from 'is the model fair' to 'can we prove it's fair — and to whom.' Lucas: Exactly right. And that shift has implications for anyone building a credit product. If you're a fintech startup designing a new scoring model, you can't just optimize for default prediction anymore. You also have to optimize for transparency — because if you can't explain why your model said no, you might not be allowed to use it at all. Luna: And that's the kind of conversation that only happens when people care enough to ask the hard questions. It's actually what keeps this show ad-free — listeners like you who support the work. If today's deep dive into credit scoring gave you something to think about, and you want to help us keep digging into these stories without any corporate sponsorship, you can throw a few dollars our way at buy me a coffee dot com slash fexingo. Every bit helps us stay independent. Lucas: And independence matters especially here, because credit scoring is a space where the biggest players — the bureaus, the large lenders — have a lot of incentive to keep their models opaque. If we were funded by a bank or a tech company, we'd have to tread carefully. So listener support really does make a difference. Luna: Alright, back to the model. One of the more promising approaches I've seen is 'counterfactual explanations' — instead of listing features, you say 'if your income were $5,000 higher, you'd have been approved.' Lucas: That's actually one of the most intuitive formats for consumers. Researchers at the University of Oxford tested counterfactual explanations against shap based explanations with real loan applicants, and the counterfactual group understood their denial reason correctly 89 percent of the time, versus 54 percent for the feature-weight group. The catch is that counterfactuals are computationally expensive — you have to run the model many times to find the minimal change that flips the decision. Luna: But it's doable, especially with cloud computing. And it gives the consumer something actionable: 'get your income up by X' or 'pay down your credit card balance by Y.' Lucas: Right. And that's the direction the CFPB is pushing. They've published guidance saying that explanations should be 'specific and actionable,' not just 'your score was below our threshold.' So a lender that uses a neural net better have a way to produce that kind of output. Luna: What about the earlier stage — model development? Is there a way to build fairness in from the start? Lucas: Yes, and it's called 'equitable machine learning.' Instead of training on raw historical data — which reflects past discrimination — you train on a balanced dataset or you use constraints that force the model to ignore certain proxies. For example, you can explicitly remove zip code, or you can require that approval rates be similar across demographic groups. But there's a tension: those constraints can reduce accuracy. A model that's forced to be fair might approve a few more riskier applicants, which means higher default rates. So lenders have to decide how much accuracy they're willing to trade for fairness. Luna: And that's a business decision, not just a technical one. But it should be a transparent decision that consumers and regulators can see. Lucas: Absolutely. And we're seeing some lenders move in that direction. One of the largest online lenders in the US — I won't name them, but they originate billions in personal loans — recently published a 'fairness audit' alongside their quarterly earnings. They showed that their model's approval rate for applicants from majority-Black neighborhoods was within two percentage points of the rate for majority-white neighborhoods, after controlling for credit score. That kind of transparency used to be unheard of. It's still voluntary, but it sets a benchmark. Luna: So the picture is complicated. There are real risks, but there are also real efforts to fix them. The question is whether regulation will catch up fast enough. Lucas: That's the open question. The EU is moving, California is moving, but the federal approach in the US is still piecemeal. And meanwhile, the models get more complex every quarter. The next frontier is 'deep learning for credit' — using transaction sequences, social media data, even phone usage patterns. Those models are even harder to explain. So the window to establish transparency standards is closing. If we wait five more years, the models will be so complex that even counterfactuals might not help. Luna: That's a sobering thought. But at least we're having the conversation now. And for anyone listening who wants to dig deeper, the Federal Reserve study I mentioned is publicly available — search for 'Fed Notes 2025 machine learning credit.' Lucas: And if you're sitting on a credit denial you think was unfair, the CFPB has a complaint portal. They actually track the number of ai related complaints now. Use it. It's one of the few ways regulators get data on what's actually happening on the ground. Luna: Good advice. Until next time, keep questioning the algorithm. Lucas: See you next episode.