Latest / Fintech Conversations with Fexingo: Banking, Payments, and Financial Software Companies / The Hidden Cost of Algorithmic Underwriting
Transcript
- Lucas: Let's start with a specific scenario that keeps me up at night. A local bakery owner in Ohio applies for a line of credit to buy a new oven. Her cash flow is solid, her tax returns are clean, and she's been in business for ten years. But the bank's AI model flags her as high-risk because her customer base skews slightly older than its training data suggests, and it denies the loan. This isn't hypothetical. It's happening right now across thousands of community banks. Luna: Right, and the kicker is that the bakery owner can't even ask why. The model is a black box. She gets a generic email saying 'insufficient creditworthiness' with no actionable feedback. Lucas: Exactly. And that's the core tension we're digging into today: the trade-off between speed and explainability in algorithmic underwriting. As of October seventeenth, twenty twenty-six, major lenders have deployed these systems at scale. They're faster, cheaper, and often more accurate at predicting default than human analysts. But they're also opaque, and that opacity is creating a new kind of financial exclusion. Luna: So you're arguing that efficiency comes at the cost of fairness, or at least transparency. Lucas: I'm arguing that the current regulatory framework is ill-equipped to handle models that don't just process data, but interpret it in ways we can't easily audit. Let's look at the mechanics. Traditional underwriting relies on linear factors: debt to income ratio, payment history, collateral value. These are easy to explain. If you miss two payments, your score drops. Simple. Algorithmic underwriting uses thousands of variables, including non-traditional data points like utility payment consistency, social media activity, or even the time of day you apply. Luna: That sounds invasive. Why would a lender care about the time of day I apply? Lucas: It turns out, there's correlation there. Studies show that applicants who apply late at night might be more stressed, or perhaps less financially organized. The model picks up on patterns humans wouldn't see. But here's the problem: if the model denies the bakery owner because of a variable related to her zip code or age, it might be violating fair lending laws without anyone realizing it. The model isn't racist or sexist, but it can reproduce historical biases embedded in the training data. Luna: So the bias isn't intentional, it's inherited from past lending practices. Lucas: Precisely. And that's hard to fix when the model itself doesn't know which variables drove the decision. This is where the concept of Explainable AI, or XAI, comes in. Lenders are starting to use techniques like SHAP values, which break down the contribution of each variable to the final score. But even then, the output is complex. Telling a small business owner 'your score was lowered by three percent due to variance in utility payments over the last eighteen months' doesn't help them understand the broader systemic issue. Luna: It feels like we're asking for a translation of something that wasn't designed to be translated. Lucas: That's a great way to put it. And the regulators are scrambling. The Federal Reserve and the Consumer Financial Protection Bureau have issued guidance emphasizing that lenders must be able to explain adverse action notices. But 'explain' is a vague term. Does it mean listing the top three factors? Or does it require a full causal explanation, which most black-box models can't provide? Luna: Most lenders are probably just sticking to the minimum legal requirement to avoid liability. Lucas: They are. And that creates a gap. If the law requires a simple explanation, but the model only provides a complex one, lenders will simplify the truth, potentially misleading consumers. Meanwhile, the actual decision-making power remains hidden. We saw this with mortgage algorithms a few years ago, where disparate impact on minority borrowers was discovered only after independent audits. Now, with small business lending moving online, the same risks are emerging, but faster. Luna: Because small business loans are smaller, so the volume is higher, and the scrutiny is lower. Lucas: Exactly. In consumer mortgages, you have strict documentation requirements. In small business lines of credit, it's often automated approval within minutes. There's no human review. So if the algorithm is wrong, or biased, thousands of businesses could be denied funding simultaneously, with no recourse. This is a scalability problem for inequality. Luna: It's inequality at scale. That's chilling. Lucas: Let's talk about the tech solution side. Some fintech companies are building 'glass box' models, which are simpler, interpretable neural networks. They sacrifice a tiny bit of predictive accuracy for full transparency. Early results suggest they perform nearly as well as black boxes, especially when combined with human oversight. But adoption is slow because the biggest lenders prefer the marginal gain in accuracy from the black boxes. Luna: So the incentive structure is misaligned. Accuracy wins over accountability. Lucas: Until the regulatory penalty outweighs the benefit. And that's changing. States like California and New York are pushing for stricter algorithmic accountability laws. The EU's AI Act already classifies credit scoring as high-risk, requiring rigorous testing and documentation. US lenders are watching closely. If the EU sets the standard, global fintechs will likely adopt it to avoid fragmentation. Luna: Will that force US banks to change their models, or just add a layer of compliance? Lucas: It depends on whether the compliance is substantive or checkbox. If it's just paperwork, nothing changes. But if it requires genuine model governance, like regular bias audits and feature importance reporting, then yes, it will force structural change. The key metric here is 'feature stability.' If a model's reliance on certain variables shifts dramatically between quarters, it's a red flag for instability or bias. Luna: So monitoring the inputs, not just the outputs. Lucas: Correct. And that requires a new skill set in banking. Data scientists need to work closely with compliance officers, not just product managers. Currently, they're siloed. The data team builds the model. The compliance team checks it after the fact. That lag is dangerous. We need continuous monitoring, not periodic reviews. Luna: Continuous monitoring sounds expensive. Do smaller banks have the resources for that? Lucas: That's the equity question. Large banks can build internal teams. Community banks might rely on third-party vendors. But if those vendors sell black-box solutions, the community bank is still in the dark. We're seeing a rise in 'compliance as a service' platforms that offer explainable AI tools specifically for smaller lenders. It's a niche market, but it's growing fast. Luna: So there's a potential market solution emerging to fill the regulatory gap. Lucas: There is. And it's worth noting that some consumers are actually preferring transparent models. A survey from last year showed that borrowers were more likely to trust a lender who could explain the reasons for a denial, even if the denial stood. Trust is a currency too. If lenders lose trust because their decisions feel arbitrary, they'll face higher churn and reputational damage. Luna: Trust is harder to rebuild than a credit score. Lucas: Absolutely. Let's pivot to the borrower's perspective. What can a small business owner do? First, diversify your financial footprint. Don't rely on a single digital wallet or platform. Second, document everything. Keep clear records of cash flow, invoices, and communications. Third, if you're denied, request a detailed reason. Don't accept the generic email. Push for specifics. Often, the pushback forces a manual review. Luna: Push back against the algorithm. That's counter-intuitive but smart. Lucas: It is. Because algorithms respond to signals. If you signal that you're paying attention, you might trigger a human review. And humans are still the ultimate arbiters of appeal. Most AI systems have a 'human-in-the-loop' override for disputed decisions. It's not efficient, but it's necessary for justice. Luna: Efficiency versus justice. Classic dilemma. Lucas: And one that defines the next decade of fintech. As we move towards real-time payments and instant credit, the speed of decision-making will increase. But the complexity of the models will also increase. We need to ensure that our regulatory and technical frameworks evolve in tandem. Otherwise, we risk creating a system that is fast, accurate, and fundamentally unfair. Luna: A system that works perfectly for everyone except the people it's supposed to serve. Lucas: Right. And the danger is that we won't notice until it's too late. Bias in lending compounds over time. Denied loans lead to missed opportunities, which lead to weaker credit histories, which lead to further denials. It's a vicious cycle. Breaking it requires deliberate effort, both from regulators and from the companies building these tools. Luna: What's the one thing listeners should take away from this conversation? Lucas: Don't assume that a digital decision is neutral. Just because it's automated doesn't mean it's objective. Question the data, demand transparency, and remember that behind every algorithm is a set of choices made by humans. Luna: And if you're building these tools, prioritize explainability from day one, not as an afterthought. Lucas: Agreed. It's not just good ethics, it's good business. Sustainable growth requires trust, and trust requires clarity. Luna: Before we go, I want to mention that these conversations are ad-free and listener-funded. If you find value in our deep dives, consider supporting us via buy me a coffee dot com slash fexingo. It helps keep the lights on and the research sharp. Lucas: Yeah, we appreciate every bit of support. It allows us to dig deeper into topics like this without compromise. Luna: Thanks for listening. We'll see you in the next episode. Lucas: Until then, keep questioning the black boxes.