Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / Why Explainable AI Matters for High-Stakes Decisions
Transcript
- Lucas: So you've probably heard the phrase 'opening the black box' applied to AI models. It's become this catch-all for transparency. But here's the thing — for certain high-stakes decisions, like whether you get a mortgage or parole, regulators are starting to demand more than just an accurate model. They want a model that can explain itself. Luna: And that's where explainable AI, or XAI, comes in. But I feel like there's a tension between a model that's really accurate and one that's easy to explain. Are we sacrificing performance for transparency? Lucas: That's exactly the debate. And it's not academic — it's playing out in real lending decisions right now. A 2025 working paper from the Federal Reserve Bank of Philadelphia looked at fintech lenders and found that in about 23 percent of loan denials, the model's stated factors didn't actually match the primary drivers of the decision. That's not just a math problem; that's a potential fair-lending violation. Luna: Twenty-three percent — that's huge. So the model might say 'debt to income ratio' was the reason, but really it was something else, like the zip code or an interaction term that effectively proxies for race. Lucas: Exactly. And that's the nightmare scenario for regulators. The Equal Credit Opportunity Act requires lenders to give specific reasons for denial. If your AI can't produce those reasons, you're in violation even if the model itself isn't intentionally biased. Luna: And if today's tech conversation gave you something usable, we should mention — we deliberately don't run ads on these episodes. It's a choice that keeps the focus on the substance. If that matters to you, the way to support it is buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small thing that lets us keep the conversation clean. No sponsors to satisfy, just the topic at hand. Luna: Alright, so back to these black boxes. The tools we use to explain them — like LIME and SHAP — they're fascinating, but they come with their own baggage. Lucas: Right. LIME, which stands for Local Interpretable Model-agnostic Explanations, works by perturbing the input data and seeing how the model's output changes. It builds a simpler, interpretable model around a single prediction. SHAP, or SHapley Additive exPlanations, uses game theory to assign each feature a contribution score. Both are post-hoc — they try to explain a model that's already trained. Luna: And the problem is that these explanations are approximations. They can be inconsistent — run LIME twice on the same prediction and you might get different explanations. That's not great when someone's mortgage is on the line. Lucas: There's a well-known paper from 2020 by Slack and others that showed you can actually fool LIME and SHAP. You can create a model that's biased but whose explanations look fair — by using adversarial techniques. So the explanation tools themselves can be gamed. Luna: That's unsettling. So we need models that are inherently interpretable, not just explained after the fact. What does that look like in practice? Lucas: Companies like ZestFinance — now called Zest AI — build what they call 'transparent' models. They use a form of machine learning that's inherently interpretable, where each feature's contribution is additive and monotonic. So you can trace exactly how each variable affects the outcome. They claim their models are as accurate as black-box ones but fully auditable. Luna: And regulators are starting to reward that. The EU AI Act, which came into force last year, classifies credit scoring as a high-risk AI system. That means deployers have to ensure their models are transparent and explainable. So it's not just best practice — it's law. Lucas: But there's a trade-off. Some problems — like image recognition or natural language processing — really do benefit from the complexity of deep neural networks. You can't easily make a deep net inherently interpretable and maintain state of the art accuracy. So for those, you're stuck with post-hoc methods. Luna: Right, but for tabular data — which is what most credit decisions use — simpler models like gradient-boosted trees often perform just as well as deep nets, and they're more amenable to explanation. So the trade-off might not be as sharp as people think. Lucas: That's a good point. A 2023 study from the University of Cambridge compared XGBoost and a deep neural net on a credit default dataset. The XGBoost model had comparable accuracy and was far easier to interpret using SHAP. So sometimes the simpler tool is both better and safer. Luna: But here's the thing — even if the model is interpretable, the explanation has to be communicated to the consumer. If I get denied a loan and the reason is 'your debt to income ratio is too high,' that's clear. But what if it's 'your payment history on revolving accounts in the third quartile of the population, weighted by a nonlinear interaction with your credit utilization rate'? That's not helpful. Lucas: No, it's not. And that's a real challenge. The EU AI Act requires that explanations be 'meaningful' to the affected person. So lenders have to translate the model's internal logic into plain language. That's a design problem as much as a technical one. Luna: And there's a risk of what some researchers call 'explanation inflation' — giving people so many reasons that they can't tell which one actually mattered. Or, worse, the explanation is technically correct but misleading in practice. Lucas: Exactly. There's a great example from a 2024 paper by researchers at Stanford and the University of Toronto. They looked at a hospital readmission model. The model used SHAP to explain its predictions, and the top feature was 'length of stay' — but that was only because the model had learned that patients who stayed longer were sicker. The real driver was an underlying condition that length of stay was correlated with. So the explanation was true but shallow. Luna: So we need explanations that capture causal relationships, not just correlations. That's a much harder problem. Lucas: It is. And it's where the field is heading. Researchers are working on causal XAI methods that try to answer counterfactual questions: 'What would need to change for you to get approved?' That's more actionable than just listing feature importance. Luna: And some regulators are already pushing for counterfactual explanations. The UK's Financial Conduct Authority has mentioned them in guidance. So we might see a shift from 'why was I denied' to 'what could I do to be approved'. Lucas: That would be a huge win for consumers. But it also puts pressure on lenders to build models that can generate those counterfactuals efficiently. Right now, most models can't do that without running thousands of simulations per prediction. Luna: So there's a computational cost too. But as compute gets cheaper, that might become less of a barrier. The bigger question is whether companies will adopt these methods voluntarily or only under regulatory pressure. Lucas: I think it'll be a mix. Some forward-thinking firms — like Zest AI and a few of the larger credit bureaus — are already investing in interpretability because they see it as a competitive advantage. If you can tell a regulator 'our model is fully auditable and here's exactly how it works,' that's a lot better than saying 'trust us, it's fair.' Luna: And for the rest, the EU AI Act and similar laws in Brazil and Canada will force the issue. By 2028, most high-risk AI systems in those jurisdictions will need to comply. So if you're a fintech operating internationally, you'd better have an XAI strategy now. Lucas: Let's talk about one specific case that illustrates the stakes. In 2023, a major US bank was sued by a group of borrowers who alleged that its AI lending model discriminated against minority applicants. The bank argued that the model was too complex to explain — that it was a trade secret. The court didn't buy it. The case settled, but it set a precedent that opacity is not a defense. Luna: So the legal risk is real. And it's not just lending. We're seeing similar pressures in hiring, insurance, and healthcare. Anywhere a model makes a decision that affects someone's life, there's a growing expectation of explainability. Lucas: One area that's often overlooked is fraud detection. Banks use AI to flag suspicious transactions. If the model flags your card as fraudulent, you get a text asking 'was this you?' and you say yes, and it's fine. But if the model denies a legitimate transaction and you lose a business deal because your card didn't work, you want to know why. The bank's explanation might just be 'our risk model flagged it,' which is useless. Luna: And that's where regulatory frameworks like the EU AI Act's 'right to explanation' become concrete. Article 22 of the GDPR already gave individuals the right to not be subject to automated decisions without meaningful explanation. The EU AI Act extends that to any high-risk system. Lucas: But there's a counterargument — some researchers say that demanding explainability could stifle innovation. If you have to use a simpler model, you might miss out on the accuracy gains of deep learning. In medical imaging, for example, a deep neural net can detect cancer from a scan more accurately than a radiologist, but it's nearly impossible to explain why it sees a tumor. Do we sacrifice those lives for explainability? Luna: That's a tough question. I think the answer depends on the stakes. In medical diagnosis, you might accept a lower level of explainability if the model is demonstrably better than the human alternative. But you'd still want validation — rigorous testing, not just a black box. And in practice, radiologists don't fully explain their own reasoning either; they rely on pattern recognition. Lucas: Right. So maybe we need a sliding scale of explainability based on risk. For low-stakes decisions like movie recommendations, no one cares. For high-stakes ones like credit or parole, we need a high bar. And for life or death ones like medical diagnosis, we might accept a black box if it's been validated to outperform experts — but with ongoing monitoring. Luna: That's essentially what the EU AI Act does — it categorizes applications into risk levels. But the implementation is still messy. How do you define 'high risk' precisely? And how do you audit a model that's constantly being updated? Lucas: Those are open questions. One approach is 'continuous monitoring' — you don't just explain the model once, you track its explanations over time to detect drift. If the top features suddenly change, that's a red flag. Some startups like Fiddler AI and Arize AI offer tools for that. Luna: And what about the people who are building these models? I think there's a cultural shift happening. Five years ago, data scientists were mostly focused on accuracy metrics. Now, more of them are thinking about fairness and explainability. But it's still not standard in most curricula. Lucas: That's changing. More universities are offering courses on ethical AI and explainable ML. Stanford has a whole track now. And companies like Google and Microsoft have published internal guidelines. But the pace of adoption is slower than the pace of deployment — which is the real worry. Luna: So where does this leave us? We have the tools — LIME, SHAP, inherently interpretable models — but they're not a silver bullet. The regulatory pressure is building, but it's uneven across jurisdictions. And the underlying tension between accuracy and interpretability isn't going away. Lucas: I think the key insight is that explainability isn't just a technical fix — it's a commitment to accountability. If you can't explain your model, you shouldn't be using it for high-stakes decisions. And as consumers become more aware of how AI affects their lives, they'll demand that commitment. Luna: And hopefully, regulators will keep pushing in that direction. The question is whether the technology can keep up with the policy — or if we'll end up with a patchwork of laws and inconsistent practices. Lucas: That's the challenge for the next few years. But for now, the message is clear: if your AI can't explain itself, it might not be ready for prime time.