Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Ethics Auditor Has a Conflict of Interest
Transcript
- Lucas: So earlier this week, a document leaked from one of the big four consulting firms. It was an internal memo to the team writing an AI ethics audit for a large regional bank. And the memo basically said, quote, 'We need to make sure the audit findings don't contradict the recommendations we gave last quarter on the same system.' Luna: Wait — so they're auditing their own work? That's like grading your own exam. Lucas: That's exactly the phrase that came to mind. And it's not an isolated case. The same firm designed that bank's mortgage lending algorithm, wrote its responsible AI framework, and then got hired to do the independent ethics audit required under the bank's own risk policy. The memo confirms what critics have been saying for years: AI ethics audits are often performed by people with a direct financial incentive to give the algorithm a clean bill of health. Luna: And if today's conversation gives you something to think about the next time you hear 'independent audit' — the reason we can do deep-dives like this without ads is listener support. If you find this useful, you can keep it going at buy me a coffee dot com slash fexingo. Lucas: Absolutely. It's a small gesture that lets us stay focused on the actual story, no sponsors to please. And speaking of the story — let's look at what this bank actually did. Luna: The bank is a mid-sized regional lender based in the Southeast. It issued about $2 billion in mortgages last year. The algorithm in question is a loan-approval model that considers credit history, employment, and something called 'neighborhood stability score' — which, according to the leaked audit, correlates heavily with race. Lucas: Right. The ethics audit was supposed to flag that correlation and recommend a fix. But the consulting firm's earlier contract included a clause that any negative audit findings would require the bank to buy a new model — which the same firm would then sell them. So the audit came back clean. No bias detected. The model stayed. Luna: And this is why the European Union's new AI audit framework, proposed in April 2026, includes a 'separation rule' — the same firm cannot both build and audit a high-risk AI system for the same client. Period. Lucas: Exactly. The EU's rule is modeled on existing auditor independence standards from the accounting world. You can't have PricewaterhouseCoopers audit Enron's books while also selling Enron consulting services. But in AI, that lesson had to be relearned. And the US still has no equivalent rule. Luna: So this bank's algorithm stays in place, potentially discriminating against borrowers of color, and the consulting firm collects fees on both ends. The only loser is the borrower who gets denied a loan for a reason the bank says is 'data-driven.' Lucas: Now, let's be fair. Not every ethics audit is a sham. There are genuinely independent firms — some are nonprofits, some are small boutiques that refuse consulting work. But the economics push clients toward the big four because they bundle audit and consulting at a discount. The leaked memo even said, quote, 'If we lose this audit we lose the implementation contract, which is five times larger.' Luna: So the incentive structure is fundamentally broken. The auditor is paid more if they find nothing wrong. Lucas: Right. And it's not just lending. We're seeing the same pattern in hiring algorithms, tenant screening, even predictive policing. The same companies that build the tools are the ones certifying they're fair. Luna: One concrete example: a large retailer last year hired a consulting firm to build a scheduling algorithm that would optimize shift assignments. Then hired the same firm to audit it for fairness. The audit found no bias — but a separate investigation by a news outlet found the algorithm was systematically giving fewer hours to women who requested weekends off. Lucas: That's the kind of bias an independent auditor with no stake in the outcome might have caught. But the conflict of interest creates a blind spot — or maybe a willing blindness. Luna: So what would a real solution look like? Beyond the EU's separation rule, I mean. Lucas: Several proposals are floating around. One is a mandatory rotation requirement — every three years, a company must switch its AI ethics auditor, similar to how public companies rotate audit firms for financial statements. Another is that the audit itself should be publicly disclosed, at least in summary form, so the press and regulators can scrutinize it. Luna: But wouldn't companies resist that? Trade secrets and all. Lucas: They absolutely will. The banking trade association already lobbied against the EU rule, arguing that it would increase costs and slow down innovation. But the counter-argument is: if your algorithm is fair, you should be happy to show the audit. If you're not, maybe it's not fair. Luna: There's also a movement to require that auditors be certified, like CPAs but for AI ethics. The IEEE and the Algorithmic Justice Alliance have been pushing for a credential called 'Certified AI Ethics Auditor' — with a code of ethics and continuing education requirements. Lucas: That would help, but only if the certification body itself is independent. If it's funded by the consulting firms, we're back to square one. Luna: Right. So let's talk about what the market is doing. Are any companies voluntarily separating audit from consulting? Lucas: A few. Google's DeepMind has an internal ethics unit that is structurally separate from its product teams — they publish their own reports and can veto projects. But that's in-house, not third-party. For external audits, some European banks have started hiring small boutique firms like EthicAI and FairNow, which only do audits and never consulting. They charge more, but the credibility is higher. Luna: And the cost of a bad audit? If the leaked memo were to become public and the bank faces a class-action lawsuit — which is already being discussed — the legal liability could dwarf the consulting fees. Lucas: Exactly. So this isn't just an ethics issue; it's a risk management issue. Boards of directors need to start asking: 'Who audited our AI, and are they independent?' The answer might save them from a crisis down the road. Luna: I want to go back to that leaked memo for a second. How did it leak? And does it name names? Lucas: It was posted anonymously to a document-sharing site favored by whistleblowers. It doesn't name the bank or the consulting firm explicitly — they're referred to as 'Client A' and 'Project X' — but the details are specific enough that anyone familiar with the industry can identify them. The SEC is reportedly looking into it. Luna: So there's a chance this becomes a regulatory case that sets a precedent. Lucas: That's the hope. If the SEC or the CFPB finds that the audit was misleading, it could establish that AI ethics audits have a duty of care similar to financial audits. That would be a game-changer. Luna: And for now, the practical takeaway for our listeners? If you work at a company that's deploying a high-risk AI system, ask who audited it. If it's the same firm that built it, you have a problem. Lucas: Agreed. And if you're a consumer, ask your bank or insurer: 'Who audited your algorithm, and can I see the report?' You might not get a straight answer, but the question itself puts pressure on them. Luna: Ultimately, this boils down to the same principle that applies to any profession: don't let the fox guard the henhouse. Or, in this case, don't let the consultant audit the algorithm they designed. Lucas: Exactly. And on that note, we'll be watching the SEC investigation. If it leads to new rules, we'll cover it. In the meantime, keep asking questions — especially the ones that make people uncomfortable. Luna: Thanks for listening, and we'll catch you next time.