Latest / B2B SaaS Talks with Fexingo: Enterprise Software, Sales Cycles, and Procurement / Why Enterprise Buyers Now Mandate a Vendor AI Explainability Audit
Transcript
- Lucas: Enterprise software contracts have a new standard clause this year, and it's one that would have sounded borderline impossible five years ago: the vendor AI explainability audit. Luna: An explainability audit — so procurement is basically asking vendors to prove their AI can tell you why it made a decision, in plain language, not just spit out a prediction. Lucas: Exactly. And this isn't academic. We're seeing it in RFPs from health insurers, banks, even logistics companies. The trigger was a couple of high-profile cases where black-box models made decisions that looked perfectly valid in aggregate but turned out to be deeply flawed when you actually asked why. Luna: I remember one from last year — a large health plan was using an AI to flag claims for denial. The model had great accuracy numbers, but when a regulator dug in, they found it was weighting zip code as a proxy for socioeconomic status, which effectively discriminated against certain patient groups. Lucas: Right. And the vendor's defense was essentially 'we can't tell you exactly why — it's a deep neural net.' That argument used to fly. Not anymore. Now buyers are saying: if your AI makes a decision that affects our customers, our compliance, or our reputation, we need a human-readable justification on demand. And we want it as a contractual obligation. Luna: So what does the clause actually look like in practice? Is it a one-pager, or is it pages of technical specs? Lucas: It's surprisingly specific. The typical version I've seen has three sections. First, the vendor must provide a 'global explanation' of the model — which features are most influential overall, how the model behaves across different segments of data. Second, they need to support 'local explanations' — for any individual prediction, they have to be able to produce a ranked list of factors that drove that specific output. Luna: So for a loan denial, you'd get something like 'your application was declined because your debt to income ratio was 48 percent, and that was the primary driver, followed by length of credit history at 22 percent.' Lucas: Exactly that. The third section is the punchline: the vendor has to commit to a maximum response time for producing that explanation. We're seeing SLAs of 24 hours for batch explanations, and for real-time decisions — like a credit card transaction being flagged — they need to have the explanation available within the same API call. That's a technical challenge for a lot of legacy AI stacks. Luna: And I imagine not all models can actually do this. Some are genuinely black boxes — like ensemble methods or certain deep learning architectures where the internal logic is distributed across millions of parameters in a way that's not easily summarised. Lucas: That's the tension. The clause usually includes a 'best efforts' standard for models where perfect explainability isn't feasible, but buyers are getting stricter. I've seen deals fall through because the vendor couldn't provide anything beyond a simple feature-importance chart. The buyer's legal team wanted something they could put in front of a regulator or a judge. Luna: So what's the workaround? Are vendors retraining models to be more explainable, or are they layering an explanation system on top? Lucas: Both. Some vendors are shifting to inherently interpretable models — like generalized additive models or decision trees — for high-stakes decisions. But for cases where you really need the accuracy of a deep neural net, they're using post-hoc explanation techniques like SHAP values or LIME. SHAP, which stands for SHapley Additive exPlanations, is actually based on cooperative game theory — it calculates each feature's contribution by averaging its marginal impact across all possible subsets of features. Luna: That's elegant. But the catch with LIME and SHAP is they're approximations. They're not the real model, they're a simplified proxy. A court might not accept that as full proof. Lucas: And that's exactly why the audit clause is so important. It forces the vendor to document not just what the explanation is, but how reliable that explanation is. Some contracts now require a 'faithfulness metric' — a score that measures how well the explanation matches the actual model behavior. If your SHAP values only correlate at 0.6 with the true decision boundary, the buyer can push back. Luna: Let's talk about who's leading this. I assume regulated industries are ahead, but are we seeing it in less regulated sectors too? Lucas: Healthcare and financial services are definitely the pioneers. But I'm seeing it creep into HR tech — ai powered resume screening, performance evaluation tools. There have been several lawsuits around biased hiring algorithms, and employers are now demanding explainability from their vendors to shield themselves from liability. Luna: Makes sense. If you're a company using an AI to filter candidates, you need to be able to tell a rejected applicant why they weren't selected, especially if there's a disparate impact on protected groups. Lucas: Right. And the clause typically requires the vendor to provide that explanation in a form that the buyer can share with the applicant — so it has to be in plain language, not a dump of SHAP values. Some contracts even specify a maximum reading level, like 'no more than an 8th grade reading level.' Luna: That's a fascinating level of detail. Let's step back — how widespread is this clause actually? Is it still niche, or are we at a tipping point? Lucas: Gartner published a survey in May that's relevant here. They polled enterprise procurement leaders at companies with over 5,000 employees, and 67 percent said their most recent software RFP that included an AI component also included some form of explainability requirement. That's up from 22 percent in 2024. So in two years, it's gone from a niche ask to a near-majority expectation. Luna: That's a massive shift. And I bet the vendors who can't comply are losing deals. Lucas: Yes. Several startups have built their entire pitch around 'explainable AI by design.' One company I spoke with, they market themselves as 'AI that testifies' — they guarantee that every model output comes with a natural-language justification that meets audit standards. They're winning contracts against much larger incumbents who are still trying to retrofit explainability onto legacy systems. Luna: It reminds me of when SOC 2 compliance became table stakes a decade ago. At first, it was a differentiator, then within three years, if you didn't have it, you couldn't sell to enterprise. Lucas: Exactly the same pattern. And just like SOC 2, the audit itself is becoming a business. There are now third-party firms that specialize in AI explainability audits — they'll come in, test your model, produce a report that both vendor and buyer can rely on. One of the bigger ones told me their revenue grew 300 percent year-over-year in 2025. Luna: So for vendors listening — what should they do if they're getting asked for this clause and they don't have it? Lucas: First, don't panic. Start with an internal audit of your most critical models. Identify which ones are truly black-box and which can be explained with current tools. Then, prioritize the models that are customer-facing or that make decisions with regulatory implications. For those, invest in either switching to an interpretable architecture or adding a reliable post-hoc explanation layer. Luna: And for buyers — how do you write a good explainability clause without being so technical that you lock out good vendors? Lucas: I'd recommend a two-tier approach. For high-risk decisions — think medical diagnosis, credit, hiring — mandate a specific level of explainability, like global and local explanations with a faithfulness score above a threshold. For lower-risk uses, accept a 'reasonable efforts' standard and an annual audit. The key is to tie the explainability requirement to the risk level of the decision, not just apply it blanket. Luna: That's smart. And it gives vendors a clear path to compliance without making every contract a negotiation nightmare. Lucas: Exactly. And before we go further — I want to mention something quickly. Conversations like this, where we dig into the nuts and bolts of what's actually changing in enterprise software procurement — those are exactly the kind of discussions that keep this show going. And we keep it completely ad-free, which is rare in business podcasting. If these episodes are useful for what you're building or running, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Luna: Yeah, it really does make a difference. Even a small contribution helps us cover research time and production costs, and it means we don't have to run interruptive ads. So if you're able, that link is buy me a coffee dot com slash fexingo. Lucas: Alright, back to it. So we've covered the what and the how of explainability audits. But there's a deeper question: does forcing explainability actually make AI safer, or does it just shift the problem? Luna: I've heard that criticism — that explanations can be gamed. A vendor could produce a plausible-sounding explanation that's technically correct but misleads the buyer about the model's true weaknesses. Lucas: That's a real risk. There's a concept called 'explanation hacking' — deliberately training the model to produce explanations that satisfy the audit but hide problematic behavior. For example, you could have a model that uses a protected attribute like race in a hidden layer but then produces explanations that point to a different, permissible feature. The explanation looks clean, but the model is still biased. Luna: So the audit clause needs to include some kind of adversarial testing, right? Not just checking that an explanation exists, but checking that the explanation is honest. Lucas: Right. Leading-edge contracts now require the vendor to submit to 'counterfactual audits' — where the auditor deliberately tries to find cases where the explanation doesn't match the model's actual decision process. If the auditor can find even one case where changing an irrelevant feature changes the prediction but the explanation doesn't mention it, that's a fail. Luna: That's rigorous. I think we're going to see a whole ecosystem of audit tools and consulting firms emerge around this, similar to how penetration testing became standard in cybersecurity. Lucas: Absolutely. And I think in three years, an AI explainability audit will be as standard as a data privacy review in any enterprise software deal. The vendors that invest now will have a serious competitive advantage. Luna: Well, on that note — it's a good place to wrap up. Thanks for diving into this with me. Lucas: My pleasure. And to our listeners — if you're negotiating a software contract right now, check whether your procurement team has added this clause yet. If they haven't, it might be time to ask why.