Latest / B2B SaaS Talks with Fexingo: Enterprise Software, Sales Cycles, and Procurement / Why Enterprise Software Deals Now Include an AI Governance Framework
Transcript
- Lucas: Alright, Luna — I want to talk about something that I'm seeing crop up in more and more enterprise software deals lately, and it's not a pricing clause or a data portability audit. It's an AI governance framework. Luna: Yeah, I've been hearing that term a lot in procurement circles. But what does it actually mean in practice? Is it just a document, or is it something more operational? Lucas: So, I talked to a VP of procurement at a large healthcare system last week. They were in the middle of a seven-figure deal with a vendor that sells a clinical decision support tool — think AI that helps doctors flag potential drug interactions. Luna: Right, and I'm guessing the conversation went beyond price and uptime SLAs. Lucas: Exactly. The buyer asked for something called a 'model card' — basically a standardized document that describes how the AI model was trained, what data it was trained on, what its known limitations are, and how it was tested. The vendor didn't have one ready. The deal slowed down by almost three months. Luna: Three months? That's huge. And I imagine that's just one piece of the framework. What else are buyers asking for? Lucas: The framework typically includes several components. First, a model card or a set of model cards for each AI feature. Second, a bias audit — especially for models that might impact patient outcomes or hiring decisions. Third, a human-in-the-loop protocol — meaning a documented process for when a human must review or override the AI's output. Luna: And I bet the fourth piece is around data governance — like where does the AI training data come from, and is it compliant with privacy regulations? Lucas: Exactly right. And then there's a fifth piece that's gaining traction: an AI decision log. That's a record of every decision the AI made, what inputs it used, and why it gave that output. This is especially important in regulated industries like healthcare and finance. Luna: So the buyer is essentially asking for an audit trail. That makes sense from a risk management perspective. But I'm curious — is this coming from legal, or from the CFO? Lucas: Great question. In the deals I'm tracking, it's actually a coalition. The CFO is driving it because they're worried about contingent liabilities — if the AI makes a mistake, who's on the hook? But it's also the legal team, because of emerging regulations like the EU AI Act and some US state laws. Luna: The EU AI Act is the elephant in the room, right. That's going to require high-risk AI systems to have these kinds of governance documents. So even if you're not selling into Europe, if your buyer has a European subsidiary, they might demand it. Lucas: Exactly. And there's a ripple effect. A large us based financial services firm I spoke to said they now require an AI governance framework from every software vendor that uses machine learning in their product — period. Not because they're legally required to yet, but because they want to be ready. Luna: So for a B2B SaaS vendor, this is becoming table stakes. What happens if you don't have it? Do you just lose the deal? Lucas: In some cases, yes. In the healthcare deal I mentioned, the vendor eventually delivered a model card, but they lost a competitor's RFP because they couldn't provide a bias audit in time. The competitor had a framework ready, and they won the deal. Luna: That's a concrete cost. So what should vendors be doing right now? I mean, building a framework from scratch sounds expensive and time-consuming. Lucas: The vendors that are ahead of this are treating it as a product feature, not a compliance burden. They're creating standardized templates that can be reused across customers. Some are even building automated tools that generate model cards from their training pipelines. Luna: And I imagine they're also training their sales teams to talk about this proactively — not waiting for the procurement team to ask. Lucas: Exactly. I've seen some vendors include a one-pager on AI governance in their initial proposal. It sets them apart and signals to the buyer that they understand the risk landscape. It's a differentiator. Luna: It's interesting that this is moving from a 'nice to have' to a 'must have' in just the last year or so. Do you think it will become as standard as SOC 2 reports? Lucas: I think within two to three years, an AI governance framework will be as common as a SOC 2 Type II report in enterprise deals. The regulatory pressure is only increasing. And honestly, buyers are getting more sophisticated about AI risk. Luna: Yeah, this is where I think the Fexingo Business podcast network audience — operators, founders, sales leaders — they need to be thinking about this now. It's not a future problem. Lucas: Right. And on that note — and I mean this genuinely — a handful of listeners already chip in monthly through buy me a coffee dot com slash fexingo, and that support is literally what keeps this show ad-free and independent. So if these B2B SaaS deep dives are useful for what you're building or running, that's a simple way to keep them coming. Luna: Yeah, it really is. And we appreciate it. But back to the governance framework — I want to dig into one more thing: how do you actually get started if you're a vendor and you don't have one yet? Lucas: Start with your highest-risk model — the one that has the most impact on customer outcomes or regulatory exposure. Build a model card for that. Then a bias audit. Then document your human-in-the-loop process. Do it iteratively. Don't try to boil the ocean. Luna: And I think the key is to involve legal and product from day one. This can't be just a sales enablement play. It has to be embedded in how you build software. Lucas: Absolutely. The vendors that treat this as a core part of their product development will have a real advantage. The ones that wait until a buyer asks will be scrambling. Luna: Agreed. And to close us out — what's one concrete takeaway? If someone is listening and they're in a B2B SaaS company, what should they do this week? Lucas: This week: identify your three most important AI features. For each one, write down what data it uses, what it does, and what could go wrong. That's the beginning of a model card. Then schedule a meeting with legal to talk about whether you need a broader framework. Just start. Luna: Simple, concrete, actionable. Good place to leave it. Thanks Lucas. Lucas: Thanks Luna. See you next time.