Latest / B2B SaaS Talks with Fexingo: Enterprise Software, Sales Cycles, and Procurement / Why Enterprise Buyers Now Demand a Vendor AI Model Card
Transcript
- Lucas: You've probably seen a SOC 2 report. Maybe a penetration test summary. But there's a new document that's starting to appear in enterprise software RFPs, and it's not about security—it's about the AI itself. Luna: You're talking about the AI Model Card, right? I've seen a few references, but I haven't actually seen one in the wild. Lucas: I have. A client of mine—a mid-market SaaS company selling to a Fortune 500 manufacturer—got a request for what they called a 'model card' as part of the security questionnaire. It wasn't even in the original RFP; it was added during the negotiation phase. Luna: So this is a demand that's coming from the buyer side, not a voluntary disclosure. What exactly is in a model card? Lucas: Think of it as a one-page nutrition label for a machine learning model. It typically includes: model architecture, training data provenance, intended use cases, known limitations, bias testing results, and performance metrics across different sub-populations. The concept was popularised by Margaret Mitchell and Timnit Gebru back in 2019 when they were at Google. Luna: Right, the 'Model Cards for Model Reporting' paper. I remember that. It was more of a proposal for ethical AI documentation. But now it's becoming a procurement checkbox? Lucas: Exactly. And the trigger seems to be a few high-profile incidents. One was a healthcare AI that had significantly lower accuracy for minority populations—the hospital that deployed it faced a lawsuit. Another was a hiring tool that discriminated against women. Buyers are now saying, 'We don't want to be the ones discovering these problems in production.' Luna: So they're shifting the liability upstream to the vendor. Show us your model card, or we assume the worst. Lucas: That's the logic. And it's not just big banks or pharma. I've seen RFPs from mid-market retailers and logistics companies that now include a requirement for model cards. The language is borrowed from NVIDIA's NeMo framework and Hugging Face's standardized cards. Luna: Let's talk about what a good model card looks like. Can you give me an example from a well-known vendor? Lucas: NVIDIA publishes model cards for their NeMo models. One card I looked at recently was for their 'Canary' model—a multilingual speech recognition model. The card lists the model architecture—it's a 1.2 billion parameter transformer—the training data, which is a mix of public and proprietary datasets, the languages it supports, and a section on biases: it notes that English and Spanish accuracy is high, but performance drops for tonal languages like Vietnamese. Luna: So they're transparent about the limitations. That actually builds trust, doesn't it? It sets expectations. Lucas: It does. But it also creates a liability. If you say your model doesn't work well for Vietnamese, and someone deploys it for Vietnamese speakers anyway and it fails, whose fault is it? The card is a disclaimer, but it's also an admission. Luna: Interesting. So from a vendor's perspective, there's a tension: disclose enough to satisfy the buyer, but not so much that you open yourself up to litigation. Lucas: That's the balance. And it's a new skill for many B2B SaaS companies. Most startups have never written a model card. They have a trained model, they have an API, but they don't have documentation that a procurement officer can review and understand. Luna: And the procurement officer might not be technical. They need something they can show to legal and compliance and say, 'See, they've tested for bias, they've listed the training data.' It's a governance artifact. Lucas: Right. And that's why I think model cards are going to become as standard as SOC 2 reports in the next few years. The cost of creating one is modest—maybe two weeks of an ML engineer's time—but the cost of not having one is starting to become losing a deal. Luna: Speaking of the cost of creating one, I'm reminded that we keep this show ad-free thanks to listeners who find value in these deep dives. If today's conversation saved you a couple hours of research or gave you a concrete angle for your next procurement meeting, consider supporting the show at buy me a coffee dot com slash fexingo. It's the smallest ask, and it directly funds more episodes like this. Lucas: Honestly, it's a good way to tell us what topics you want more of. We see every donation and it shapes our editorial calendar. No pressure, but if it felt worth it, that's the link. Luna: Alright, back to model cards. What about startups that are building proprietary models and don't want to reveal their secret sauce? A model card forces you to disclose architecture and training data—that could be competitive intelligence. Lucas: It's a valid concern. But the model card doesn't have to include proprietary details like exact hyperparameters or the specific data sources if they're trade secrets. It can say 'trained on a curated corpus of financial documents' without naming the documents. The key is to provide enough information for a buyer to assess risk, not to reverse-engineer your model. Luna: So there's room for negotiation. But if a buyer insists on more detail, what then? Lucas: That's where the non-disclosure agreement comes in. Some vendors are willing to share a more detailed model card under NDA. Others are drawing a line and saying, 'We'll share our SOC 2, our pen test results, and our model card. That's the package.' And buyers are accepting that. Luna: I've also heard that some VCs are now asking for model cards during due diligence for AI startups. Is that a trend? Lucas: Absolutely. I've spoken to a couple of venture partners who said that for a Series A, they want to see a model card as part of the technical due diligence. It's a signal that the team is disciplined about understanding their model's behavior. If a startup can't articulate what their model does and doesn't do well, it's a red flag. Luna: So model cards are becoming a hygiene factor, not a differentiator. You don't win deals because of them, but you lose deals without them. Lucas: Exactly. And it's happening faster than I expected. I think two factors are accelerating it: first, the regulatory environment in the EU with the AI Act, which requires documentation for high-risk systems; second, the insurance market. Some cyber insurers are starting to ask about model cards when writing policies for companies that use AI in critical decisions. Luna: So it's not just procurement—it's also insurance and compliance. That creates a multiplier effect. Lucas: Right. Once you have the card for one purpose, you can reuse it for others. So the marginal cost is low once you've done it the first time. But the first time is painful because you need to actually test your model across sub-populations, measure performance, and document limitations. Luna: And that testing might reveal uncomfortable truths about your model. What if you find a significant bias that you hadn't known about? Lucas: That's the risk. But most vendors I've worked with say it's better to discover it internally than to have a customer discover it in production. You can fix it or document it as a known limitation. If you document it, you're transparent. If you ignore it, you're liable. Luna: I want to circle back to the NVIDIA example. They put model cards out publicly for their enterprise models. What's the incentive for a vendor that's not required by regulation to do this voluntarily? Lucas: Brand differentiation. If you're a vendor selling to risk-averse buyers—think healthcare, finance, government—having a public model card signals that you take responsible AI seriously. It shortens the sales cycle because the buyer doesn't have to ask for it. It's pre-emptive. Luna: Do you think model cards will eventually be standardised by an industry body? Like, everyone uses the same template? Lucas: There are efforts. The ML Commons group is working on a standard for model cards. Hugging Face already has a widely used template. But I think we'll see fragmentation for a while, similar to how SOC 2 reports are structured around the same five trust principles but vary by auditor. The core elements will be the same, but the level of detail will differ. Luna: Let's talk about the cost in more detail. You said two weeks of an ML engineer's time. Is that enough for a complex model? Lucas: For a simple classification model, yes. For a large language model with billions of parameters, you're looking at more like a month. You need to run evaluation across diverse test sets, measure performance for different demographic groups, and document known adversarial vulnerabilities. And you need a product manager or technical writer to make it readable for non-technical stakeholders. Luna: So it's not trivial. But compared to the cost of a lawsuit or a failed deployment, it's cheap. Lucas: Exactly. And that's the argument I'm making to startups: think of it as insurance, not overhead. One of my clients had a prospect who was evaluating two vendors. One had a model card; the other didn't. The buyer told me they went with the one that had the card because it saved them weeks of internal due diligence. The cost of creating the card was covered by the first deal. Luna: So it's actually an roi positive investment. I can see why VCs are starting to require it. It's a sign of maturity. Lucas: It is. And I think we're at an inflection point. Six months ago, maybe one in ten RFPs asked about model cards. Now, I'm seeing it in three out of ten for deals over a million dollars. And the trend is accelerating. Luna: What about open-source models? If a vendor is using Llama or Mistral, do they need a model card for their fine-tuned version? Lucas: Yes, because the fine-tuned version is a new model with different behavior. The base model card from Meta or Mistral is a starting point, but you need to document what you changed and how it performs on your specific use case. Buyers want to know if your fine-tuning introduced any biases or degraded performance in certain scenarios. Luna: So there's no way around it. Even if you're building on someone else's work, you still have to do your own documentation. Lucas: Right. And the good news is that tools are emerging to automate parts of this. There are libraries that can generate model cards from training logs and evaluation results. Hugging Face's AutoTrain can produce a basic card automatically. But the human review is still critical—especially for the bias and limitation sections. Luna: I want to ask about enforcement. What happens if a vendor provides a model card that's inaccurate or incomplete? Lucas: That's the big question. In the RFPs I've seen, the model card is included as a contractual exhibit. If it's inaccurate, it's a breach of contract. And if a buyer suffers damages because the model didn't perform as documented, they can sue. So there's real legal teeth behind it. Luna: That changes the incentive. Suddenly, the model card isn't just a marketing document—it's a legally binding representation. Lucas: Exactly. And that's why some vendors are pushing back on the level of detail. They want to include disclaimers like 'performance measured on internal test sets' and 'results may vary in production.' But buyers are sophisticated—they know that. They're looking for good-faith effort, not guarantees. Luna: So the model card is a negotiation point. Both sides have to agree on what's disclosed and what's not. Lucas: Right. And I think the market will settle on a standard set of fields that everyone expects. Just like SOC 2 reports always cover security, availability, processing integrity, confidentiality, and privacy—I think model cards will converge around architecture, training data, performance, bias, and limitations. That's what we're already seeing. Luna: Lucas, for a B2B SaaS founder listening to this—what's the one action they should take this week? Lucas: If you have a product that uses any machine learning model, start drafting a model card today. Use the Hugging Face template as a starting point. Run your model on a few test sets. Document what it does well and where it fails. Even if you never share it publicly, having it ready will shorten your sales cycle when a buyer asks for it. And they will ask—sooner or later. Luna: And if you're a buyer, start adding model cards to your RFI template. It's a simple ask that can save you a lot of downstream trouble. Lucas: Agreed. The tools are there, the precedent is set, and the market is moving. It's one of those rare moments where a best practice is becoming a standard practice in real time.