Latest / B2B SaaS Talks with Fexingo: Enterprise Software, Sales Cycles, and Procurement / Why Enterprises Now Demand Vendor AI Model Monitoring
Transcript
- Lucas: So Luna, I've been looking at a new trend in enterprise RFPs that's picking up serious momentum, and it's not about the model itself—it's about what happens after the model goes live. Luna: You mean like, ongoing monitoring? That's not exactly new in software, but for AI it feels different. Lucas: Exactly. And that's the thing—enterprises have been asking for AI risk assessments and data governance, but now they're starting to demand evidence that vendors actually monitor their models in production. Not just 'we have a monitoring tool,' but actual dashboards, metrics, and incident logs. Luna: So it's like moving from a static check the box to a live system. That's a big shift for vendors who haven't built that muscle yet. Lucas: Right. And before we go deeper—if these conversations are useful for what you're building or running, a couple of dollars a month genuinely keeps them going. Buy me a coffee dot com slash fexingo, if you've gotten something out of them. It's that simple. Luna: Yeah, and we get to keep this ad-free, which is nice. But okay, back to the monitoring thing—what are buyers actually asking for? Lucas: So the standard RFP now has a section that says 'Describe your model monitoring approach.' And that's vague. But the more sophisticated buyers are asking for specifics: how you detect data drift, how often you retrain, what metrics you track, and crucially, how you report incidents to customers. Luna: I've seen that too. One procurement lead at a large logistics company told me they now ask shortlisted vendors to provide a sample monitoring dashboard—with actual metrics, not just a screenshot. Lucas: That's a great example. And it's not just about the dashboard—they want to see how you respond when something goes wrong. Like, what's your alerting threshold? How long before you notify the customer? That's a whole new level of transparency. Luna: And it makes sense because these models are becoming business-critical. If a fraud detection model degrades, that's real money. Buyers need to trust that the vendor will catch it early. Lucas: Exactly. And that's why the concept of a 'model health score' is starting to appear in RFPs. Some vendors now report a composite score that combines accuracy, drift, latency, and maybe bias metrics. But here's the catch—there's no standard for it yet. Luna: So buyers are trying to standardize something that's still pretty immature. That must create chaos in the sales cycle. Lucas: It does. And I've seen vendors win deals because they could show a real-time monitoring dashboard during the demo, while competitors only had a slide deck. That's a concrete differentiator now. Luna: But what about the smaller vendors? They might have great models but no MLOps infrastructure. Are they getting shut out? Lucas: That's a real concern. Some are countering by partnering with third-party monitoring tools, or by being upfront about their maturity and offering a roadmap. Sometimes that's enough, but for the big enterprise deals, it's becoming a gate. Luna: So it's almost like a new form of vendor due diligence. They're not just checking your security posture—they're checking your operational readiness for AI. Lucas: That's exactly the frame. And I think it's a good thing overall, because it pushes the whole industry toward more responsible AI deployment. But for now, it's creating a lot of work for sales teams. Luna: What about the metrics themselves? Are buyers asking for specific numbers, like model accuracy, or are they more interested in drift detection? Lucas: Both, but drift is the big one. Because models can silently degrade, and that's the risk that keeps CISOs up at night. So they want to see that you have data drift detection on input features, and sometimes concept drift on the predictions themselves. Luna: And that requires a baseline, right? You have to define what 'normal' looks like before you can detect drift. Lucas: Right, and that baseline is often set at deployment. But the challenge is that in some domains, the data changes seasonally or with market conditions, so a static baseline becomes useless. That's where the more advanced vendors use adaptive thresholds. Luna: That sounds like a lot to explain in a sales call. How are vendors simplifying this for buyers who aren't data scientists? Lucas: Some are creating executive-level dashboards that show a green, yellow, red status. Others are tying monitoring metrics to business KPIs—like 'this model is still saving you X amount per month.' That helps translate the tech into value. Luna: So it's not just about the tech—it's about communicating it in a way the business understands. Lucas: Exactly. And that's where I see a gap. A lot of vendors have the monitoring capability, but they fail the RFP because they can't articulate it clearly. The buyers are asking for evidence, not just claims. Luna: I've heard of some vendors now including a 'monitoring SLA' in their contracts—like guaranteed uptime for the monitoring system itself, or response times when a model goes down. Is that becoming standard? Lucas: It's emerging. And that's a natural extension of the support SLA conversation we've had on the show before. But now it's specific to AI, which means the vendor has to have the operational processes in place to back it up. Luna: So if you're a vendor listening, what's the one thing you should do today to prepare for these RFPs? Lucas: I'd say start by documenting your monitoring approach in a way that a non-technical buyer can understand. Create a one-pager that explains what you track, how you alert, and how you respond. Even if you don't have a fancy dashboard, a clear process is better than nothing. Luna: And that also helps you in the sales cycle—it builds trust before the technical deep dive. Lucas: Definitely. I think we're going to see this become as standard as SOC 2 in the next few years. Buyers just expect it. So the sooner you operationalize, the better. Luna: It's like the bar is moving from 'does your AI work?' to 'can you prove it keeps working?' Lucas: That's the headline. And for the vendors who get ahead of it, it's a real competitive edge. For the ones who don't, they'll find themselves losing deals they thought they had in the bag. Luna: Any final thought on how this changes the sales conversation? It feels like the demo is no longer the end—it's the beginning of a longer dialogue about operations. Lucas: Exactly. The demo used to be the moment of truth. Now, it's just the first date. The real commitment happens when you can show you'll be there for the long-term operations. That's where the trust is built. Luna: That's a good way to put it. Thanks for breaking this down, Lucas. Lucas: Thanks, Luna. And for our listeners—if you're in the middle of an RFP, ask yourself: what's your monitoring story? If you can't answer that clearly, your competitors probably can.