Latest / B2B SaaS Talks with Fexingo: Enterprise Software, Sales Cycles, and Procurement / Why Enterprise SaaS Buyers Now Demand Algorithmic Transparency
Transcript
- Lucas: The most dangerous thing in an enterprise software contract today isn't the price per seat, and it's not even the data residency clause. It's the algorithm itself. If you're selling any kind of predictive model, any scoring engine, or any automation tool to a Fortune five hundred company, you no longer get to say 'it's a trade secret.' You have to show them how it thinks. Luna: That feels like a massive shift from just three years ago. Wasn't the default assumption that buyers just wanted the outcome, regardless of the method? Lucas: Exactly. The old playbook was: give us the dashboard, prove the ROI, and let our engineers figure out the backend later. But now, when I talk to CIOs and Chief Risk Officers, they're asking for the weights. They want to know if a loan approval algorithm is penalizing certain zip codes, or if a lead-scoring model is deprioritizing candidates based on non-business criteria. They treat opacity as a liability, not a feature. Luna: So we're moving from trust-based vendor relationships to evidence-based audits. That changes the sales cycle entirely, doesn't it? You can't just demo the shiny result anymore. Lucas: It adds months to the deal cycle. And honestly, it filters out a lot of vaporware. I was looking at a case last week involving a major global logistics firm. They were weeks away from signing a nine-figure contract with a routing optimization vendor. Then during the technical due diligence, the buyer asked for the decision-tree documentation. The vendor couldn't provide it because their model was a deep learning neural net that had evolved beyond human-readable rules. Luna: Did the deal collapse because of that, or did they find a workaround? Lucas: They walked away. The buyer's legal team flagged it as an uninsurable risk. If the algorithm makes a mistake that causes a supply chain disruption, who is liable? The vendor argued it was an emergent property of the training data. The buyer said that wasn't good enough for insurance underwriting. So the contract died over something as abstract as interpretability. Luna: That is a staggering amount of capital destroyed over a philosophical disagreement about code visibility. It really highlights how high stakes these procurement decisions have become. Lucas: It does. And before we dive deeper into the mechanics of what vendors are actually being asked to disclose, I want to pause for a second. If these conversations are useful for what you're building or running, consider supporting the show directly. A couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo. It helps us keep the lights on and brings in more guests who are actually in the trenches of these negotiations. Thanks for keeping us independent. Luna: Yeah, and it ensures we can keep digging into these unglamorous but critical parts of business without answering to advertisers. Back to the logistics case—was the vendor completely unable to explain their model, or was it just a matter of terminology? Lucas: It was both. The vendor used terms like 'proprietary heuristics' and 'dynamic weighting,' which sounded vague to the procurement team. The reality was probably simpler: they had a black-box model that optimized for speed, but the buyer needed to know if that speed came at the cost of fuel efficiency or driver safety metrics. The gap wasn't just technical; it was semantic. The vendor spoke engineer, the buyer spoke risk manager. Luna: So part of the challenge is translation. Vendors need to learn how to articulate their algorithmic logic in a way that satisfies legal and compliance teams, not just engineering teams. Lucas: Precisely. And this is where the concept of 'explainable AI,' or XAI, moves from a marketing buzzword to a contractual obligation. We're seeing new clauses in RFPs that demand local interpretability. That means for every individual decision the software makes—say, rejecting a credit application—the vendor must be able to generate a reason code that a human can understand within seconds. Not a summary of the whole model, but an explanation for that specific transaction. Luna: That's a much higher bar than just saying 'the model is accurate.' Accuracy is a global metric; explainability is a local one. You can have a highly accurate model that is impossible to explain for edge cases. Lucas: Right. And those edge cases are where the lawsuits happen. If your model denies coverage to one person in a million, and that denial turns out to be discriminatory or factually wrong, you need to be able to trace back exactly why. If you can't, you're sitting on a ticking time bomb. I've seen several fintech firms now hiring 'algorithmic auditors' specifically to stress-test their models against these explainability standards before they even go to market. Luna: It sounds like the role of the product manager is shifting too. They can't just build the best predictor anymore; they have to build the most transparent one. That's a huge constraint on innovation. Lucas: It constrains the type of innovation. We're likely going to see a bifurcation in the SaaS market. On one side, you'll have simple, rule-based systems that are fully transparent but less powerful. On the other, you'll have complex AI systems that require extensive third-party certification to prove their fairness. The middle ground is disappearing. Vendors can't just hide behind complexity anymore. Luna: Do you think smaller SaaS startups can compete in this environment? Or does this effectively gatekeep enterprise sales for well-funded incumbents who can afford these compliance departments? Lucas: That's the real risk. Building an explainability layer takes engineering time and legal expertise. A two-person startup might have a better algorithm than a legacy provider, but if they can't produce the audit trail the buyer needs, they lose the deal. We're already seeing enterprise buyers create 'trusted partner' lists that only include vendors with established governance frameworks. It raises the floor for entry. Luna: So the moat isn't just the technology anymore; it's the bureaucracy around the technology. That's interesting. It reminds me of how pharmaceutical companies operate—you can't just sell the drug; you have to sell the clinical trial data and the manufacturing process details. Lucas: That's a perfect analogy. Software is becoming regulated like a physical good. You're not just buying the output; you're buying the integrity of the production line. And that changes how vendors price their services. Some are starting to charge premiums for 'certified explainability modules.' It's basically an upsell on trust. Luna: I can see buyers being skeptical of that premium. If explainability is a baseline requirement, charging extra for it might look like nickel and diming. How do vendors justify that cost? Lucas: They frame it as risk reduction. The argument is that the cost of a single regulatory fine or a class-action lawsuit outweighs the premium. It's an insurance policy baked into the license fee. But it's tricky because not all buyers have the same risk tolerance. A bank will pay for full transparency; a mid-sized marketing agency might just want the clicks and not care how the AI picks them. Luna: So maybe we won't see a universal standard immediately. Perhaps it will remain segmented by industry verticals. Healthcare and finance will demand high transparency, while creative industries will remain low-regulation zones. Lucas: Very likely. But even in creative industries, there's pressure coming from copyright holders. If an AI generates content, the buyer wants to know which training data was used. That's another form of transparency—provenance. So the definition of 'how it thinks' is expanding to include 'what it consumed.' Luna: Provenance tracking. That's going to be a nightmare for data pipelines. Are vendors currently equipped to track that level of detail, or is that still theoretical? Lucas: Most are still theoretical. There are emerging tools for data lineage tracking, but integrating them into existing machine learning workflows is messy. I spoke with a data scientist at a media company last week who described their current process as 'manual reconciliation.' They literally have spreadsheets mapping which datasets went into which models. It doesn't scale. Luna: Manual reconciliation for enterprise-scale AI. That sounds unsustainable. The market is going to need better tooling for this very soon. Lucas: Absolutely. And this creates a new category of vendors: the infrastructure providers for transparency. Companies that don't sell AI outputs, but sell the monitoring and logging layers that make AI outputs auditable. We're going to see a boom in this space. It's boring, unsexy work, but it's essential plumbing for the next decade of enterprise software. Luna: Plumbing that everyone forgets until the pipe bursts. It's funny how the most visible problems in tech often come from the least visible foundations. Lucas: Exactly. And for the buyer, the lesson is clear: stop negotiating on features alone. Start negotiating on fidelity. Ask for the model cards. Ask for the bias tests. Ask for the rollback procedures. If a vendor hesitates, that's your answer. Luna: A practical checklist for the listeners: demand model cards, request bias testing results, and insist on rollback protocols before signing any contract that involves automated decision-making. Lucas: Spot on. Because in the end, transparency isn't about giving up secrets. It's about building confidence. And in a world of increasingly autonomous systems, confidence is the most valuable currency you can have. Luna: It's also the only thing that allows you to sleep at night when the algorithm makes a mistake at three in the morning. Lucas: True. So the question for anyone reviewing their vendor portfolio this quarter is: do you actually know how your tools decide, or are you just hoping they decide right? Luna: Hopefully, by now, most of you are asking the first question. Thanks for listening.