Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / Why Cloud Contracts Now Include AI Inference Speed Guarantees
Transcript
- Lucas: If these conversations are useful for what you're building or running, you've probably noticed that the real leverage in cloud negotiation isn't just about price per compute hour anymore. It's about performance guarantees for specific workloads. Luna: And right now, the hottest workload is AI inference. Not training, but actually running the model in production. Lucas: Exactly. We've been talking about GPU clauses for months, but this episode is about a different kind of guarantee that's starting to show up in enterprise cloud contracts: AI inference speed guarantees. Specifically, contractual commitments from the cloud provider on inference latency and throughput. Luna: So instead of just promising you a certain number of GPUs or a certain instance type, they're promising how fast the model will respond? Lucas: Yes. And this is a big shift. Traditionally, cloud SLAs covered uptime — the instance is running. But uptime doesn't tell you whether your AI model is responding in 50 milliseconds or 500 milliseconds. For a real-time application like a chatbot or a fraud detection system, that difference is everything. Luna: I've seen some of these contracts. They're getting very specific. I've heard of one case where a healthcare AI company negotiated a clause guaranteeing that 95 percent of inference requests would complete within 40 milliseconds, measured over a rolling five-minute window. Lucas: That's a very tight window. And it's not just about latency — some contracts also guarantee a minimum throughput, like a certain number of inferences per second. And if the provider misses that, there are financial penalties, typically in the form of service credits. Luna: Right, but unlike the standard uptime credits, which are usually a small percentage of your monthly bill, these inference speed credits can be much larger. I've seen clauses that offer up to 25 percent monthly credit for sustained violations. Lucas: That gets the provider's attention. The interesting thing is which cloud providers are willing to offer these guarantees. AWS, for example, is pushing Inferentia instances for inference, and they're offering latency SLAs on those, but not on general-purpose GPU instances. Azure has similar guarantees for their nd series VMs, but only if you commit to a reserved instance. Luna: And Google Cloud? They've always been strong on TPUs for training, but for inference, they're more flexible. I've seen contracts where they guarantee latency for specific model architectures on their TPU v5e. Lucas: But here's the catch — most of these guarantees are workload-specific. You can't just say 'all our inference workloads need to be under 50 milliseconds.' You have to define the model, the input size, the batch size, even the framework version. The clause becomes very complex. Luna: That complexity is actually the point. The provider needs to know exactly what they're promising, because inference performance varies wildly based on those parameters. If you change the model from a small BERT variant to a large language model, the latency changes by an order of magnitude. Lucas: So the negotiation is really about defining the scope. And that's where a skilled procurement team earns their keep. You want the guarantee to cover your most critical production paths, not just a toy benchmark. Luna: And if today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. We keep the show independent so we can dig into these specifics. Lucas: Yeah, that's what allows us to spend an episode on a single contract clause instead of rushing through surface-level takes. Appreciate everyone who chips in. Luna: Alright, let's get back to the mechanics. One thing I find fascinating is how these inference SLAs interact with the multi-tenant nature of cloud infrastructure. Lucas: Right. Because if you're sharing a GPU or an Inferentia chip with other customers, their workload can affect your latency. The provider has to isolate your inference traffic somehow. Some offer dedicated instances, but that's more expensive. Others use techniques like CPU pinning or network QoS. Luna: I've seen contracts where the provider commits to a specific 'noisy neighbor' mitigation strategy. For example, AWS might guarantee that your inference traffic is routed to a dedicated network path, separate from other tenants on the same host. Lucas: That's a deep technical commitment. And it's becoming more common as AI inference moves from experimental to mission-critical. I talked to a procurement manager at a fintech company that uses AI for real-time fraud detection. They negotiated a clause that ties the inference SLA to their own regulatory compliance requirements. Luna: That's smart. If your fraud detection model has to respond within 100 milliseconds to meet a payment processing standard, you want the cloud provider's guarantee to match that regulatory clock. Lucas: And the penalties in that contract weren't just credits. If the provider missed the latency target for more than four hours in a month, the fintech had the right to terminate the contract without penalty. That's a nuclear option, but it shows how important these guarantees are becoming. Luna: What about smaller companies? Do they have any leverage to negotiate these kinds of clauses? Lucas: It's harder, but not impossible. The key is to start with a specific, well-characterized workload. If you can show the provider that your inference pattern is predictable and that you're willing to commit to a certain volume, they're more likely to offer a custom SLA. Luna: And there are also third-party tools now that monitor inference latency across providers. I've seen startups that offer a dashboard showing real-time latency for your models on AWS, Azure, and GCP, which gives you data to bring to the negotiation. Lucas: That's a great point. Because without independent measurement, how do you prove the provider missed the guarantee? Most cloud providers have their own monitoring, but you want an independent source. Some contracts now specify the measurement methodology — for example, using a third-party agent running on the instance. Luna: So we're moving toward a world where cloud contracts for AI look more like traditional enterprise software contracts, with detailed SLAs, penalties, and termination rights. It's a maturation of the market. Lucas: And it's also a differentiator. The providers that can offer strong inference speed guarantees will win more enterprise AI business. AWS is betting on Inferentia, Google on TPUs, Azure on their nd series. But the guarantee itself is becoming a competitive feature. Luna: I wonder how this will evolve as models get more efficient. If inference becomes cheaper and faster by default, maybe the guarantee becomes less important. But for now, it's a real negotiation point. Lucas: I think it becomes more important, not less. As models become more efficient, the bar will rise. Customers will expect sub-10-millisecond latency for conversational AI. And the contracts will have to reflect that. We're at the beginning of this trend. Luna: I think you're right. And for anyone listening who's about to sign a cloud contract for AI inference, the lesson is: don't just look at the price per inference. Look at the guarantee behind it. Lucas: And make sure the guarantee covers your real workload, not just a benchmark. That's where the value is.