Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / Why Cloud Contracts Now Include AI Inference Guarantees
Transcript
- Lucas: So last week I was looking at a cloud re-up agreement from a Fortune 500 media company, and buried in the service-level exhibits was a clause I hadn't seen six months ago: a guaranteed P99 latency for AI inference, measured in milliseconds, with a service credit if they miss it. Luna: Wait — they're guaranteeing inference latency now? Not just training compute or spot instance availability? Lucas: Exactly. Training guarantees have been around for a while — you commit to a GPU reservation, the provider commits to cluster availability. But inference is fundamentally different. It's real-time. It's spiky. And it's becoming the dominant cost driver in enterprise AI. Luna: I've seen numbers that inference now accounts for something like 65 percent of AI cloud spend. Is that what's pushing this? Lucas: That's exactly it. A year ago, most enterprises were still training models. Now they're putting them into production — content moderation, personalized recommendations, real-time fraud detection. And those workloads can't tolerate a two-second jitter because some other tenant spun up a training job on the same GPU node. Luna: Right — the multi-tenant noise problem. I remember talking to an architect at a fintech who said their inference latency would double during peak training hours on shared clusters. Lucas: That's the exact pain point these new clauses address. The media company I mentioned — they're running a real-time content moderation pipeline. Every video uploaded to their platform gets scanned for policy violations. They need inference to complete within 300 milliseconds, or the user experience degrades and they risk regulatory fines in certain jurisdictions. Luna: So what did the clause actually look like? Was it a flat latency promise, or tiered? Lucas: It was tiered. For inference workloads running on reserved GPU instances, the provider guaranteed a P99 latency of under 250 milliseconds for the specific model architecture they'd certified. If latency exceeded that threshold for more than one percent of requests in any rolling five-minute window, the provider owed a 10 percent service credit on that instance's monthly bill. If it happened three times in a quarter, the credit went to 25 percent. Luna: And how did the provider ensure that? Did they have to carve out dedicated GPU partitions? Lucas: That's the interesting part. They didn't do dedicated hardware — that would defeat the cloud economics. Instead, they used what they called 'inference-optimized scheduling.' The hypervisor prioritizes inference inference workloads over training and batch jobs on the same GPU. And they over-provisioned the cluster by about 20 percent to absorb spikes. Luna: So the provider is essentially buying a buffer to avoid penalties. That extra capacity must show up in the pricing. Lucas: It does. The media company paid a premium of roughly 15 percent above the base reserved-instance price for that inference SLA. Their finance team ran the numbers — the premium was cheaper than buying dedicated GPU hosts, which would have cost about 40 percent more. So it was a net win. Luna: But that premium also reflects the provider's risk. If they over-provision by 20 percent, they're eating that cost if utilization stays low. Lucas: Exactly. And that's why not every workload qualifies. The provider required the media company to submit a model profile — expected request volume, token lengths, batch sizes — and they only guaranteed latency for models they'd profiled and approved. If you swap in a new model architecture without re-certifying, the guarantee is void. Luna: That makes sense from the provider's side. They don't want to guarantee latency on a model that's wildly variable. But it also means the enterprise loses flexibility. You can't just hot-swap a new version of your model without renegotiating. Lucas: That's the trade-off. The media company's cloud architect told me they now run a certification pipeline — every new model version goes through a week-long profiling process before it hits production with the SLA. They've built that into their release cycle. Luna: So it's becoming a operational process, not just a procurement checkbox. Are we seeing these clauses from all three major providers? Lucas: AWS and Azure both have formal inference SLA programs now. GCP is a bit behind — they offer something similar but it's negotiated case by case, not a published offering. All three are responding to the same market pressure: enterprises won't move inference to the cloud if they can't guarantee response times. Luna: I've also heard that some enterprises are using inference guarantees as a bargaining chip in broader contract negotiations. 'Give us inference SLAs on this deal, or we walk.' Lucas: That's happening. A manufacturing company I spoke with — they're rolling out computer vision for quality inspection — they made inference latency guarantees a condition for renewing their three-year enterprise agreement. The provider agreed, but only for a subset of their factories. So they got a phased rollout. Luna: That phased approach seems smart. It lets both sides build confidence before scaling. What about pricing models? Are we seeing per-token guarantees or per-request? Lucas: Mostly per-request, with throughput floor clauses. The media company's contract guaranteed a minimum of 1,000 inference requests per second per GPU instance with that latency SLA. If throughput dropped below that, the provider had to add capacity within four hours or credits kicked in. Luna: So it's not just about speed — it's about consistency under load. That's a much stronger commitment than typical compute SLAs, which usually only cover uptime. Lucas: Far stronger. And it changes how cloud architects design their systems. Instead of building in redundancy to handle latency spikes, they can rely on the provider's guarantee — at least for the certified models. Some are even moving away from multi-region inference failover because the single-region SLA is good enough. Luna: That's a big operational savings. But it also concentrates risk. If the provider has a regional issue, you're stuck. Lucas: True. The media company kept a backup region with a simpler spot-instance pool for non-critical inference. But their primary pipeline runs on the guaranteed cluster. And the finance team calculated that the inference guarantee premium saved them about 30 percent overall compared to running their own on-premise GPU infrastructure — even after accounting for the 15 percent premium. Luna: That's a compelling number. And I imagine it's only going to become more common as inference workloads grow. What do you see happening over the next year? Lucas: I think we'll see inference SLAs become a standard line item in enterprise cloud contracts within 18 months. The providers will compete on granularity — per-model, per-region, even per-time-of-day guarantees. And pricing will become more transparent, like a published premium schedule. Luna: That would be a big shift from today's negotiated deals. It would really commoditize inference as a cloud service. Lucas: And that's ultimately what enterprises want. They want inference to be as predictable as virtual machine uptime. The clause we saw in that media company's contract is just the first iteration. Luna: It's also worth noting — and I know this is a bit of a meta point — but the fact that we can even have this conversation shows how fast the cloud is adapting to AI workloads. A year ago, inference guarantees weren't on anyone's radar. Lucas: Totally. And speaking of that — the reason we can dig into these details every week is because this podcast is listener-supported. A number of you chip in a few bucks a month over at buy me a coffee dot com slash fexingo, and that's what lets us keep the episodes ad-free and focused on the real contract language, the real numbers. Luna: Yeah, it genuinely makes a difference. We don't have to chase sponsors or soften the analysis. Just straight talk about what's actually happening in cloud contracts. Lucas: Exactly. And if these conversations help you in your own negotiations or architecture decisions, that's the whole point. So — back to inference. One thing I didn't mention: the media company's contract also included a renewal option after 12 months where they can renegotiate the latency target based on actual performance data. That's the kind of flexibility that makes these clauses work long-term. Luna: That's smart. It gives both sides an incentive to optimize. The provider wants to hit the target to avoid credits, and the enterprise gets a baseline they can improve upon. Lucas: Right. And as models get more efficient — smaller architectures, better quantization — the same hardware can deliver lower latency. So the next negotiation might target 150 milliseconds instead of 250. That's real progress. Luna: It's also a reminder that cloud contracts are becoming as much about AI operations as about infrastructure. The line between compute and model performance is blurring. Lucas: For sure. And that's exactly the kind of shift we'll keep tracking. Next episode, we'll look at how these inference guarantees interact with data residency requirements — because if your model needs to stay in a specific region, the SLA options shrink. Luna: That's a good one. See you next time.