Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / Why Cloud Contracts Now Include GPU Performance Benchmarks
Transcript
- Lucas: So you're an AI startup, you just signed a multi-year cloud deal for GPU compute — maybe fifty A100s, maybe a hundred H100s — and your contract says the provider will deliver 'best effort' performance. That's basically a handshake. Luna: Right. And if your training job runs thirty percent slower than you expected, you have no recourse — unless you negotiated benchmarks. Lucas: Exactly. And that's what's changing. Over the past nine months, I've seen a real uptick in cloud contracts — AWS, Azure, GCP — that include explicit GPU performance benchmarks. Not just uptime SLAs, but actual compute throughput guarantees. Luna: Give me a concrete example. What does a benchmark guarantee look like in a contract? Lucas: Sure. One contract I reviewed — a midsize AI company that does generative AI for legal documents — they were renting a cluster of 64 NVIDIA A100s on Azure. They negotiated a clause that says, quote, 'Azure will achieve a minimum of 85 percent of the theoretical peak TFLOPS for FP16 training workloads, measured over any rolling 24-hour period.' Luna: That's specific. And the theoretical peak — that's basically the manufacturer's advertised number, right? Lucas: Yes. NVIDIA says an A100 can do 312 teraflops for FP16 with sparsity. But in practice, real workloads rarely hit that because of memory bandwidth, data pipeline bottlenecks, PCIe overhead. So 85 percent of peak is actually quite aggressive. Luna: Did the provider push back? Lucas: Initially, yes. The cloud vendors have long argued that GPU performance depends on workload characteristics, so a blanket benchmark is unfair. But the startup came back with data from their own test runs — they had benchmarked the same workload on-premises and on all three clouds. They showed that on GCP, they consistently got 82 to 87 percent of peak, on AWS it was 78 to 84, and on Azure it was 75 to 81. Luna: So they used competitive data to push Azure to match the best-case scenario. Lucas: Exactly. And Azure eventually agreed to an 80 percent guarantee for the first year, stepping up to 85 percent in year two after they expected to have newer infrastructure. The key was that the benchmark wasn't just a single number — it was tied to a specific MLPerf training benchmark, which is a standardized industry test. Luna: MLPerf — that's the benchmark suite from MLCommons, right? Lucas: Right. The contract referenced MLPerf Training v3.1 for their model size and batch size. That gave both parties an agreed-upon yardstick. Without that, you'd be arguing about what 'good performance' means. Luna: So the benchmark itself becomes part of the contract. That's a big shift from even two years ago. Lucas: It is. And I think it signals a maturation of the cloud GPU market. When GPUs were scarce — like during the 2023 crunch — customers had no leverage. They took whatever instance type was available and paid the premium. Now that supply has loosened up, customers are getting more sophisticated. Luna: If these conversations are useful for what you're building or running — if they help you think about your own cloud strategy — that's exactly why we keep the show ad-free. It's listener-supported, and honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Lucas: Yeah, that's the only reason we can do deep dives like this without a sponsor break. So if you've gotten value from the show, consider it. Luna: Okay, back to benchmarks. You mentioned that the startup got a performance guarantee — but what happens if Azure fails to meet it? Lucas: Good question. The typical remedy is service credits, similar to downtime SLAs. In this case, if the cluster drops below 80 percent for more than two consecutive measurement periods, the customer gets a 10 percent credit on their GPU compute bill for that month. But there's a cap — usually the total credits can't exceed 50 percent of monthly spend. Luna: So it's not a free month, but it's meaningful. And it puts pressure on the cloud provider to actually optimize their stack. Lucas: Exactly. And the providers are responding. AWS recently announced a new instance family, the P5e, which uses NVIDIA H100s with what they call 'performance-optimized networking' — and they're offering a throughput SLA for certain workloads. Luna: So they're using performance guarantees as a competitive differentiator now. Lucas: Yes. And it's not just about raw FLOPS. Some contracts now include latency benchmarks — for instance, guaranteeing that the 95th percentile latency for inference requests stays under a certain threshold. That's huge for real-time AI applications like fraud detection or chatbots. Luna: That seems even harder to guarantee than throughput, because latency depends on the network, the model architecture, the request size. Lucas: It is. But one financial services firm I talked to negotiated exactly that for their fraud detection models deployed on GCP. They required that at least 99 percent of inference requests complete within 50 milliseconds over any five-minute window. GCP agreed — but only for a specific model size and batch size, and only if the customer used their recommended VM configuration. Luna: So the guarantee comes with strings attached. The customer has to follow best practices. Lucas: Right. And that's actually fair. Because if you configure your instances wrong — say, you don't enable GPU direct RDMA, or you don't use the right CUDA version — you can't blame the cloud provider for poor performance. Luna: So the contract becomes a two-way commitment. The provider promises a certain performance level, and the customer promises to use the infrastructure correctly. Lucas: Exactly. And that's a healthier dynamic than the old model where the customer just hoped the GPU would work fast enough. Luna: What about smaller companies that don't have a dedicated cloud negotiator? How can they get these kinds of clauses? Lucas: It's harder, but not impossible. A few practical steps: first, run your own benchmarks on the cloud provider's free trial or with a small pilot. Second, use an open-source benchmark like MLPerf or a custom one using your own model. Third, ask for a specific performance commitment in the contract — even if it's just 'consistent with published benchmarks' — most providers will at least put something in writing. Luna: And if the provider says 'we don't do performance guarantees for GPU instances'? Lucas: Then you know you're on a commodity tier. And maybe that's fine if your workload isn't latency-sensitive. But if performance matters, you might need to look at a different provider or a different instance type. Some of the smaller GPU cloud providers — CoreWeave, Lambda Labs — are actually more flexible on this because they want to win business. Luna: So the hyperscalers are being pushed by both customers and nimble competitors. Lucas: Exactly. And I think we'll see this become standard within the next year. If you're signing a cloud contract for GPU compute in 2026, and there's no performance benchmark in it, you're leaving money on the table. Luna: And maybe leaving performance on the table too. Lucas: Exactly. Because in the end, a cloud GPU is only as good as the performance you actually get from it.