Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / Why Cloud Bills Surge When GPUs Go Idle
Transcript
- Lucas: So here's a number I keep coming back to. A single NVIDIA H100 GPU, running in the cloud, costs somewhere between thirty and fifty dollars an hour. That's per GPU. And most enterprises using them for AI workloads are running clusters of hundreds or even thousands. Luna: Right, and I've seen reports that those GPUs sit idle something like forty to sixty percent of the time. That's millions in compute you're paying for but not using. Lucas: Exactly. And this isn't a niche problem. This is the single biggest line-item shock I'm hearing from CFOs and CTOs who moved into AI over the past eighteen months. They budgeted for the sticker price of GPU instances, but they didn't budget for the waste. Luna: Before we go deeper — and this is genuinely tied to what we're talking about — if these conversations about real cloud costs are useful for what you're building or running, listener support is what keeps the show ad-free and focused on the numbers that matter. You can find us at buy me a coffee dot com slash fexingo. Just a quick mention, and we're back to those idle GPUs. Lucas: Yeah, and it's a small thing that makes a big difference. So, where does that waste come from? Let me walk through the anatomy of an idle GPU. Lucas: First, there's provisioning delay. A team requests GPU nodes, the cloud spins them up — that takes minutes. But the data pipeline isn't ready. The model hasn't been compiled. So those GPUs sit there, burning money, waiting for the rest of the stack to catch up. Luna: I've seen this firsthand. A startup I spoke with provisioned fifty H100s for a fine-tuning job. The data prep took six hours longer than expected. That's over fifteen thousand dollars in idle compute before a single training step ran. Lucas: Right, and that's a specific case. Let me give you a broader one. A mid-sized AI company — I'll keep them anonymous — ran three months on AWS with a mix of reserved and on-demand H100s. Their total GPU spend was about one point two million dollars. Of that, roughly four hundred and twenty thousand went to idle time. Luna: That's thirty-five percent waste. And that's not unusual. I've seen data from a cloud cost management firm that tracks utilization across hundreds of accounts. Average GPU utilization across AI workloads is around thirty to forty percent. Lucas: And the thing is, this waste is invisible on the invoice. You see a line item for 'EC2 P5 instances' or 'GCP A3 instances' — it just says the total hours. It doesn't say 'active hours' versus 'idle hours.' So the CFO sees a number and assumes that's the cost of doing AI. Luna: So how do you even start measuring the waste? Because the cloud providers don't surface it easily. Lucas: That's the first problem. AWS has something called 'Compute Optimizer' that gives utilization metrics, but it's not gpu aware in the same granular way. Third-party tools like Vantage or CloudHealth can pull GPU metrics from the instance's own monitoring, but you have to set it up. Most teams don't. Luna: And even if you measure it, what's the fix? You can't just turn off a GPU mid-training. It ruins the checkpoint. Lucas: True, but the waste isn't during training runs — it's between them. It's the idle time when the cluster is spinning but no job is running. That's where you need automation. Kubernetes with node autoscaling can scale down GPU nodes when no pods are requesting them. But that requires the team to containerize their workloads properly. Luna: Right, and a lot of AI teams still run on bare metal or pass-through GPU access. They don't use Kubernetes because they think it adds latency. But the cost of not using it is higher, in many cases. Lucas: Exactly. Another lever is spot instances. Preemptible GPUs are often seventy to eighty percent cheaper. But they can be terminated with short notice, so you need fault-tolerant training that can checkpoint and resume. That's a code change, not an infrastructure change. Luna: And there's a newer option that's getting attention: serverless GPU. AWS Bedrock, Google's Colab Enterprise, and even Azure's managed inference services let you pay per inference or per token, not per gpu hour. That eliminates idle waste entirely. Lucas: But it comes with a trade-off. Those services are more expensive per inference than running your own cluster at high utilization. So if you have steady, high-volume traffic, owning the GPU is cheaper. The serverless path is for spiky or unpredictable workloads. Luna: So it's not a one-size-fits-all. What about the cultural side? I've talked to teams where the engineers treat GPUs as a 'right' — they reserve a cluster and leave it running 'just in case.' Lucas: That's a huge factor. The mindset shift needs to be: GPU time is a perishable resource, like an airline seat. Once the hour passes, you can't get that money back. I've seen companies implement internal chargebacks — each team gets a GPU budget, and unused time is tracked and reported. Luna: And that drives behavior. One company I know publishes a weekly 'GPU utilization leaderboard' — not to shame, but to show which teams are running efficiently. It cut their idle time by almost half in three months. Lucas: That's a great example. Now, let's look at what the cloud providers are doing. AWS just launched 'Capacity Blocks for ML' — you reserve GPU capacity for a specific future window, like a reservation, but you only pay for the time you specify. That's a direct response to the over-provisioning problem. Luna: And Google Cloud has 'A3 Mega' instances with NVIDIA H100s, but they also push their 'Colab Enterprise' as a serverless option. Microsoft Azure has the 'ND H100 v5' series, and they're pushing 'Azure Machine Learning' managed compute that can auto-scale. Lucas: The interesting thing is, all three are experimenting with 'gpu as a service' models that essentially let you pay for compute without committing to a specific number of GPUs. But the pricing is opaque — you have to contact sales for many of these. Luna: Right, and that opacity itself is a cost. You can't optimize what you can't see. Lucas: So let me give listeners one concrete takeaway. If you're running GPU workloads in the cloud today, start by measuring GPU utilization per instance for one week. Use the native monitoring tools — AWS CloudWatch, Azure Monitor, GCP Cloud Monitoring — and look at the 'GPU utilization' metric. If it's below fifty percent, you have a waste problem. Luna: And then look at the gaps. Are you seeing idle time between jobs? That's a job scheduler or autoscaling fix. Are you seeing low utilization during training? That's a data pipeline or model parallelism issue. Lucas: And if you're a startup or a mid-size company, consider spot instances or serverless inference for non-critical workloads. The savings can be dramatic — I've seen cases where companies cut their GPU bill by fifty percent just by moving batch inference to serverless. Luna: Fifty percent. That's the difference between a viable AI initiative and one that gets killed in the next budget cycle. Lucas: Exactly. And that's the real story here. The cloud GPU market is still maturing. We're in a phase where the technology is ahead of the cost management practices. But that gap is closing fast — driven by CFOs asking the right questions. Luna: And that's a good place to leave it. Next time, we'll look at the other side of the coin: how cloud providers are pricing inference versus training, and why that might shift your architecture decisions. Lucas: Looking forward to it.