Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / Why Cloud Contracts Now Charge for AI Training Job Interruption Costs
Transcript
- Lucas: So you're training a large language model on AWS or Azure, and you decide to use spot instances to save money. That's a smart move, cutting your GPU costs by up to 70 percent. But here's something that caught me off guard: the cloud contract now includes a charge every time your training job gets interrupted and has to restart. Luna: Wait, really? I thought spot interruptions were free — you just lose the compute and move on. Who's charging for the restart? Lucas: Both AWS and Azure have introduced a new metering line item this year. They call it something like 'interrupted compute restart fee' or 'spot interruption recovery charge.' It's not huge per event — typically a few cents per restart — but when your training job runs for weeks and gets interrupted dozens of times a day, it adds up fast. Luna: That sounds like death by a thousand paper cuts. Do you have a concrete example? Lucas: Yeah, I spoke with a startup that was fine-tuning a 7-billion-parameter model on Azure. They used spot GPU instances to keep costs low. Over three months, they had about 1,200 interruptions. At an average of 12 cents per restart, that's an extra $144 dollars. Not huge, but then they also got charged for the data re-read from blob storage each time — another $300. And the network egress to re-download checkpoints? $200 more. Suddenly, their 'savings' from spot instances shrank by almost 20 percent. Luna: Ouch. So the hidden costs eat into the benefit. And this is exactly the kind of detail that's invisible unless you're really digging into your bill. You know, a lot of listeners tell us these deep dives help them save real money. Lucas: Absolutely. And that's why we do it — it's just the two of us, no ads, no sponsors. If you want to support the show, it's as simple as buy me a coffee dot com slash fexingo. Even a couple of dollars a month genuinely keeps these episodes going. And listeners have said it's helped them negotiate better contracts or catch overcharges. So if you've gotten something out of it, that's the place. Luna: Yeah, we appreciate every bit. And speaking of digging into the bill, let's get back to those restart fees. Why are cloud providers suddenly charging for this? Wasn't the whole point of spot instances that you pay only for the time you use? Lucas: Right, the original spot model was strictly pay-per-second. But the providers argue that restarting a training job consumes internal resources — orchestrating the new allocation, re-mounting storage, re-establishing networking. They say it's similar to how they now charge for container registry pulls or Terraform plan runs. It's all about metering every infrastructure action. Luna: So it's not just about compute; it's about the orchestration overhead. How much does each restart actually cost on average? Lucas: From the bills I've seen, it ranges from 5 cents to 25 cents per restart, depending on the GPU type and region. For an A100 instance in us-east-1, it's about 8 cents. That might sound trivial, but if you have 100 GPUs in a cluster and each gets interrupted once an hour, that's $8 per hour in restart fees alone. Over a month, that's nearly $5,800. Luna: And that's on top of the compute cost. So for large-scale training, these fees could easily reach tens of thousands of dollars. What about Google Cloud? Do they do this too? Lucas: Google Cloud hasn't introduced a separate restart fee yet — at least not as of mid-2026. But they do charge for 'preemption overhead' in their TPU spot instances, which is similar. They style it as a higher per-second rate for spots that have a lower preemption probability. So you're indirectly paying more for fewer interruptions. Luna: Interesting. So the general trend across all three hyperscalers is to monetize the ephemerality of spot instances. What can teams do to minimize these charges? Lucas: First, use checkpointing aggressively. The faster you can resume from the last save, the less you pay in re-compute and data re-read. Some frameworks like PyTorch Lightning now support incremental checkpointing that saves every few minutes. Second, consider using a mix of on-demand and spot — keep the critical few nodes on-demand to avoid restarts, and use spot for the rest. Third, set a interruption budget: if your restart fee exceeds a threshold, switch that job to on-demand. Luna: That last point is clever — treat restart fees as a variable cost and cap it. Are there tools that automate this? Lucas: Yes, some third-party cost management platforms are starting to include restart fee tracking. Also, AWS has a feature called 'Capacity Rebalancing' in EC2 Auto Scaling that tries to proactively replace spot instances before they're interrupted. That can reduce the number of restarts by 30 to 40 percent. Azure has something similar called 'Eviction Policy' that lets you set a notification time. Luna: So it's not all bad news. The cloud providers are giving you levers to control these costs. But you have to know they exist. That's the real takeaway: don't assume billing is transparent. Lucas: Exactly. And that's why we dedicate episodes to these overlooked charges. One more thing: the startup I mentioned actually ended up switching to on-demand for the final week of training because the interruption rate spiked. They paid more per hour but avoided over $4,000 in restart fees. So it's not just about defaulting to spot; you need to monitor the interruption rate dynamically. Luna: That's a great strategy. So for our listeners who are running AI training, what's the single most important action item from this episode? Lucas: Open your cloud billing console and search for 'interrupt' or 'spot restart' in the line items. If you see a charge, you're being billed for restarts. Then track the total over the past month. If it's significant, implement aggressive checkpointing and consider a hybrid spot-on-demand architecture. And if you're on Azure, look for the 'Spot Restart Fee' in your Azure Cost Management reports — it's under 'Other Compute Charges'. Luna: Great advice. And while you're at it, maybe check out the show notes for links to the tools we mentioned. Lucas, any final thoughts? Lucas: Just that cloud pricing is increasingly granular. The era of 'pay only for what you use' is being replaced by 'pay for what you use, plus every action that supports that use.' For AI teams, that means the cost of flexibility has a line item. Understand it, or it will eat your budget. Luna: Well said. Thanks for joining us on The Cloud Business Podcast. Until next time. Lucas: Take care, everyone.