Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / How Cloud Giants Profit From Your Idle Compute Hours
Transcript
- Lucas: So here is a number that keeps me up at night when I audit cloud bills for mid-market enterprises: roughly thirty percent of all provisioned cloud compute hours are effectively idle or running at minimal utilization. And yet, if you look at the balance sheets for Amazon, Microsoft, and Alphabet heading into September 2026, their cloud infrastructure segments are posting record growth. Luna: That sounds like a classic case of selling air, but they must be doing something clever with it because those idle servers aren't exactly free to maintain. Lucas: They aren't free, but they are sunk cost. The hardware is already bought, the data center rent is paid, and the cooling is running. What these giants have done over the last decade is create a secondary market for that excess capacity. They call it spot instances on AWS, spot virtual machines on Azure, and preemptible VMs on Google Cloud. It is essentially a liquid market for spare compute power. Luna: Right, so you are betting on the fact that no one will ever need that exact slot at that exact second, which allows them to sell it for pennies while still making a margin. Lucas: Exactly. And the margins are surprisingly healthy because the marginal cost of running an additional virtual machine on an existing physical host is nearly zero. If a server has five cores left after serving its primary customer, filling those cores with a spot instance user who is willing to accept a ninety percent discount relative to on-demand pricing generates pure incremental profit for the provider. Luna: But there is a catch, obviously, otherwise every company would run their entire production stack on spot instances and save millions. Lucas: The catch is preemption. When demand spikes for on-demand workloads, or when the cloud provider needs to reclaim resources for maintenance, they can terminate your spot instance with very little notice. Typically two minutes. So you are trading price for stability. It is a fundamental architectural trade-off that forces engineering teams to rethink how they build applications. Luna: Two minutes is terrifying if you are in the middle of a database transaction. How do you even design software to survive being murdered by a billing algorithm? Lucas: You have to design for statelessness. That is the golden rule of the spot economy. If your application state is stored locally on the instance disk, you lose everything when it dies. But if you externalize your state to managed databases like Aurora or Cosmos DB, and keep your logs in object storage like S3 or Blob Storage, then the compute instance becomes disposable. You treat it like cattle, not pets. Luna: So the value shifts from the server itself to the architecture around it. That seems like a huge shift in mindset for traditional IT shops. Lucas: It is massive. In the past, you provisioned for peak load and sat there with empty capacity during lulls. Now, you provision for average load and use spot instances to handle the spikes. Or conversely, you use spot for batch processing, CI/CD pipelines, and rendering farms where interruptions just mean a job restarts later. It changes the entire procurement strategy from static budgeting to dynamic optimization. Luna: I remember reading about a media company that used this for video encoding. If a render gets killed, they just spin up another node and pick up where it left off, right? Lucas: Precisely. And they save about eighty-five percent compared to standard on-demand rates. For a company processing petabytes of video daily, that is not just savings; that is competitive advantage. It lowers the barrier to entry for compute-intensive tasks because the cost floor drops dramatically. Luna: But doesn't the cloud provider worry that if they kill too many jobs, users will get angry and leave? Lucas: They do, which is why the pricing model includes a feedback loop. When supply is tight and demand is high, spot prices go up. Sometimes they can approach on-demand prices. This acts as a circuit breaker. It tells customers to slow down or switch back to stable instances, which frees up capacity for higher-paying on-demand users. It is a self-regulating market mechanism built directly into the infrastructure. Luna: That is actually quite elegant. It prevents the provider from starving their best customers during a crisis because the spot users simply become too expensive to keep running. Lucas: Yes, and it also incentivizes efficiency. If you know your spot instances might die in two minutes, you optimize your code to start fast and checkpoint frequently. You stop writing long-running monolithic processes and start breaking things into smaller, resilient microservices. The pricing pressure drives better engineering practices. Luna: It feels like the cloud providers are subtly forcing us to adopt modern DevOps practices whether we like it or not. Lucas: I would argue they are just exposing the inefficiencies of legacy architectures. If your app cannot survive a reboot, it was probably fragile anyway. Spot instances just make that fragility costly. Though, there is a nuance here regarding availability zones. Not all regions have the same spot inventory. Luna: So you can’t just assume you’ll find cheap compute everywhere? That adds another layer of complexity to the deployment strategy. Lucas: Correct. In some mature regions like us-east-1, spot inventory is abundant and prices are stable. In newer or specialized regions, supply can be scarce, causing prices to spike unpredictably. Smart operators use multi-region strategies to access deeper pools of capacity, spreading their risk across different geographic markets. Luna: This brings up the question of orchestration. Managing hundreds of short-lived instances manually is impossible. Kubernetes has become the de facto tool for this, hasn't it? Lucas: Absolutely. Kubernetes native spot support is now table stakes. Tools like Karpenter on AWS or Virtual Kubelet allow clusters to automatically request spot nodes, fill them with pods, and gracefully drain them when preemption signals arrive. The platform handles the churn so the developers don't have to think about it. Luna: And what about the financial side? How do CFOs track this if the instances are constantly appearing and disappearing? Lucas: That is where FinOps tools come in. You need granular tagging and real-time cost allocation. Since spot instances change price dynamically, historical cost analysis can be messy. Modern platforms show you the effective hourly rate at the moment of usage, allowing finance teams to see the true delta between on-demand and spot spending. Luna: It seems like the biggest risk isn't technical failure, but financial surprise. If someone misconfigures a cluster and requests ten thousand spot instances during a price spike, the bill could explode. Lucas: That is a real danger. That is why hard limits and budget alerts are non-negotiable. You set maximum bid prices for your spot requests. If the market price exceeds your bid, the instance is never launched. It protects you from runaway costs while still capturing the savings when conditions are favorable. Luna: So you are playing a game of chicken with the market price, but with guardrails. Lucas: Exactly. And as we move further into 2026, with AI workloads consuming more and more GPU capacity, the dynamics are shifting. GPU spot instances are much scarcer than CPU instances. The preemption risk is higher, and the price volatility is wilder. But for training runs that can be interrupted and resumed, the savings potential is enormous. Luna: It’s fascinating how a simple concept like 'spare time' has become a sophisticated asset class. I guess nothing in business is truly wasted if you can find a buyer for it. Lucas: True. And for the listeners building systems today, ignoring spot instances is leaving money on the table, but adopting them blindly is asking for instability. The sweet spot is a hybrid approach: stable core services on reserved or on-demand capacity, and volatile, scalable layers on spot markets. Luna: It really comes down to knowing what part of your system can afford to take a hit versus what part must stand firm. Lucas: Precisely. Architecture is economics. Every line of code and every configuration file has a price tag attached to its resilience. Understanding that price is what separates good engineers from great ones. Luna: If these conversations are useful for what you're building or running, consider supporting the show directly. Lucas: We deliberately don't run ads on these. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. Luna: Your support helps us keep digging into the details that matter without compromise. Lucas: Back to the technical side, one final thought on resilience. Even with perfect architecture, network partitions happen. Preemptions happen. The goal isn't to prevent failure, but to contain it. Luna: Chaos engineering has made that philosophy mainstream, testing systems precisely to see how they break. Lucas: Right. And when you combine chaos testing with spot instance pricing, you get a very clear picture of your system's true robustness. If your app falls apart every time a node disappears, you know exactly where to invest before you scale out. Luna: So the spot market acts as a stress test that pays you to participate. Lucas: In a way, yes. You get cheaper compute and better engineering discipline simultaneously. It is one of those rare win-win scenarios in tech. Luna: I wonder if other industries will start mimicking this model, offering 'interruptible' discounts for other types of services. Lucas: Probably. We already see it in energy grids with dynamic pricing. As digital infrastructure becomes more abstract, maybe physical resources will follow the same pattern. It is worth watching how those boundaries blur. Luna: A world where everything has a spot price. I suppose we should all learn to live with two minutes of warning. Lucas: Just make sure your data is saved before the timer hits zero.