Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / Cloud Contracts Now Charge for Kubernetes Pod Eviction Costs
Transcript
- Lucas: If you've been running Kubernetes in production for more than a few months, you know the feeling when kubectl get nodes starts returning NotReady. That moment right before the cluster autoscaler kicks in and pods start flipping to Pending. Luna: Right, that gut-drop when you see a node drain happening and you're just waiting for the pager to go off. Lucas: Exactly. Well, as of late last quarter, that moment now has a direct line-item cost attached to it. AWS, Azure, and GCP have all quietly introduced charges triggered specifically when a pod is evicted from a node. Not rescheduled — evicted. Meaning the kubelet on the node decides the pod has to leave, either because of resource pressure or a node failure. Luna: Wait, so they're charging for what used to be just a standard Kubernetes lifecycle event? That feels like a pretty big change. Lucas: It is. And it's not a single fee. It's a cascade. When a pod gets evicted, the control plane has to update the API server, detach any persistent volumes, reattach them on the new node, and potentially re-route networking. Each of those steps now appears as a separate charge on your cloud bill. Luna: So you're paying for the cluster to recover from a state that it was already trying to recover from. Lucas: Exactly. And the pricing varies by provider. Let's use a concrete example. Say you're a mid-size e-commerce company running around 1,200 pods on Spot instances across three availability zones. Last month, you hit a price spike in the Spot market — which happens — and within about three minutes, you lose 40 nodes. That triggers a wave of evictions. Luna: Three minutes, 40 nodes gone. That's a lot of evictions. Lucas: Yeah, it's hundreds of pods in that window. Under the new pricing model on AWS, each eviction generates an API call to the EKS control plane. That's billed per million requests — but suddenly you're adding tens of thousands of extra requests in a very short span. Then each pod that had a persistent volume attached triggers a volume detachment API call, which also has a cost. Luna: And when those pods get rescheduled, you're also paying for volume reattachment, right? Lucas: Yes. The reattachment is a separate charge. So for that 3-minute event, one company I spoke with saw an extra fourteen thousand dollars in line items they had never seen before. Fourteen thousand dollars for what was essentially a normal Spot instance churn event. Luna: That's real money. And it's not just Spot instances, is it? What about when a node fails due to hardware or when the kubelet crashes? Lucas: No, it applies there too. Any eviction where the pod is terminated by the node rather than by a user-initiated rollout. GKE actually bills it under a line called 'forced pod migration events.' On Azure, it's baked into the AKS surcharge for control plane operations. Each provider has its own name, but the mechanism is the same. Luna: So what do the providers say is the justification? I'm guessing they frame it as aligning cost with resource consumption. Lucas: That's exactly the line. They argue that evictions consume control plane capacity — API server CPU, etcd writes, network I/O — and that heavy eviction patterns should be priced accordingly. And to be fair, if you're running a very noisy cluster with bad pod affinity rules that cause constant churn, you probably should feel some cost pressure to fix it. Luna: But the counter argument is that it punishes people for doing exactly what the cloud promised — being elastic. Lucas: That's the tension. The whole value proposition of Spot instances is that you get deep discounts in exchange for interruptibility. Now the cost of that interruptibility is partly being priced back in through the eviction charges. It changes the math on how much you actually save. Luna: Let's talk about what operators can do. I assume pod disruption budgets are the first line of defense? Lucas: PDBs help, but they don't prevent evictions — they only control how many pods can be unavailable at once. They won't reduce the API call volume. What really matters is node pool design. If you spread pods across smaller, more numerous nodes, each eviction affects fewer pods, which means fewer API calls and volume operations. Luna: So instead of running 10 big nodes, you run 40 small nodes. Lucas: Right. You also want to use topology spread constraints to avoid packing pods into a single failure domain. And on the volume side, using CSI drivers that support volume inlining or ephemeral volumes can skip the detach-reattach cycle entirely for pods that don't need persistent storage. Luna: What about using node pools with On-Demand instances as a backup? That would reduce the eviction frequency from Spot interruptions. Lucas: That's a common pattern — use a mixed instance policy where Spot is the primary and On-Demand kicks in when Spot is unavailable. But you still pay the eviction charge when Spot nodes are reclaimed. It's just less frequent. The key is to audit your eviction rate before these charges become material. Luna: And if you're already seeing those charges, what's the first thing to check? Lucas: Look at the 'kube-system' namespace. A lot of operators don't realize that add-on pods — like the metrics server, CoreDNS, the cluster autoscaler itself — can be evicted too. If your system pods don't have priority classes set correctly, they can get evicted during resource pressure, and those evictions also generate charges. Luna: So you could be paying to evict the very pods that keep the cluster healthy. Lucas: Exactly. And that's the kind of cost that's completely invisible until you get the bill. The providers have published the pricing, but it's buried in the fine print of the control plane pricing pages. I'd be surprised if more than five percent of Kubernetes operators have actually read those updates. Luna: It makes me wonder — what other standard Kubernetes operations are going to start showing up as line items? Is pod scheduling going to be charged per decision next? Lucas: I think that's the direction of travel. We've already seen charges for container image pull frequency, for secret rotation, for DNS response latency. The pattern is that any operation that touches the control plane is becoming a metered resource. The underlying logic is that the control plane was always a shared resource, and now they're unbundling it. Luna: It does make you appreciate the times when these things were just included in the cost of the service. Lucas: It does. And that's actually something we think about a lot on this show. These episodes take time to research — looking at contract changes, talking to operators, building out the scenarios. We do it because we believe understanding the fine print matters for people who are building and running infrastructure. Luna: Yeah. And we keep the show ad-free deliberately. No sponsors, no ad reads. It's just this conversation, and the archive is there whenever you need it. Lucas: If you find that useful and want to support that choice, the link is buy me a coffee dot com slash fexingo. No pressure, no perks. Just a way to keep this independent. Luna: And it genuinely helps. Okay, back to eviction costs — there's one more thing I want to cover. Is there any difference in how the three hyperscalers handle the charge for pods that are part of a StatefulSet? Lucas: Great question. Yes, and it's significant. On GKE, if a StatefulSet pod with a PVC is evicted, the reattachment charge is per volume, which for a StatefulSet with 100 pods, each with its own volume, gets expensive fast. On EKS, the charge is per API call, so it scales with the number of pods, not volumes. On AKS, they bundle the first 10 volume operations per month per cluster, then charge after that. Luna: So the cost profile depends heavily on workload type. Stateless deployments on small nodes are cheap to evict. StatefulSets with large volumes on big nodes — that's where the cost spikes. Lucas: Exactly. And knowing that lets you predict which workloads will be hit hardest. If you're running a high-traffic StatefulSet for a real-time application, you might want to consider using On-Demand instances instead of Spot, even if the compute cost is higher, because the eviction cost savings might outweigh it. Luna: Which brings us back to finops. This is another layer of complexity where finance and engineering have to sit down together. Lucas: It is. And the good news is that the tools are catching up. CloudHealth and Kubecost have started surfacing eviction-related costs in their dashboards. But the first step is still awareness — knowing that the line item exists and what it means. Luna: I think that's the biggest takeaway. If you're running Kubernetes, check your latest bill for anything labeled 'control plane operations' or 'pod migration.' It might be small right now, but if you scale, it will grow. Lucas: And that's the thing about cloud contracts — they don't change overnight to make headlines. They change in the line items you haven't looked at yet.