Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / How Cloud Teams Are Killing Idle Zombie Processes
Transcript
- Lucas: There is a specific type of waste in enterprise cloud spending that almost nobody talks about until they find it. It is not egress fees, and it is not reserved instance mismanagement. It is what engineers call zombie processes — compute instances that are technically running, consuming memory and storage IOPS, but doing absolutely zero useful work. Luna: So you mean servers that are just sitting there, humming along and burning cash? Lucas: Exactly. Think about a microservice that crashed six months ago. The code is dead, the team moved on to the next feature, but the virtual machine is still allocated. It is holding onto its IP address, its attached storage volume, and maybe even some cached data. It looks alive in the dashboard, but it is effectively a ghost. Luna: And because it is still technically online, the billing engine keeps charging for it as if it were active. Lucas: Right. And here is the tricky part. These zombies are often hiding in plain sight. They might be running on a shared cluster where their individual CPU usage drops to near zero, but they are still paying for the full baseline of memory and block storage. A mid-sized company could easily have hundreds of these, quietly adding up to tens of thousands of dollars a year. Luna: Is it hard to spot them? Or do you just need to look at the monthly invoice? Lucas: The invoice is too late. You have to look at behavior. The standard heuristic is straightforward: if an instance has less than five percent CPU utilization for fourteen consecutive days, and no outbound network traffic, it is a candidate for termination. But simply killing it isn't enough, because sometimes those zombies are holding dependencies for legacy systems nobody remembers. Luna: So you need a safety net before you pull the plug? Lucas: Absolutely. You need to check the logs. If there are error spikes around the time the service went dark, that tells you something failed catastrophically rather than gracefully. That changes how you handle the cleanup. You want to preserve the state, just in case someone needs to debug why the old system died, but you don't need the compute power anymore. Luna: That sounds like a job for automation. Manual audits never catch everything. Lucas: They don't. The best teams use policy as code tools to tag idle resources automatically. For example, you can set up a rule that says any EC2 instance or Azure VM that meets the idle criteria gets tagged as 'candidate for shutdown'. Then, you get a weekly report instead of a surprise bill at the end of the quarter. Luna: Does this apply to databases too, or is it mostly just app servers? Lucas: It applies heavily to databases. We see a lot of dev and test databases that spin up during the day and then sit idle for twelve hours a night. If you are running those seven days a week, you are paying for peak capacity when you only need half of it. Scaling those down overnight can cut costs significantly. Luna: Nightly scaling seems like a no-brainer. Why doesn't everyone do it? Lucas: Fear. People are terrified that the database won't wake up fast enough when someone needs it at eight in the morning. So they leave it running hot all night. But modern cloud providers have optimized cold-start times so much that the latency penalty is usually negligible compared to the savings. Luna: Okay, so let's talk about how we keep this show going. If these conversations are useful for what you're building or running... Lucas: A couple of dollars a month is genuinely what keeps these going. You can support us directly at buy me a coffee dot com slash fexingo. It helps us keep the research deep and the ads off, and it makes a real difference for a small independent team. Luna: Thanks for listening, and thanks for supporting the work we do here. Lucas: Let's go back to the technical side, because there is a nuance here about storage IOPS that people often miss. When you shut down a zombie process, you might think the storage bill disappears immediately. It doesn't always. Luna: Why is that? Is the disk still spinning? Lucas: Not literally, but the cloud provider still charges for the provisioned capacity. If you left a ten-terabyte volume attached to that idle server, you are still paying for that ten terabytes of storage, plus whatever performance tier you selected. Even if the server is off, the data has to live somewhere. Luna: So the real savings comes from detaching and deleting the volumes? Lucas: Yes, but with caution. You want to take a snapshot first. Snapshots are cheaper than raw volumes because they are incremental. You pay for the change blocks, not the whole disk. So you archive the state cheaply, then delete the expensive volume. Luna: That makes sense. Archive once, delete the rest. But what about networks? Do those zombies still charge for IP addresses? Lucas: This is one of the most annoying parts of cloud billing. Elastic IP addresses, for instance, often incur a small hourly fee if they are not attached to a running instance. It is pennies per hour, but multiplied across hundreds of abandoned projects, it adds up. You have to actively release those IPs back to the pool. Luna: So it is a three-part cleanup: compute, storage, and networking. Lucas: Precisely. And you have to do it regularly. This isn't a one-time fix. New zombies sprout every week as developers spin up temporary environments and forget to tear them down. The culture shift is more important than the tooling. Engineers need to feel comfortable destroying things. Luna: Destroying things. That is definitely a mindset shift for many traditional IT teams. Lucas: It is. In legacy data centers, hardware was expensive, so you kept it running forever. In the cloud, hardware is infinite and cheap, but waste is invisible. The goal is to make waste visible. When you see a line item for an idle server, it feels bad. That feeling drives better behavior. Luna: I like that. Making waste painful is a good incentive. Lucas: Exactly. Another layer to this is monitoring actual user engagement. Some services claim to be idle, but they are actually processing background jobs. You need to distinguish between 'no human interaction' and 'no system activity'. If a server is pinging an external API every thirty seconds, it is working. Don't kill it yet. Luna: Good point. Heartbeats are a sign of life, even if no one is clicking buttons. Lucas: Right. So your filter should be based on total resource exhaustion, not just CPU. Look at memory leaks, too. Sometimes a zombie is leaking memory slowly. It looks fine for a week, then crashes. Those are dangerous because they can take down neighbors in a shared environment. Luna: So stability matters as much as cost. You don't want to save money by breaking production. Lucas: Never. Cost optimization must never compromise reliability. The safest way to clean up is to start with non-production environments. Dev and staging are full of ghosts. Start there. Get the habit right, then move to production. Luna: Start small. Prove the concept in dev, then scale up. Lucas: Yes. And document everything. When you kill a zombie, write a note about why. Did we lose revenue? Did a customer complain? If not, the process was safe. Build a knowledge base of these decisions so future teams don't panic when they see an empty slot in the dashboard. Luna: Documentation is key. Otherwise, it just looks like chaos. Lucas: It does. But the end result is a leaner, faster infrastructure. Less noise in your logs, lower bills, and fewer moving parts to secure. Security is another angle here. Every running server is a potential attack surface. Fewer zombies means fewer doors for hackers to try. Luna: I hadn't thought about the security benefit, but it makes perfect sense. Lucas: It is huge. You reduce your blast radius simply by turning things off. It is the simplest security control that also happens to save you money. Double win. Luna: Turn it off to stay safe and save cash. Hard to argue with that logic. Lucas: Exactly. So if you are looking at your cloud bill this month, don't just look at the total. Drill into the idle resources. Find the ghosts. Turn off the lights. Luna: A cleaner cloud, a lighter bill, and a safer system. That is a pretty good Tuesday.