Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One Team Cut Their Cloud Bill By Changing Nothing
Transcript
- Lucas: You know that feeling when you open your cloud billing dashboard and just… wince? Luna: Every month. It's like a subscription you forgot you had, times a thousand. Lucas: Right. Most teams respond by trying to squeeze pennies out of compute — reserved instances, spot instances, rightsizing. All good tactics. But I want to talk about a different approach today, one that a team I know of used and saved four hundred thousand dollars a year without changing a single line of code. Luna: Four hundred K? That's real money. And they didn't touch the code? Lucas: Not a line. What they did was find and kill zombie workloads and enforce tagging so aggressively that they could finally see what they were paying for. Luna: Zombie workloads — so, resources that are running but nobody's using them? Lucas: Exactly. Old staging environments, orphaned load balancers, unused volumes, snapshots that pile up. The company I'm thinking of is a mid-stage SaaS startup, about two hundred engineers. They did a six-week cleanup project. No new tooling. Just a spreadsheet and a lot of Slack messages. Luna: And they found four hundred thousand dollars? That's a lot of zombie. Lucas: It was about a hundred and eighty separate resources that were running with zero active traffic. Some had been forgotten for over a year. One single database instance — a big one — was still running because a team had spun it up for a proof of concept and then never turned it off. That one instance alone was costing about twenty thousand a year. Luna: Ouch. And nobody noticed because it was in a shared account with a thousand other resources. Lucas: Exactly. And the tagging piece is equally critical. Without tags, you can't attribute cost to any team or service. So you see a big number in the dashboard but you have no idea who to ask. Luna: Right. It's like getting a restaurant bill with no itemization. You know you spent a lot, but was it the steak or the dessert? Lucas: That's the metaphor. So they implemented a mandatory tagging policy: every resource must have an owner, a service name, and an environment tag. If a resource is untagged after a two-week grace period, it gets automatically terminated. Luna: Auto-termination sounds aggressive. What if it kills something important? Lucas: They had a safety net: resources could be added to an exception list with a manager's approval. But for a resource to get on that list, it had to justify why it couldn't be tagged. And the bar was high. I think only about five things made it onto the exception list. Luna: So the cleanup itself saved money, but the tagging is what prevents the waste from coming back. Lucas: Exactly. The four hundred thousand wasn't a one-time saving. The ongoing benefit is maybe fifty thousand a year just from the tagging discipline. New resources come up tagged, so when they're no longer needed, someone can easily see who owns them and ask. Luna: What about the cultural side? Did engineers resist being told to tag everything? Lucas: Initially, yes. But the engineering manager framed it as a quality of life improvement. Instead of spending hours each month trying to figure out which team owns which resource, tagging makes it everyone's job for thirty seconds at creation time. Luna: And I imagine it also helps with security and compliance. You know what's out there. Lucas: Massively. They found three unpatched instances that had been running for months. That's a security risk nobody knew about. So the cleanup wasn't just a cost exercise — it was a hygiene exercise. Luna: I love that this didn't require a dedicated FinOps team. It was just a focused effort for a few weeks. Lucas: Exactly. And it's repeatable. Any team could do this. The formula is simple: inventory, tag, kill. But you have to be willing to actually turn things off. Luna: That's the hard part. Fear of breaking something. Lucas: Yeah. That's why they started with resources that had zero traffic for ninety days — those are very unlikely to be critical. And they communicated constantly. Every resource they planned to terminate was listed in a shared doc with the owner name, and they would ping that person on Slack. If no one responded in a week, it got terminated. Luna: That seems fair. And I bet after the first few terminations, people started paying attention. Lucas: Absolutely. The first time they terminated a resource that someone actually needed, it was a learning moment. But because they had the communication trail, they could spin it back up quickly. I think that happened maybe twice. Luna: So the net result was huge savings, better security, and a culture of cost awareness. Lucas: Yeah. And the engineering manager who led it said the biggest surprise was how little code change was needed. It was all operational discipline. Luna: I think a lot of teams default to 'we need a new tool' or 'we need to rewrite this service' when the real answer is often 'we need to clean up what we have'. Lucas: That's the lesson. Before you optimize your compute, check if half of it is even being used. It's the low-hanging fruit that nobody picks. Luna: If today's tech conversation gave you something usable, you know the drill. Buy me a coffee dot com slash fexingo. That's what keeps this show ad-free and running. Lucas: Yeah, it's a small thing that makes a big difference. We really appreciate it. Luna: Alright, back to the tactics. Lucas, you mentioned they used a spreadsheet — not a fancy tool. What was their actual process for discovery? Lucas: They started with the cloud provider's native cost explorer. Just a simple export of all resources and their monthly cost. Then they cross-referenced with metrics like CPU utilization, network traffic, and last login timestamps. Luna: So they looked for resources that had basically no activity. Lucas: Right. The threshold was anything with less than one percent CPU average over a month and zero network traffic. That flagged about two hundred resources. Then they manually reviewed each one. Luna: That sounds tedious, but for four hundred thousand dollars, it's worth the effort. Lucas: Exactly. And now they run that report quarterly. It takes an engineer about two hours per review. Luna: Two hours to save tens of thousands. That's a pretty good return on time. Lucas: The best part is, it scales. As the company grows, the same process catches new zombies. The tagging policy ensures that when someone leaves the company, their resources don't become orphans. Luna: So that's the playbook: inventory, tag, kill. And then repeat quarterly. Lucas: Exactly. No new tools, no code changes, just good housekeeping. Luna: I think that's a great note to end on. Thanks, Lucas. Lucas: Thanks, Luna. And to our listeners — go check your cloud dashboard. You might find a zombie or two.