Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One Engineering Team Cut Cloud Costs by 40 Percent Without Hurting Performance
Transcript
- Lucas: So last week I was talking to a CTO at a Series B company — about 80 engineers — and she told me their AWS bill had hit two hundred thousand dollars a month. And that number wasn't the problem. The problem was that nobody on the engineering team could tell you which features or customers that spend was tied to. Luna: That's a pretty common story. Cloud spend becomes this abstract line item that finance flags every quarter, but engineering doesn't feel ownership over it. Lucas: Exactly. And the typical response is to hire a FinOps person or to send out a memo saying 'please turn off unused resources.' Neither works long-term. But this team did something different. Over six months, they cut their cloud costs by forty percent — without degrading performance or slowing down feature work. Luna: Forty percent is significant. What was the first move? Lucas: They started with a simple tagging policy. Every resource had to have three tags: a service name, an owner team, and a cost center linked to an OKR. It took about two weeks to clean up the existing resources, but once it was in place, they could run a report that showed exactly which team's OKRs were driving which costs. Luna: So they created visibility before asking anyone to change behavior. That's smart — you can't optimize what you can't measure. Lucas: Right. And the visibility alone drove some quick wins. One team discovered they had six load balancers pointing to the same service because no one had cleaned up after a migration. That was about eight thousand dollars a month in wasted spend, just sitting there. Luna: Nice. Those are the low-hanging fruit. But the forty percent reduction suggests they went deeper. Lucas: They did. Next step was right-sizing. They used AWS Compute Optimizer to look at their EC2 instances and found that about thirty percent were over-provisioned by at least two sizes. For example, they had m5.xlarge instances running a service that was averaging five percent CPU utilization. Downgrading to m5.large saved them fifty percent on those instances with zero performance impact. Luna: That seems obvious, but I've seen teams resist it because they're afraid of capacity issues during a spike. Lucas: Yeah, that fear is real. So what they did was implement a two-week canary period. They'd change the instance type for a subset of traffic, monitor p99 latency and error rates, and only rolled it out fully if metrics stayed flat. That gave teams confidence. Luna: Smart. So the process itself had guardrails. Lucas: Exactly. After right-sizing, they moved to purchasing commitments. They had about forty percent of their compute running on on-demand instances. They analyzed the last three months of usage and identified workloads that were stable — things like their main application servers and databases. They bought one-year reserved instances for those, which saved them about twenty-five percent on those resources. Luna: One-year, not three-year? That's interesting — usually the bigger savings come from three-year commitments. Lucas: True, but they wanted flexibility. The company was growing fast, and they didn't want to lock into a three-year architecture that might change. The one-year reserved instances still gave them a meaningful discount, and they could reassess annually. Luna: That makes sense for a Series B. What about spot instances? Lucas: They used spot instances for non-production workloads first — CI/CD runners, staging environments, batch processing. That alone cut those costs by sixty to seventy percent. Then they got bold and moved some stateless production services onto spot instances with a fallback to on-demand. They used a Spot Fleet with a diversified instance pool, so if one type got reclaimed, the traffic would shift to another. Luna: Did they have any disruptions? Lucas: A few. But because the services were stateless and designed to handle instance termination gracefully — using load balancer draining and health checks — users never noticed. They measured it, and over three months they had less than thirty seconds of cumulative impact on any single service. Luna: So the architectural decisions enabled the cost savings. That's a key point — you can't just throw spot instances at a stateful monolith and expect it to work. Lucas: Exactly. And that brings me to the cultural piece. The CTO told me the biggest win wasn't the forty percent savings. It was that engineers started thinking about cost as a design constraint. When they proposed a new feature, they'd include a rough estimate of the cloud resources it would need, and the team would discuss whether the business value justified it. Luna: That's a real shift. How did they get engineers to care? Lucas: They made cost visible in the tools engineers already used. They set up a Slack bot that posted a weekly cost report per team, and they integrated cost data into their deployment pipeline. Every time a team deployed a service, the pipeline would show the projected cost change for that release. It turned cost into a metric like latency or error rate. Luna: So it became part of the definition of done. Lucas: Yeah. And they gamified it a bit. They had a quarterly 'cost efficiency award' for the team that reduced their per-customer cost the most without breaking anything. The winning team got a budget to throw a team lunch. It sounds small, but it created friendly competition. Luna: I love that. It's positive reinforcement instead of a cost-cutting mandate from above. Lucas: Right. And the results sustained. A year later, their absolute cloud spend was still lower than before the initiative, even though their customer base had grown by forty percent. Their cost per customer had dropped by more than half. Luna: That's the kind of metric that justifies the effort to any CFO. Lucas: Absolutely. And you know, conversations like this — where we dig into real tactics that teams have used — they're exactly what keeps this show going. If you've gotten something useful out of today's episode, a couple of dollars a month at buy me a coffee dot com slash fexingo genuinely makes a difference. It keeps us ad-free and focused on practical stories like this one. Luna: Yeah, it's a small way to support the kind of detailed, engineer to engineer conversation we try to have here. Lucas: So back to the tactics — one last thing I want to mention. The team also implemented an auto-shutdown policy for staging environments. A junior engineer wrote a script that turned off all non-production resources at 8 PM and turned them back on at 8 AM. That saved them about fifteen thousand dollars a month, and it became the engineer's first production-level contribution that got noticed by leadership. Luna: That's a great example of empowering engineers to solve problems they see. It wasn't a top-down mandate — it came from someone who was annoyed that staging servers ran all night. Lucas: Exactly. And that's the takeaway for me. The most effective cost optimization isn't about a single big initiative. It's about creating visibility, giving teams autonomy, and making cost a natural part of how engineers think about their work. The forty percent savings was a side effect of a better culture. Luna: So if someone wants to start this on their team, what's the first step? Lucas: Start with tagging. Pick three tags — service, owner, cost center — and enforce them in your infrastructure as code templates. Then run a report and look for the biggest anomalies. You'll almost certainly find something that's been running for months that nobody knew about. Fix that, and you've got a story to tell the rest of the team. Luna: And once you have that story, the momentum builds. Lucas: Right. It doesn't have to be perfect out of the gate. It just has to start.