Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Used Observability to Find a Memory Leak Saving Millions
Transcript
- Lucas: So there's this story from a mid-sized SaaS company — about a hundred and twenty engineers — where the CTO noticed their cloud bill had drifted up about eighteen percent over six months. No new service launches, no traffic spike. Just a steady, silent climb. Luna: That's the kind of drift that drives finance people crazy. It's too slow to trigger an alarm, but over a year it's hundreds of thousands of dollars. Lucas: Exactly. And the engineering team had been chasing it with the usual playbook — rightsizing instances, looking at reserved instances, checking for orphaned resources. Standard stuff. But the bill kept creeping. So the CTO did something different. She went straight to the observability stack. Luna: Before we go deeper — if today's tech conversation gave you something usable, the way this stays ad-free is listener support. It's buy me a coffee dot com slash fexingo. Lucas: Yeah, that's right. Totally optional, but it's what keeps the show independent. So she dives into Prometheus and starts looking at metrics she hadn't looked at in detail before. Luna: What did she find? Lucas: She noticed a gradual tail-latency creep in one microservice — the p99 was going up about two milliseconds per week. That's invisible to most dashboards unless you're looking at a trend over months. And separately, container restarts in that same service had increased by a factor of three. Luna: Those two together suggest something like a memory leak. A leak would cause the container to gradually consume more memory, eventually get killed by the orchestrator, restart, and then the cycle repeats. Lucas: Right. And the restarts were frequent enough to keep the service alive, but each restart meant a cold start — which is why the tail latency was creeping. But here's the thing: the team had alerts on restarts and alerts on latency. They just never connected them. The restarts alert fired maybe once a day, and the latency alert never fired because the p99 stayed under their threshold. Luna: So they were looking at individual signals in isolation. The CTO's insight was to combine them into a single composite metric. Lucas: Exactly. She built a custom alert based on what's called the residual sum of squares, or RSS. She took the expected memory usage under normal load, computed the residual — the difference between actual and expected — and then squared it to amplify small deviations. When that RSS started trending upward over a 24-hour window, the alert fired. Luna: That's clever. Because a memory leak is a slow drift, not a spike. You need a statistical approach to catch it early. Lucas: And it worked. The alert caught the leak when the excess memory was still under 200 megabytes. The team traced it to a caching layer that was holding references to objects that should have been garbage-collected. A single developer fixed it in an afternoon. The cloud bill dropped by forty thousand dollars a month. Luna: Forty thousand? That's almost half a million a year for one bug. And the fix was trivial. Lucas: That's the thing about observability — it's not just about uptime. It's a cost-control lever. But you have to build the right signals. The CTO told me that before this, the team treated observability as a reliability tool. Now they treat it as a financial instrument. They have a dashboard that shows cost per request broken down by service. Luna: That's a cultural shift. You're asking engineers to think like a CFO. Did that create friction? Lucas: Some at first. Engineers pushed back — they said they didn't want to be responsible for cost. But the CTO framed it as a learning tool, not a performance metric. She said, 'I'm not going to evaluate you on cost per request. But if you see a service where cost per request is going up, you should be curious.' Luna: That's smart. It turns cost into a signal, not a target. Lucas: Exactly. And the curiosity paid off. Within two months, three other teams found similar patterns — memory leaks, inefficient queries, even a misconfigured load balancer that was routing traffic to an expensive instance type. Combined, those optimizations cut their monthly cloud spend by another twenty-two percent. Luna: So the initial forty thousand was the tip of the iceberg. The cultural shift was the real win. Lucas: I think so. The CTO said the biggest lesson was that observability needs to ask the right questions. You can have all the metrics in the world, but if you're not looking for the right patterns, you're just collecting noise. Luna: And the RSS approach — is that something a typical team could implement without a data scientist? Lucas: Absolutely. Prometheus has built-in support for predicting time series. You can use the 'predict_linear' function to forecast memory usage, then subtract the actual value. It's maybe ten lines of PromQL. The hard part is knowing what to look for. Luna: The pattern. A leak plus restarts plus latency creep. Lucas: Right. And that pattern is common. I've seen it in at least five different companies. But most teams don't connect the dots because they're looking at each metric on a separate dashboard. Luna: So the practical takeaway: if your cloud bill is drifting, don't just look at instance sizes. Look at memory usage over time, container restart rates, and tail latency — all on one graph. Lucas: And if you see that pattern, build a composite alert. The CTO in this story now runs a quarterly 'observability audit' where the team looks for these kinds of drifts. It's become part of their operational rhythm. Luna: I like that. It's proactive, not reactive. And it saves real money. Lucas: Yeah. And it's a reminder that sometimes the most valuable engineering work isn't shipping features — it's finding the thing that's quietly costing you money. Luna: Well said. That's a good place to leave it.