Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose a Silent Memory Leak
Transcript
- Lucas: You know that sinking feeling when your monitoring dashboard shows a server's memory usage climbing day over day, but nothing in the logs screams 'problem'? Luna: Oh, I know it well. It's the ghost in the machine — everything looks fine until it's not. Lucas: Exactly. And that's a memory leak. Not the dramatic segfault kind — the slow, cumulative kind that eventually gets your process killed by the OOM killer at 3 a.m. on a Sunday. Luna: So let's talk about how you actually catch one before that happens. Lucas: Right. Let's start with a real case. A mid-sized ad-tech company I worked with had a Node.js microservice that handled bid requests. It was a 12-line script, basically. And over six days, it silently consumed 32 gigabytes of memory. Luna: 32 gigs from 12 lines? That's impressive in the worst possible way. Lucas: Right? And what made it tricky was that the memory usage looked linear — not exponential — so their percent-used alarm never tripped until day six, when the server hit 95 percent and the OOM killer stepped in. Luna: So what was the root cause? Lucas: A closure that captured a reference to a large response object, and the developer never set it to null after processing. Every bid request allocated a new object, and the garbage collector couldn't free it because the closure still held a reference. Classic Node.js leak pattern. Luna: Okay, so how would you catch that before it becomes a crisis? I mean, monitoring percent used obviously isn't enough. Lucas: The first thing I look at is /proc/meminfo. Specifically the 'available' metric. Not 'free' — 'available'. Free memory can be misleading because Linux uses free memory for caches. Available memory is what's actually available for new allocations. If available memory is slowly shrinking over days or weeks, that's your early warning. Luna: So you'd graph /proc/meminfo's 'available' over time. What else? Lucas: Then you want to look at per-process memory. But here's the trap: most people look at RSS — resident set size. RSS includes shared libraries, so if two processes use the same library, RSS double-counts it. What you actually want is USS — unique set size — or PSS — proportional set size. USS tells you how much memory only that process is using. Luna: And how do you get USS on a running system? Lucas: You can use 'smem' or parse /proc//smaps. Smaps gives you detailed breakdowns of each mapping, and you can sum the Private_Clean and Private_Dirty fields. That's your USS. For that Node.js process, the USS was climbing by about 200 megabytes per hour — but the RSS looked almost flat because it was sharing library pages with other Node processes. Luna: So the leak was hidden in plain sight because the metric everyone watches — RSS — wasn't moving much. Lucas: Exactly. That's why you need to graph USS for critical services. Now, for the actual deep dive — once you suspect a leak — the most reliable tool is valgrind's memcheck. It instruments every memory allocation and deallocation and can pinpoint exactly where memory is allocated but never freed. Luna: Valgrind is great, but it slows down the process by a factor of 10 or more. You can't run it in production. Lucas: No, you absolutely cannot. But you can run it on a staging server with production traffic replayed. That's what we did. We recorded a few minutes of real bid requests, replayed them against a staging instance under valgrind, and within 20 minutes we had a leak report showing exactly the line number where the reference was being held. Luna: That's smart. So the diagnosis process is: graph /proc/meminfo available, graph per-process USS for critical services, and if something looks off, use valgrind in staging with replayed traffic. Lucas: That's the skeleton, yes. But I want to add one more thing: the 'slowly shrinking available memory' curve. It doesn't look like a straight line — it's a step function. Each step corresponds to a request spike. So on your graph, you'd see available memory drop by, say, 500 megabytes each time traffic surges, and it never recovers. That's the signature of a leak. Luna: Right, because normal memory usage goes up and down with load — the garbage collector frees it. A leak goes up and stays up. Lucas: Precisely. And if you graph the rate of change of available memory, you can set an alarm when the slope is consistently negative over a 24-hour window. That would catch a slow leak days before it hits critical. Luna: So the actionable takeaway here is: graph available memory, graph USS for your critical processes, and set a slope alarm. That's a solid playbook. Lucas: Yeah. And one more habit: after any deployment that involves new code, run a quick valgrind check on a staging instance with synthetic load. It takes an hour, and it catches leaks before they ever reach production. Luna: That's a great operational practice. I want to add a quick note about tools: besides valgrind, there's also 'heaptrack' for C++ and 'memwatch' for Node.js. Both are lighter weight. Lucas: Good additions. Heaptrack is excellent for C++ — it gives you a flame graph of allocations. But for interpreted languages, the principle is the same: you need to track allocations over time, not just snapshots. Luna: If today's episode was useful to you and you want to keep it ad-free, buy me a coffee dot com slash fexingo helps. Lucas: Yeah, it really does. Okay — back to the diagnosis flow. So after you've identified the leaking process, the next step is to figure out what data structure is growing. For that, you can take a heap dump. Luna: In Node.js, that's 'node --heapsnapshot-signal' to enable, then send a SIGUSR2 to dump the heap. Then load the snapshot into Chrome DevTools. Lucas: Exactly. And you can compare two snapshots taken hours apart. The diff will show you which object types are growing. In that ad-tech case, the diff showed 'Object' instances growing — which pointed to the closure-captured response object. Without the heap diff, you'd be guessing. Luna: So the full diagnostic chain is: system-level via /proc/meminfo, process-level via USS, allocation-level via valgrind or heaptrack, and object-level via heap snapshots. Lucas: That's it. And each layer rules out certain causes. System-level tells you something is wrong. Process-level tells you which process. Allocation-level tells you the code pattern. Object-level tells you the exact data. Luna: I want to emphasize one thing: don't jump straight to valgrind. Start with the system metrics. A lot of 'memory leaks' turn out to be misconfigured caches or connection pools that are working as designed but just sized wrong. Lucas: Great point. I've seen cases where someone increased the database connection pool from 10 to 200, and the memory grew proportionally. That's not a leak — that's a configuration error. The difference is that a leak's growth is unbounded, while a cache grows to a limit and then stabilizes. Luna: Exactly. So if you see memory growing and then plateauing, it's probably a cache. If it keeps growing without bound, it's a leak. Lucas: And that plateau test is one of the easiest checks you can do. Just watch the graph for 48 hours. If it flattens, you're probably fine. If it keeps climbing, start the diagnostic chain. Luna: What about tools like 'top' or 'htop'? Are they useful for this? Lucas: They're useful for a quick sanity check, but they don't give you the resolution you need. 'top' shows RSS by default, and as we said, RSS can be misleading. 'htop' can show PSS if you enable it in the setup, but it's still a snapshot, not a trend. Luna: So the real answer is: set up time-series monitoring for these metrics. Grafana with Prometheus, or even a simple script that logs to a file and you graph with spreadsheets. Lucas: Right. For small shops without full observability, a cron job that writes /proc/meminfo and /proc//smaps to a file every minute, then import into Excel, will catch leaks. It's not elegant, but it works. Luna: Okay, let's round this out with a concrete checklist. If you suspect a memory leak tomorrow, what do you do in order? Lucas: Step one: graph available memory from /proc/meminfo over the last 24 hours. If the slope is negative, proceed. Step two: identify the top memory consumers by USS. 'ps aux --sort -%mem' is a start, but use 'smem' for USS. Step three: for the suspicious process, take two heap snapshots an hour apart. Step four: if the diff shows unbounded growth, run valgrind on staging with replayed traffic to find the exact allocation site. Luna: And step five: fix the code, deploy, and verify that the slope flattens. Lucas: Exactly. And then set up an alarm so you never have to do this detective work again. Luna: That's the kind of process that separates reactive sysadmins from proactive ones. Lucas: Yeah. And honestly, most of this is just about looking at the right numbers. The tools are free and built into Linux. It's knowing which metric to trust that makes the difference.