Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose Linux Server Memory Leaks
Transcript
- Lucas: So you set up a new web service on your Linux server. It runs great for the first day. By day three, response times are crawling. By day five, the OOM killer shows up and takes out a process you actually need. That's the classic memory leak — and it's still one of the most common production headaches I see. Luna: Yeah, and the annoying part is it's not always obvious. The process might look fine for hours, and then over days the memory usage just keeps climbing. Lucas: Exactly. And if you're running a language with automatic memory management — JavaScript, Python, Ruby — people sometimes assume leaks can't happen. But they absolutely can. A forgotten reference in a closure, a global cache that never evicts, a listener that never unregisters — you've got a slow drain. Luna: And if this episode gives you a practical way to catch that drain before it kills your users' session, maybe toss a coffee our way. We keep this show ad-free thanks to listener support at buy me a coffee dot com slash fexingo. Lucas: Yeah, it really helps keep the lights on. So — first step in diagnosing a leak. You need to confirm it's a leak and not just a process that legitimately needs more memory over time. Luna: Right. So how do you tell the difference? Lucas: You look at the resident set size over time. RSS is the portion of memory held in RAM. If you sample it every hour and it's monotonically increasing without ever plateauing, that's a leak. A process that legitimately grows — like a database caching queries — will eventually level off when its cache is warm. Luna: So you need a baseline. What's the simplest way to capture that data? Lucas: One-liner in a cron job. Something like: 'ps aux --sort -rss | head -10 >> /var/log/topmem.log'. Or if you want per-process RSS, you can use 'ps -eo pid,rss,comm --sort -rss'. Log that every five or ten minutes. Luna: And then you watch the log. If PID 1234's RSS goes from 50 megabytes to 200 to 800 over a day, you've got your suspect. Lucas: Right. Now, the next question is: what's inside that 800 megabytes? That's where /proc gets interesting. You can check /proc/PID/maps for memory regions, or /proc/PID/smaps for detailed per-mapping RSS. Luna: smaps is powerful but it can be a lot of output. There's a tool called smem that aggregates it nicely — gives you USS and PSS. Lucas: Yeah, smem is great. USS — unique set size — is the memory that's only used by that process and would be freed if it died. PSS — proportional set size — splits shared libraries proportionally. For leak hunting, I usually watch RSS because it's simpler, but PSS gives you a more accurate picture of actual RAM pressure. Luna: Once you've identified the process, how do you pin down the leak source? Lucas: For compiled languages like C or C++, valgrind with memcheck is the gold standard. You run 'valgrind --leak-check=full./your-program' and it reports every allocation that wasn't freed. But for interpreted languages, you need language-specific tools. Luna: Right. For Node.js, there's the built-in inspector. You can start the process with '--inspect' and then take heap snapshots using Chrome DevTools. Compare two snapshots taken a few hours apart and look for objects that grew in count or retained size. Lucas: For Python, there's tracemalloc. You enable it with 'tracemalloc.start' at the beginning of your script, and then you can take snapshots and compare them. It gives you a traceback of where each allocation was made. Luna: But what if you can't restart the process? Or it's a production server and you can't run valgrind because it slows things down too much? Lucas: Then you use /proc and hope the leak is large enough to spot. You can also use /proc/PID/status to see VmRSS and VmPeak. If VmPeak is much larger than VmRSS, the process once allocated and freed a lot of memory — that's normal. But if VmRSS keeps climbing toward VmPeak, that's a leak. Luna: So you're essentially using the OS as your leak detector. Lucas: Exactly. And once you've confirmed a leak, you need to stop the bleeding before you fix the root cause. The quickest way is to set a memory limit using systemd or cgroups. Luna: Yeah, systemd makes it easy. If your service is managed by a unit file, you add 'MemoryMax=500M' or whatever limit you want, then 'systemctl daemon-reload' and 'systemctl restart your-service'. The process won't be able to exceed that limit — it'll get killed and restarted. Lucas: Cgroups v2 is even more granular. You can set 'memory.max' in the cgroup directory. And you can combine that with 'memory.high' which triggers reclaim before hitting the hard limit. That way the process slows down instead of getting killed abruptly. Luna: Right, and you can monitor memory usage per cgroup with 'memory.current'. So you get a cleaner picture than /proc sometimes. Lucas: Now, if you want to be proactive instead of reactive, you should integrate memory leak detection into your CI pipeline. For Node.js, you can run the '--gc-global' flag and take heap snapshots before and after a test suite. If the heap grows by more than, say, 5 percent, fail the build. Luna: That's a really good practice. For Python, you can do the same with tracemalloc in your unit tests. Assert that the difference between two snapshots is below a threshold. Lucas: The key is to treat memory leaks like any other bug. You need visibility, a repeatable way to reproduce the growth, and a hard limit to contain damage in production. Luna: And always log the OOM events. The kernel logs them to dmesg and journald. If you see 'Out of memory: Killed process' in your logs, you know you've got a problem. Lucas: Exactly. So to wrap it up: sample RSS over time, use smem or /proc to zoom in, use language-specific tools to find the source, and set systemd or cgroup limits to prevent crashes in the meantime. Luna: And if you catch the leak before it catches you, you'll save yourself a lot of late-night pages. Lucas: Absolutely. Next time we'll talk about using eBPF to trace memory allocations in real time — that's a whole other level. Luna: I'm looking forward to that.