Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Tame Linux Kernel OOM Killer with Systemd
Transcript
- Lucas: Luna, I want to talk about something that's probably woken you up at 3 AM more times than you'd like. Luna: Go on—is this the OOM Killer again? Lucas: It is. The Linux kernel's Out-Of-Memory Killer. And I think most sysadmins have a love-hate relationship with it. On one hand, it prevents the entire system from freezing when memory runs out. On the other hand, it has this delightful habit of killing your most critical service instead of that rogue Python script that's leaking memory. Luna: Right. The classic—your PostgreSQL or your SSH daemon gets the axe, while the actual offender keeps chugging along. Lucas: Exactly. And the default behavior is essentially a black box heuristic. The kernel calculates an oom_score for each process based on things like memory usage, CPU time, and how long it's been running. Then it kills the one with the highest score. Luna: But that heuristic doesn't know that your PostgreSQL instance is more important than a cron job that's gone haywire. Lucas: Right. So what we need is a way to tell the kernel: 'Hey, protect this service. If you have to kill something, start with these less important ones.' And that's exactly what the oom_score_adj interface does. Luna: We should probably start by explaining how the scoring works, because I think a lot of people don't dig into the /proc filesystem for this. Lucas: Absolutely. Every process has a file at /proc/<pid>/oom_score. That's the raw score the kernel computes. It's read-only. But there's also /proc/<pid>/oom_score_adj, which is writable. You can set a value between -1000 and +1000. A value of -1000 means the process is immune to the OOM Killer. A value of +1000 makes it the first to be killed. Luna: So for your critical services, you'd set a negative value. For your batch jobs or Web crawlers, maybe a positive one. Lucas: Precisely. But the problem is, manually setting these across hundreds of processes is tedious. And you have to do it every time a process restarts. That's where systemd comes in. Luna: Systemd has the OOMD feature now, right? Lucas: Yes. Starting with systemd v243, there's a built-in OOM manager called systemd-oomd. It integrates with the kernel's PSI—Pressure Stall Information—to detect memory pressure and then act based on cgroup-level policies. You don't have to mess with individual PIDs anymore. Luna: So you can define rules like 'if memory pressure exceeds 60 percent for more than 10 seconds, kill the lowest-priority cgroup first'. Lucas: Exactly. And you set the priority per service unit. In the service file, you add OOMScoreAdjust=—that's the same as writing to oom_score_adj, but systemd applies it when the service starts. And then you use ManagedOOMSwap= and ManagedOOMMemoryPressure= to tell systemd-oomd how to handle that service. Luna: Let's walk through a practical example. Say I have a PostgreSQL database and a Web scraper that runs daily. Lucas: Perfect. For PostgreSQL, you'd set OOMScoreAdjust=-500, ManagedOOMMemoryPressure=kill, and ManagedOOMSwap=kill. That tells systemd: 'If memory pressure gets high, this service is eligible to be killed, but only after other services with higher OOM scores.' For the Web scraper, you might set OOMScoreAdjust=500, so it's one of the first to go. Luna: But there's a nuance, right? The default value for ManagedOOMMemoryPressure is 'auto', which uses the cgroup's memory pressure info. You have to set it explicitly to 'kill' to make it actually eligible for termination by oomd. Lucas: That's an important detail. If you don't set ManagedOOMMemoryPressure=kill, systemd-oomd won't touch that service even if memory pressure is through the roof. It'll look elsewhere. So for your essential services, you might not even set that flag—you just set a low OOM score and let them be protected by default. Luna: I've seen a real-world case where a company had a misconfigured PostgreSQL OOM score. The DBA had set OOMScoreAdjust=0, which is the default. But the Web server had a score of -100 because someone thought that would make it immune. When a traffic spike hit, the OOM Killer took out PostgreSQL—because it had a higher score than the Web server—and the whole site went down. Lucas: That's a classic mistake. Setting a negative score without understanding the relative ordering. If you set it to -100, you're still in the pool. Only -1000 makes you truly immune. And even then, immunity isn't always wise—you want the kernel to have some flexibility. Better to set your most critical services to, say, -800 or -900, so they're virtually immune but not completely locked out. Luna: Let's talk about the monitoring side. How do you check what OOM scores are set currently? Lucas: The quickest way is to run 'ps aux --sort=-oom' or look at /proc/; do echo $ $ $; done' to get a quick table. But for systemd services, you can use 'systemctl show <service> | grep -i oom' to see the configured values. Luna: And for systemd-oomd specifically, you can check its logs with 'journalctl -u systemd-oomd' to see what decisions it's made. Lucas: Right. So the key takeaway is: don't leave your OOM Killer configuration to chance. By default, the kernel's heuristic might not prioritize the way you want. Use OOMScoreAdjust in your systemd service files. Decide which services are expendable and which are critical. And if you're on a distro with systemd v248 or later, consider enabling systemd-oomd for more granular control. Luna: I think a lot of people don't realize that systemd-oomd is a separate daemon that needs to be installed and enabled. It's not on by default in most distributions. Lucas: That's true. On Fedora and RHEL 9+, it's available but might not be active. You need to install the systemd-oomd package if it's not already, then enable and start the service. And you also need to configure the oomd.conf file—usually in /etc/systemd/oomd.conf—to set global thresholds like DefaultMemoryPressureLimit and DefaultMemoryPressureDuration. Luna: Let's talk about a real scenario. Let's say you have a dedicated monitoring agent—say, Prometheus or Nagios—that must stay alive. How would you protect it? Lucas: I'd set OOMScoreAdjust=-900 in the service unit. That gives it a very low score. I'd also set ManagedOOMMemoryPressure=kill and ManagedOOMSwap=kill, but with such a low score, it would be among the last to be killed anyway. The idea is that if everything else is gone, the kernel still has the option to take it out if it's the only way to prevent a system hang. Luna: And for something like a batch data processing job that you can easily restart? You'd set a positive score. Lucas: Yeah, OOMScoreAdjust=500 or even 800. And you'd set ManagedOOMMemoryPressure=kill and ManagedOOMSwap=kill so that oomd can terminate it proactively before the system gets into trouble. Luna: One thing I've noticed is that swap configuration also affects OOM behavior. If you have swap enabled, the kernel might swap out pages instead of invoking the OOM Killer, which can delay the inevitable but also cause performance degradation. Lucas: That's a good point. Systemd-oomd's ManagedOOMSwap= option lets you decide whether to kill processes when swap is under pressure. If you set it to 'kill', oomd will act when swap usage is high, even if memory pressure is low. This can prevent the system from thrashing. Luna: So the combination of OOMScoreAdjust for priority and ManagedOOM* for policy gives you a lot of control. Lucas: Exactly. And the best part is it's declarative—you define it once in your service file, and it persists across restarts. No more cron jobs that adjust scores after boot. Luna: One more thing—what about containers? If you're running Docker or Podman, does this apply? Lucas: It does, but with caveats. For container runtimes, you typically set OOM score adjustments via Docker's --oom score adj flag or in the container's resource limits. But systemd-oomd works at the cgroup level, so if your containers are managed by systemd—like with podman or systemd-nspawn—you can apply the same ManagedOOM* settings. Luna: So it's not a silver bullet for containerized environments, but it's a solid start for bare-metal or vm based services. Lucas: Right. And if you're in Kubernetes, you'd rely on the pod's QoS classes and resource limits instead. But for traditional Linux servers, this is the best way to tame the OOM Killer. Luna: This is exactly the kind of practical advice that can save a sysadmin a lot of late-night debugging. We put these episodes together because we believe solid, ad-free tech knowledge should be easy to access. Lucas: And we deliberately keep the show free of sponsors. If you find these conversations useful and want to support that choice, you can find us at buy me a coffee dot com slash fexingo. Luna: No pressure—just a way to keep the server lights on, so to speak. Lucas: Now, back to the OOM Killer. Let me give you one final recommendation: audit your existing services. Run that loop I mentioned to see which processes have high oom_scores. Then decide which ones you want to protect. It's a simple change that could save you a major outage. Luna: And test it. Set a service to -1000 and then try to trigger OOM—you'll see it survives. Then set another to +1000 and watch it get killed. It's a good way to build confidence in your configuration. Lucas: Great idea. And if you're using systemd-oomd, you can simulate memory pressure with tools like stress-ng to verify your policies work. Luna: Alright, I think we've given listeners enough to go on. Next time, maybe we'll dive into the PSI subsystem itself—how pressure stall information works under the hood. Lucas: That's a good one. I'll start gathering some metrics. For now, go update those OOMScoreAdjust values.