Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Control Groups for Resource Isolation
Transcript
- Lucas: Alright, Luna — I want to talk about something that every sysadmin eventually runs into: one process on a server decides it's going to eat all the memory or peg the CPU, and suddenly your entire machine is thrashing. You've got MySQL fighting with Apache, a cron job goes rogue — it's chaos. Luna: Right, the classic noisy neighbor problem. You'd think by now the kernel would just prevent that automatically. Lucas: It actually can, with control groups — cgroups version 2. This is the kernel feature that lets you partition resources: CPU, memory, I/O, even PID limits. And it's built into systemd, so if you're on a modern distro like Ubuntu 24.04 or RHEL 10, you already have it. The trick is knowing how to configure it properly. Luna: And we should probably mention — if this kind of practical sysadmin content helps you keep your servers stable, a couple of dollars a month genuinely makes these episodes possible. Buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: Yeah, listener support is what keeps this ad-free and focused on real examples. So, back to cgroups. Let me set the scene: you've got a web server running Nginx and php fpm. One day a WordPress plugin goes haywire, spawns a bunch of PHP processes, and each one starts allocating memory like there's no tomorrow. Before you know it, the OOM killer kicks in and takes out your SSH daemon. Luna: I've been there. It's not fun. So how do you use cgroups to stop that? Lucas: First, make sure you're on cgroups v2. You can check by looking at /sys/fs/cgroup — if you see a file called cgroup.controllers, you're on v2. If you only see a bunch of subdirectories like memory and cpu, you might be on v1. Most distros have switched by now, but it's worth verifying. Luna: So what's the practical difference between v1 and v2? Lucas: v1 was a mess — different controllers hung off different hierarchies, and you couldn't enforce limits consistently. v2 unified everything under a single hierarchy, so one process can be limited on CPU, memory, and I/O at the same time. It's also designed to work with systemd, which manages slices for services. Luna: Alright, so we're on v2. Now how do we actually limit php fpm's memory? Lucas: The cleanest way is to use systemd's resource control directives. If php fpm is running as a systemd service — which it usually is — you can create a drop-in file to set limits. For example, add a file at /etc/systemd/system/php8.3-fpm.service.d/limits.conf with these lines:, MemoryMax=512M, MemoryHigh=384M. Luna: So MemoryMax is a hard limit and MemoryHigh is a soft throttle? Lucas: Exactly. MemoryHigh is a throttle — when the service exceeds that, the kernel will try to reclaim memory by swapping or dropping caches. MemoryMax is a hard wall — if it goes over, the OOM killer will kill a process inside the cgroup, but it won't touch anything outside. That's key: it contains the damage. Luna: So if php fpm hits 512 megabytes, the OOM killer takes out a PHP worker, but Nginx and SSH stay alive. That's way better than the system-wide OOM. Lucas: Right. And you can do the same for CPU. Say you don't want php fpm to use more than 50% of a single core. You'd add CPUQuota=50%. That's a relative percentage of one CPU. If you have 4 cores, 50% means half of one core, not half of all cores. To cap at two full cores, you'd set CPUQuota=200%. Luna: What about I/O? That one's trickier. Lucas: It is, because I/O is more complex. With cgroups v2, you have IOReadBandwidthMax and IOWriteBandwidthMax, but they apply to specific devices. You'd write something like IOReadBandwidthMax=/dev/sda 100M. That limits read throughput on sda to 100 megabytes per second. But honestly, I/O limits are less commonly used in practice because they can cause subtle performance issues. Luna: So let's say I want to monitor whether my limits are actually being hit. How do I check? Lucas: Systemd gives you a nice command: systemd-cgtop. It's like top but for cgroups — shows you CPU, memory, and I/O usage per slice. You can also look directly at the cgroup filesystem. Under /sys/fs/cgroup/system.slice/php8.3-fpm.service, there's a file called memory.current that shows current usage in bytes, and memory.events shows how many times you hit the limit. Luna: So memory.events would show a count of 'max' events if the service kept hitting the hard limit? Lucas: Exactly. It'll show 'max' for how many times MemoryMax was exceeded. That's a good indicator you need to tune the limit. Now, what if you don't want to tie limits to a systemd service? Maybe you're running a custom script or a container. Luna: Then you'd create your own cgroup hierarchy manually? Lucas: You can, but it's easier to use systemd's transient scopes. You can run a command under a temporary cgroup with systemd-run. For example: systemd-run --scope -p MemoryMax=256M -p CPUQuota=30%./my-script.sh. That runs the script in its own scope with those limits applied. When the script exits, the scope is cleaned up. Luna: That's elegant. And it works with containers too, right? Docker and Podman both use cgroups under the hood. Lucas: Absolutely. When you run docker run --memory=512m, Docker creates a cgroup for that container and sets the memory limit. With cgroups v2, Podman can even use systemd slices directly, which gives you better integration with the host's resource accounting. Luna: So if you're running containers in production, cgroups v2 is the foundation for resource isolation. It's not an optional extra. Lucas: Right. And the best part is, you don't need to be running containers to benefit. Any sysadmin can apply these limits to any service. If you have a backup script that runs nightly and sometimes hogs the disk, you can cpu quota it to 20% so the web server stays responsive. Luna: What about the swap? Can you limit swap usage per cgroup as well? Lucas: Yes, with MemorySwapMax. It works like MemoryMax but for swap. You might set MemorySwapMax=0 to disable swap for a critical service, ensuring it never gets swapped out and slows down. Or you could allow some swap for a batch job. The kernel enforces it based on the cgroup hierarchy. Luna: So the key takeaway is: know your cgroups version, use systemd drop-ins for persistent limits, and use systemd-run for one-off jobs. Lucas: Exactly. And remember, cgroups v2 also supports pressure stall information — PSI files that show you resource pressure in real time. You can alert on those before limits are even hit. It's a proactive approach. Luna: Alright, I'm convinced. Next time I set up a server, I'm adding cgroup limits to every non-critical service. Lucas: And if you ever have a question about a specific scenario — like how to limit a Java app with multiple threads — drop us a line. We might cover it in a future episode. For now, keep your servers stable and your neighbors quiet.