Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Control Groups for Resource Limits
Transcript
- Lucas: Alright, let's talk about a situation almost every sysadmin has faced. You're running a shared web server — maybe a handful of client sites — and one of them goes viral, or gets hit by a bot flood. Suddenly your php fpm workers start gobbling RAM, the system starts swapping, and before you can SSH in, the OOM killer has already taken out your database. Luna: Yeah, the classic noisy neighbor problem. And the usual fix is just trusting the kernel's OOM score, which is basically a coin flip. Lucas: Exactly. But there's a much cleaner solution built right into the Linux kernel: control groups, or cgroups. Specifically, cgroups v2 — which has been the default on most modern distros for a few years now. It lets you set hard limits on CPU, memory, and I/O per process group, so one runaway PHP pool can't starve the rest of the system. Luna: Right — and we're not talking about containers here. Cgroups work at the process scope, so you can use them directly with systemd or even manually. Lucas: Let's anchor this with a real example. Say you have a php fpm pool serving a WordPress site. The pool spawns, say, 10 workers by default. Each worker might consume 50 to 100 megabytes of RSS. One poorly optimized plugin — like a caching plugin that leaks memory — can push each worker to 300 megabytes. Suddenly your 10 workers are using 3 gigs of RAM. Luna: And if your server only has 8 gigs total, that's almost half the system for one site. Lucas: Right. So instead of letting that happen, you can create a systemd service scope for the entire php fpm pool and set a memory limit. With cgroups v2, that limit is enforced by the kernel itself — not by some userspace daemon that might be killed first. Luna: How do you actually set that up? Is it a systemd directive? Lucas: It is. If your php fpm service is managed by systemd — and it probably is — you can drop in an override file. So you'd run 'systemctl edit php-fpm.service'. Then add a block under that says 'MemoryMax=2G' to cap it at 2 gigabytes. That's it. systemd translates that into the cgroup v2 memory.max file under /sys/fs/cgroup. Luna: So the kernel enforces that limit. What happens when PHP tries to allocate more? Lucas: The allocation fails — malloc returns NULL, or the process gets a SIGKILL from the kernel's cgroup OOM killer. But here's the key: only processes in that cgroup get killed, not your database or SSH session. Luna: That's way better than a system-wide OOM. But what about CPU limits? Memory's only half the story. Lucas: Good point. For CPU, cgroups v2 uses a weight-based model by default, but you can also set a hard quota. The systemd directive is 'CPUQuota='. For example, 'CPUQuota=50%' means the entire php fpm pool gets at most half a core. It's a hard cap, so even if the server is idle, that pool won't burst above it. Luna: But that might throttle legitimate traffic, right? A better approach for CPU is often just using the weight — letting it burst when the system is free, but getting throttled under contention. Lucas: Exactly. cgroups v2 CPU weight is set via 'CPUWeight=' in systemd. The default is 100. If you give your PHP pool a weight of 50, and your database gets 200, then under contention the DB gets four times the CPU time. That's usually what you want — you don't want to artificially limit throughput during low load. Luna: And for I/O? Disk I/O bandwidth can also be a problem if a site does a lot of logging or file operations. Lucas: Yes — cgroups v2 also supports I/O limits via 'IOReadBandwidthMax', 'IOWriteBandwidthMax', and the corresponding IOPS limits. The systemd directives are 'IOReadBandwidthMax=', 'IOWriteBandwidthMax='. You specify a device path, like '/dev/sda', and a rate. For example, 'IOReadBandwidthMax=/dev/sda 100M' limits reads to 100 megabytes per second. Luna: So you can essentially say: this PHP pool can use at most 2 gigs of RAM, half a core of CPU, and 100 MB/s of disk reads. All enforced by the kernel. Lucas: Exactly. And you can monitor usage in real time with 'systemd-cgtop'. That shows you CPU, memory, and I/O per cgroup — similar to htop but organized by control group. If you have a service named 'php-fpm.service', you'll see its cgroup listed there with current usage. Luna: That's a great way to validate your limits are being hit. One thing to watch out for though: cgroups v2 has a unified hierarchy. You can't have separate hierarchies for CPU and memory like in v1. Lucas: Right — that's actually a simplification. In v2, all controllers are attached to the same tree. So if you want to limit both memory and I/O, you write to the same cgroup directory. systemd handles that for you, but if you're doing manual cgroup management, it's good to know. Luna: Manual management? When would someone want to do that? Lucas: If you're running a custom daemon that's not managed by systemd — maybe a legacy application or a container runtime that doesn't use systemd scopes. You can create cgroups directly by writing to the cgroup fs. For example, to create a subgroup under /sys/fs/cgroup/myapp, you mkdir that directory, and the kernel automatically populates the controller files. Luna: And then you write limits to files like memory.max, cpu.max, io.max. That's powerful but also easy to mess up — you need to be careful about delegation. Lucas: Absolutely. One common pitfall: not all controllers are enabled by default. You can check with 'cat /sys/fs/cgroup/cgroup.controllers'. On a typical server, you'll see cpu, memory, io, pids, and maybe others. But if you want to use the 'pids' controller to limit the number of processes, you need to enable it in the root cgroup. Luna: Right — and the root cgroup is special. You can't set limits on the root cgroup itself; you have to create child cgroups. That's why systemd creates a slice hierarchy: system.slice for system services, user.slice for user sessions, and so on. Lucas: Exactly. And you can create your own slices. For example, you could have a 'www.slice' that contains all your web-facing services, and set a total memory limit of 4 gigs for the whole slice. Then each php fpm pool inside that slice gets its own limit, and the slice itself acts as a hard ceiling. Luna: That's a clean architecture — hierarchical limits. The slice gets the aggregate, and the individual services are further constrained. Lucas: Let's take a step back and look at the bigger picture. Why use cgroups instead of just setting ulimits or adjusting PHP's memory_limit? Well, ulimits are per-process and inherited, and they don't aggregate across a pool. PHP's memory_limit is per-request — a single request can't exceed it, but 10 concurrent requests can each use that limit, potentially consuming 10 times the memory. Cgroups enforce across all processes in the group. Luna: So it's a true hard aggregate limit. That's the killer feature. And it works for any resource — not just memory. Lucas: Right. And it's not just for PHP. You can use it for any service. Maybe your log shipper is using too much CPU during peak hours. Drop a CPUQuota line in its systemd override. Or your backup script is saturating the disk I/O every night. Set an IOWriteBandwidthMax. Luna: Let's talk about a real-world scenario you mentioned earlier — the runaway PHP pool. Walk us through the commands to diagnose and fix that. Lucas: Sure. First, you'd notice the server load is high and memory is tight. Run 'systemd-cgtop' to see the top cgroups by memory. You'll likely see 'php-fpm.service' at the top. Then check its current memory usage: 'systemctl show php-fpm.service -p MemoryCurrent'. That gives you the current RSS of the entire cgroup. Luna: And then you set a limit with 'systemctl set-property php-fpm.service MemoryMax=2G' — that's runtime, no restart needed. Lucas: Exactly — and to make it persistent across reboots, you add the override file. The runtime command writes to the cgroup filesystem immediately, but it's not saved. The override file ensures it's applied on service start. Luna: One thing I've learned the hard way: if you set a limit too low, the service might start failing requests because it can't allocate memory. So you need to monitor and find a reasonable baseline first. Lucas: Good advice. A good approach is to start with a generous limit — say, 4 gigs — and use 'systemd-cgtop' over a week to see peak usage. Then set the limit at, say, 125% of that peak. That gives headroom but still protects against runaway growth. Luna: And for CPU, you can use 'CPUQuota' if you want a hard cap, but I prefer 'CPUWeight' for most services. It's more flexible. Lucas: Same. The only time I use hard CPU caps is for batch jobs like cron scripts that shouldn't interfere with interactive services. But for a web pool, weight is usually better. Luna: Alright — let's also mention that cgroups v2 is mandatory on some distros now. RHEL 9, Ubuntu 22.04 — they boot with cgroups v2 by default. So if you're still on v1 for legacy reasons, you might want to plan a migration. Lucas: Good point. The migration is mostly transparent if you use systemd. But if you have custom scripts that touch the cgroup filesystem directly, you'll need to update them for the unified hierarchy. Also, Docker and podman have supported v2 for a while. Luna: If today's deep dive was useful to you, the way these episodes stay ad-free is listener support. You can find us at buy me a coffee dot com slash fexingo. Every contribution helps keep the server lights on — and the coffee flowing. Lucas: Yeah, and it's a small way to say this kind of practical content matters. We appreciate everyone who chips in. Luna: So to sum up: cgroups v2 give you kernel-enforced, hierarchical resource limits that work across process pools. Use systemd directives like MemoryMax, CPUWeight, and IOReadBandwidthMax to tame noisy neighbors without restarting anything. Lucas: And the next time a PHP script goes haywire, you'll have a fighting chance. Or at least your database will. Thanks for listening — we'll see you next time on Linux Server Admin with Fexingo.