Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux cgroups v2 for Server Resource Control
Transcript
- Lucas: If you manage a Linux server that runs more than one thing — and let's be honest, that's most of us — you've probably had that moment where one process eats all the CPU or memory and brings everything else down. Luna: The noisy neighbor problem. Classic. Lucas: Right. And the tool for solving that is control groups — cgroups. But the version most of us grew up with, cgroups v1, had some design warts. The kernel community has been pushing v2 for years now, and on modern distros like Ubuntu 24.04 and RHEL 9, it's the default. Luna: So what makes v2 different? I've heard it's simpler, but also stricter. Lucas: Both true. The big change is the unified hierarchy. In v1, you had separate controllers for CPU, memory, IO — each of them could be mounted in different places. If a process was in a memory cgroup but not in a CPU cgroup, the accounting got confusing. V2 puts everything under a single hierarchy, typically at /sys/fs/cgroup. And a process can only be in one cgroup at a time for all controllers. Luna: So no more orphaned processes falling through the cracks. Lucas: Exactly. And v2 also enables proper delegation. If you're running containers, you can hand off a subtree of the cgroup tree to an unprivileged process — like Docker or Podman — without giving it root access to the whole system. That was tricky with v1. Luna: Okay, let's get concrete. Say I have a web server — Nginx — and a background Python script that sometimes goes wild. How do I keep the web server responsive? Lucas: Great scenario. First, check what cgroup version you're on. Run 'mount | grep cgroup'. If you see a line like 'cgroup2 on /sys/fs/cgroup type cgroup2', you're on v2. If it says 'tmpfs' with 'cgroup' type, that's v1. Luna: Got it. And if I'm on v2, I can use systemd slices, right? Lucas: Yes. systemd integrates deeply with cgroups v2. By default, every service runs in its own cgroup under the systemd slice hierarchy. You can create a custom slice for your web server and a separate one for batch jobs. Let's walk through it. Lucas: Create a slice for Nginx. In /etc/systemd/system/web.slice, add a file with 'Slice=web.slice' in the section. Then assign the Nginx service to it by running 'systemctl set-property nginx.service Slice=web.slice'. Luna: And then set resource limits on the slice itself? Lucas: Right. You can set CPU shares, memory max, IO limits. For CPU, use 'CPUQuota=' as a percentage of a core. If you want Nginx to use at most 200% of one core — meaning two full cores — you'd run 'systemctl set-property web.slice CPUQuota=200%'. Luna: And for memory? Say I want to cap it at 1 gigabyte. Lucas: That's 'MemoryMax=1G'. But note: in v2, 'MemoryMax' is the hard limit. If the process exceeds it, the kernel invokes the OOM killer. You can set a softer limit with 'MemoryHigh' — that starts throttling before the hard cap. Luna: Nice. So I'd use MemoryHigh for a warning-level limit and MemoryMax for the actual ceiling. Lucas: Exactly. Now for IO throttling. This is where v2 really shines. In v1, the IO controller had a separate hierarchy and was often disabled by default. In v2, it's part of the unified tree. You can set a maximum read and write bytes per second on a per-device basis. Lucas: For example, to limit the web slice to 50 megabytes per second read on /dev/sda, you'd do 'systemctl set-property web.slice IOReadBandwidthMax="/dev/sda 50M"'. Same for write — 'IOWriteBandwidthMax'. Luna: What about IOPS? Can I limit by operations per second instead of bandwidth? Lucas: You can. It's 'IOReadIOPSMax' and 'IOWriteIOPSMax'. So if your backup script is hammering the disk with small random reads, you can cap the IOPS to, say, 1000. That keeps the disk responsive for the web server. Luna: Let's say my Python script is the noisy neighbor. How would I isolate it? Lucas: Create another slice — call it 'batch.slice'. Set its CPUQuota to 50% and MemoryMax to 512M. Then move the Python service into that slice. But a cleaner approach is to run the script under a scope. systemd-run lets you launch a command in its own cgroup scope that you can control dynamically. Lucas: So: 'systemd-run --scope -p CPUQuota=50% -p MemoryMax=512M python3 heavy_script.py'. That creates a transient scope, and you can even change the limits while it's running with 'systemctl set-property'. Luna: And those limits are enforced immediately? Lucas: Yes. No restart needed. That's the beauty of cgroups v2 — dynamic adjustment without downtime. Luna: What about delegation for containers? I know Docker uses cgroups, but does it work differently with v2? Lucas: Docker supports cgroups v2 since version 20.10. But the key change is the delegation model. In v2, the container runtime needs to own a sub-hierarchy. When you install Docker on a v2 system, systemd creates a 'docker' slice, and Docker delegates a subtree of that to the containerd process. The container processes inside can't escape to the parent cgroup because the kernel enforces the hierarchy. Lucas: That's a huge security improvement. In v1, a container could sometimes write to a parent cgroup file if the permissions were loose. In v2, the delegation is explicit — the parent writes the 'cgroup.subtree_control' file to hand off controllers. Luna: So how does that work in practice? Say I want to let my container set its own CPU limits. Lucas: You enable the 'cpu' controller in the parent cgroup by writing '+cpu' to 'cgroup.subtree_control'. Then the child cgroup can set values like 'cpu.max'. The container runtime does this automatically, but if you're writing your own manager, you need to understand that handshake. Luna: Let's talk about tools. Besides systemd, what can I use to inspect cgroups? Lucas: The classic tool is 'systemd-cgtop'. It shows a top-like view of cgroup resource usage — CPU, memory, IO. There's also 'systemd-cgls' to show the tree hierarchy. And directly reading the files: 'cat /sys/fs/cgroup/<path>/cpu.stat' gives you usage and throttling stats. Luna: Any gotchas with v2 that catch people off guard? Lucas: One big one: the 'memory' controller in v2 doesn't have a separate 'kmem' limit. In v1, you could limit kernel memory separately. In v2, kernel memory is accounted in the same limit, so if your process uses a lot of kernel memory — like mounting many filesystems — it counts toward the same cap. Also, swap is accounted by default. You can disable swap accounting by setting 'memory.swap.max = 0'. Luna: And the CPU controller — no more 'cpu.shares' weighting, right? Lucas: Right. v2 replaced shares with 'cpu.weight'. The default is 100, and it's a relative weight. So if one slice has weight 200 and another has 100, the first gets twice the CPU when there's contention. For hard caps, you use 'cpu.max'. Luna: Is there any reason to stay on v1? Lucas: Not really, unless you have legacy software that directly manipulates v1 cgroup files — older versions of LXC or some monitoring agents. Most modern tools support both. The kernel is moving toward removing v1 entirely. On RHEL 9, v1 is deprecated. Luna: Alright, let's do a real walkthrough. I want to set up a web.slice with limits for Nginx, and a batch.slice for a backup script. Show me the commands. Lucas: Assuming you have systemd 245 or newer — which you do on any recent distro. First, create the slice unit files. 'sudo mkdir -p /etc/systemd/system/web.slice.d' and 'sudo mkdir -p /etc/systemd/system/batch.slice.d'. Lucas: Then set properties directly with systemctl — the.d directories are for drop-in overrides. Actually, the easiest way is to just use set-property on the slice, which persists the settings. Run 'sudo systemctl set-property web.slice CPUQuota=200% MemoryMax=1G IOReadBandwidthMax="/dev/sda 50M" IOWriteBandwidthMax="/dev/sda 30M"'. Luna: And for the batch slice, maybe 50% CPU and 512MB memory? Lucas: Exactly. 'sudo systemctl set-property batch.slice CPUQuota=50% MemoryMax=512M'. Then assign your backup script's service to batch.slice. If it's a oneshot timer, you can set the Slice= explicitly in the unit file. Luna: How do I verify the limits are in place? Lucas: Check 'sudo systemd-cgls' to see the tree. Then look at the actual cgroup files: 'cat /sys/fs/cgroup/web.slice/cpu.max' should show '200000 100000' — meaning 200,000 microseconds out of 100,000 microsecond period, so two cores. For memory: 'cat /sys/fs/cgroup/web.slice/memory.max' shows '1073741824' — that's 1 gig. Luna: What if I need to adjust on the fly — say the backup is taking too long and I want to give it more CPU? Lucas: Just run 'sudo systemctl set-property batch.slice CPUQuota=75%'. The change applies immediately. No restart of the service needed. Luna: That's a killer feature for prod emergencies. Lucas: Absolutely. And that's really the core of cgroups v2 — precise, dynamic, hierarchical resource control. It's a fundamental building block for server stability. Luna: You know, this level of detail is exactly why a handful of listeners keep this show going. We don't run ads — not now, not ever — and that's only possible because some folks chip in monthly through buy me a coffee dot com slash fexingo. It's not a big ask, just a quiet acknowledgment that this kind of deep-dive server engineering takes time to produce. Lucas: Yeah, and it's genuinely appreciated. But back to the tech — one last tip: when you're testing limits, use 'stress-ng' to generate load. You can simulate CPU, memory, and IO stress, then watch how your cgroups enforce the caps. It's the best way to validate your configuration before a real incident. Luna: Good call. So to wrap: cgroups v2 is the default, it's simpler, and it gives you fine-grained per-service limits. Start by identifying your noisy processes, create slices, set CPU, memory, and IO limits, and monitor with systemd-cgtop. Lucas: And don't forget delegation — if you're running containers, v2 makes it safer. That single-hierarchy model fixes a lot of the edge cases that made v1 painful. It's one of those upgrades that, once you make the switch, you wonder why you waited.