Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Performance Tuning with cgroups v2
Transcript
- Lucas: So we've talked a lot about systemd, namespaces, and audit logging on this show, but one thing we haven't touched yet is control groups — specifically cgroups version two, which is now the default on Ubuntu 22.04 and later, RHEL 9, and pretty much every modern distro. Luna: Right, cgroups v2 has been out for a while, but a lot of people are still running v1 out of habit or because older tools expect it. Lucas: Exactly. And the thing is, if you're managing a server with multiple tenants or containers, cgroups v2 gives you a much cleaner interface for limiting CPU, memory, and I/O. It's also what Docker and Podman use under the hood now. Luna: So what's the big difference from v1? I remember v1 having separate hierarchies for each resource — it could get messy. Lucas: That's the key improvement. In cgroups v1, you had one hierarchy for CPU, another for memory, another for devices — all mounted separately under /sys/fs/cgroup. It was flexible but complicated, especially when a process needed to be in multiple groups. cgroups v2 uses a single unified hierarchy. One tree, one set of controllers, and every cgroup can enable or disable controllers independently. Luna: And that unified tree makes it easier to enforce consistency — no more weird interactions between different hierarchies. Lucas: Right. Plus there's the 'no internal processes' constraint. In v2, only leaf nodes — cgroups that don't have child cgroups — can contain processes. That prevents a common v1 mistake where a parent group had processes and also child groups, and the accounting got fuzzy. Luna: So it's a bit more rigid, but that rigidity forces you to think clearly about how you structure your resource pools. Lucas: Exactly. Now let's walk through a concrete example. Say you have a database process — PostgreSQL, for instance — that's eating up too much memory and CPU on a shared server. You want to cap it so it doesn't starve other services. Luna: What's the first step? Make sure cgroups v2 is enabled. Lucas: Yeah, you can check with 'mount | grep cgroup'. If you see cgroup2 on /sys/fs/cgroup, you're on v2. If not, you might need to add 'systemd.unified_cgroup_hierarchy=1' to your kernel boot parameters. But most modern distros ship with v2 by default now. Luna: Okay, so let's say it's enabled. How do you actually create a cgroup for PostgreSQL? Lucas: The simplest way is through systemd, since it's the init system and already manages cgroups for services. You create a drop-in file for the PostgreSQL service — for example, 'systemctl edit postgresql' — and add resource limits under the section. Something like 'CPUQuota=50%' and 'MemoryMax=2G'. Luna: And that's persistent across reboots, right? Because systemd applies it on service start. Lucas: Exactly. Then you restart the service and verify with 'systemd-cgtop' — that shows live resource usage per cgroup. You can also check /sys/fs/cgroup/system.slice/postgresql.service/ for the actual files like cpu.max and memory.max. Luna: But what if you want to limit a process that isn't managed by systemd? Say a legacy script or a user's shell? Lucas: You can do it manually. Create a directory under /sys/fs/cgroup, like /sys/fs/cgroup/mydb/. Then write to the controller files. For example, 'echo "50000 100000" > /sys/fs/cgroup/mydb/cpu.max' limits CPU to 50%. Then write the PID to cgroup.procs. But you need to have root, and you need to enable the controllers first. Luna: Wait, enable controllers? Isn't that automatic? Lucas: Not in v2. By default, the root cgroup has all controllers enabled, but when you create a child cgroup, you have to explicitly enable the controllers you want by writing to cgroup.controllers and cgroup.subtree_control. It's a security measure — you can't give a child group a controller that the parent hasn't delegated. Luna: So the parent controls what the child can use. That makes sense — it prevents a rogue process from creating its own cgroup and bypassing limits. Lucas: Right. And it also makes delegation safer. If you're running a container runtime like Podman in rootless mode, the runtime can create its own cgroup subtree under the user's slice, but only controllers that the parent allows. Luna: I've heard that memory limits work differently in v2 — no more soft limits? Lucas: That's correct. In v1, you had memory.soft_limit_in_bytes and memory.limit_in_bytes. In v2, it's simplified to memory.max — the hard limit — and memory.high, which is a throttle threshold. When memory usage exceeds memory.high, the kernel starts reclaiming memory aggressively, but it doesn't OOM kill immediately. Luna: So memory.high is like a soft limit, but with actual pressure instead of just a hint. Lucas: Exactly. And there's also memory.low and memory.min for protection — to guarantee a minimum amount of memory to a cgroup. That's useful for critical services like SSH or monitoring agents. Luna: Let's talk about I/O. Does cgroups v2 have a unified I/O controller? Lucas: Yes, it's called io.max for limiting bandwidth and IOPS, and io.weight for proportional weight-based control. But there's a catch: it only works for direct I/O and buffered writes. Reads from the page cache aren't throttled by the I/O controller — that's a known limitation. Luna: So if your database does a lot of cached reads, you can't limit that with cgroups v2 alone? Lucas: Not easily. You'd need to combine it with other tools like ionice or use dm-cache or something. But for write-heavy workloads, it works well. Luna: What about the pressure stall information — PSI? I think that's a v2 thing too. Lucas: Yeah, PSI files — memory.pressure, cpu.pressure, io.pressure — are available in each cgroup directory. They show a percentage of time that tasks were stalled due to resource contention. It's a great way to proactively detect bottlenecks before they become critical. Luna: So you could set up monitoring to alert when pressure exceeds a certain threshold? Lucas: Exactly. For example, if memory.pressure.some stays above 10% for five minutes, you might want to increase memory limits or add swap. Luna: I want to go back to practical use. If I have a production server with multiple microservices running as systemd units, what's the best practice? Do you set limits on each service individually? Lucas: That's one approach. But you can also create slice units — systemd's way of grouping services. For example, you could have a 'database.slice' that contains PostgreSQL and Redis, and then set aggregate limits on the slice. That way you don't have to tune each service individually. Luna: And that slice becomes a cgroup automatically? Lucas: Right. You create a file like /etc/systemd/system/database.slice with section, set CPUQuota and MemoryMax there, then set the 'Slice=database.slice' property in each service unit. Systemd handles the rest. Luna: Okay, but what about Docker containers? Are they just using cgroups under the hood? Lucas: Yeah, Docker uses cgroups to enforce the --cpus, --memory, and --blkio-weight flags. On a cgroups v2 host, Docker uses the unified hierarchy. You can see it with 'docker inspect' or by looking at /sys/fs/cgroup/system.slice/docker-<containerid>.scope/. Luna: So if you're already using Docker, you might not need to touch cgroups directly — but understanding how it works helps you debug resource issues. Lucas: Exactly. And there are times when you need to adjust limits at runtime. For example, if a container is hitting its memory limit, you can write a higher value to memory.max in its cgroup directory without restarting the container. Luna: And that's safe? Live tuning? Lucas: In my experience, yes. The kernel picks it up immediately. But you have to be careful with CPU — changing cpu.max can cause throttling to kick in instantly, which might affect latency-sensitive apps. Luna: Let's talk about a real-world scenario where cgroups v2 saved your bacon. Lucas: I had a client running a Java application that was leaking memory — it would slowly grow until the OOM killer killed it or a critical service. They were restarting it every night via cron. Luna: A nightly restart — that's a band-aid, not a fix. Lucas: Right. So I put that Java service into a systemd service with MemoryMax=4G and MemoryHigh=3G. The service would get throttled at 3G, giving the JVM time to GC, and if it still grew, it would hit the 4G hard limit and get OOM killed — but only that service, not the whole system. Luna: And did it work? Did the memory leak still cause problems? Lucas: It still got killed once a week, but the rest of the system stayed stable. They eventually fixed the leak, but cgroups bought them time. And because we used memory.high, the kernel had time to throttle before the kill, so the service didn't just vanish — it gracefully degraded. Luna: That's a great example of using the high limit as a warning system. Lucas: Exactly. And you can couple it with pressure stall monitoring — set up a script that watches memory.pressure and sends an alert when it goes above 50%. Then you know something's off before it hits the limit. Luna: Okay, one more question: what about swap? Does cgroups v2 handle swap differently? Lucas: Yes, there's a separate memory.swap.max file. By default, if you set memory.max, swap is still unlimited — a process could use swap to bypass the memory limit. So if you want to limit total memory plus swap, you set memory.swap.max to a value equal to memory.max plus allowed swap. Luna: That's a common gotcha — people think setting memory.max covers swap too, but it doesn't. Lucas: Yeah, I've seen servers get into swap thrashing because someone set a memory limit but left swap unlimited. The process just swapped out and kept consuming resources. Luna: So for production, you probably want to set both, or disable swap entirely if you can. Lucas: Exactly. Or at least put it on a fast SSD. Luna: You know, we've covered a lot of ground — from systemd slices to manual cgroup creation to Docker integration. If a listener wants to get hands-on, what's one thing they should try today? Lucas: I'd say take a service you run — maybe a web server or a database — and use systemd to set a CPUQuota and MemoryMax on it. Then run a load test and watch systemd-cgtop. You'll see the cgroup in action, and you'll understand how the kernel enforces those limits. It's a great confidence builder. Luna: And if they want to go deeper, check out the kernel documentation under Documentation/admin-guide/cgroup-v2.rst — it's actually readable. Lucas: It's one of the better kernel docs. And if today's tech conversation gave you something usable, a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that literally funds making this many episodes possible. No ads, just listener support. Luna: Yeah, it's a small group that keeps it going — and it means we can keep these shows ad-free and focused on what matters. Lucas: Exactly. So back to cgroups — one final tip: if you're ever unsure whether your system is using v1 or v2, just check the mount. If you see cgroup2, you're on v2. If you see multiple cgroup mounts, you're on v1. And if you're on v1, it's worth planning a migration. Luna: And most container runtimes now support v2, so there's really no reason to stay on v1 for new deployments. Lucas: Right. We'll link to a migration guide in the show notes. Until next time, keep your servers stable and your resource limits tight.