Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / When Your Server Swaps Itself to Death
Transcript
- Lucas: If you run a Linux server with swap enabled — and most of us do — there is a specific failure mode that does not look like a crash, does not trigger your standard CPU or memory alerts, and will quietly destroy performance until the process scheduler decides to murder something. Luna: You are describing the swap death spiral. Lucas: I am. And if you have never diagnosed one live, it is one of the more surreal experiences in server engineering. The machine responds to SSH, top shows 20 percent CPU, memory usage looks fine, but every command takes thirty seconds. Your database pool drops connections. Your web server starts returning 503s. And the logs are just silence. Luna: Silence because the kernel is too busy paging to write anything. Lucas: Exactly. So let's walk through a real case. A few months ago, a team I consult with had a Postgres server on Ubuntu 22.04. Standard setup — 64 gigs of RAM, 8 gigs of swap on an SSD, default kernel parameters. One afternoon an analytics query that should have scanned only the last week accidentally scanned the entire partition. That query alone demanded about 40 gigs of working set. The database buffer pool was already using 30. The kernel saw memory pressure and started swapping anonymous pages. Luna: Which is fine, that's what swap is for. The problem is what happens next. Lucas: Right. The system's default swappiness is 60. That means the kernel is fairly eager to swap even when there's plenty of free page cache. So as pressure ramps up, it swaps more aggressively. But now the SSD's IO queue fills with swap writes, and the reads — the actual query data — have to wait. The query slows down. It holds memory longer. The kernel swaps more to compensate. It's a classic positive feedback loop. Luna: And at some point the OOM killer fires. Did it hit Postgres? Lucas: It hit Postgres's main worker process. The database went down hard. Took them forty minutes to recover because the server was still thrashing on restart. The Postgres logs just showed 'terminated by signal 9' and the kernel logs showed 'Out of memory: killed process 1234 '. No indication that the real cause was swap misconfiguration. Luna: So what would you fix first? Swappiness? Lucas: Swappiness is the easiest lever. For any server running a database — Postgres, MySQL, anything with its own buffer pool — you want swappiness set very low. I use 10. Some people go as low as 1. The idea is: only start swapping when you are absolutely out of page cache options. You can set it live with sysctl vm.swappiness=10 and make it permanent in /etc/sysctl.d/99-swap.conf. Luna: Does that alone prevent the death spiral? Lucas: No. It buys you time. The kernel will still swap under extreme pressure, but it won't accelerate the thrash. The next layer is systemd-oomd. Ubuntu and Fedora ship it now — it's a userspace OOM manager that gets involved before the kernel's own OOM killer. You configure it to act on swap pressure rather than memory pressure. Luna: So systemd-oomd kills the runaway query, not Postgres itself. Lucas: Exactly. You set ManagedOOMSwap=kill in the slice for your database service. When swap usage exceeds a threshold — I set it at 90 percent — systemd-oomd kills the cgroup that is using the most swap. In our scenario, that would have been the analytics query. Postgres stays up. The query retries later, and you get a log entry saying 'oomd killed /system.slice/postgresql.service/analytics.slice'. Luna: That log entry is gold. But you also need monitoring that catches the spiral before oomd acts, right? Lucas: Right. And this is where most people miss it. Standard monitoring checks CPU, memory percentage, disk space. It does not check swap IO. The metric you want is the number of pages swapped in and out per second. On Linux that is /proc/vmstat — specifically pgpgout and pgin. If you graph those, a healthy server shows near-zero swap IO. The moment you see sustained swap writes over a few hundred pages per second, you are in trouble. Luna: I want to pause on that because it's genuinely useful. If today's tech conversation gave you something usable — a sysctl command, a systemd config snippet, a metric name — that's the whole point. This show stays ad-free because a small group of listeners chips in on buy me a coffee dot com slash fexingo. Literally just enough to cover hosting and our time. So if you got something out of this swap walkthrough, that's where you can make sure it keeps coming. Lucas: Yeah, and we mean it — the support from listeners is what lets us dig into this level of detail instead of doing surface-level hot takes. So thank you to everyone who already does. Now back to the spiral. Let's talk about the PromQL alert that catches this. Luna: Please. I want a rule I can paste into Prometheus. Lucas: You use the node_exporter's memory metrics. The key ones are node_memory_SwapTotal_bytes and node_memory_SwapFree_bytes. But that only tells you how much swap is used, not the IO. For IO you need node_vmstat_pswpin and node_vmstat_pswpout. Those are cumulative counters. You rate them over a one-minute window and alert if the rate exceeds something like 100 pages per second. Luna: And you want to alert on pswpout, not pswpin. Because swapping out is what causes the IO pressure. Lucas: Correct. The alert expression is: rate > 100. That catches the start of the spiral. You can tune the threshold — on a server with fast NVMe, 500 might be fine — but 100 is a safe default. Pair that with a second alert that fires if swap usage exceeds, say, 80 percent for five minutes. Luna: Now what about swap size? Is there a rule of thumb for how much swap to allocate on a modern server with, say, 128 gigs of RAM? Lucas: The old rule was twice RAM. That is from the era when RAM was measured in megabytes. Today, for a server with 64 gigs or more, I usually allocate zero swap or, if the application requires it — for crash dumps or memory overcommit — allocate a fixed 8 gigs. Do not let swap scale with RAM. The death spiral gets worse the more swap you have because the kernel has more room to thrash. Luna: Some people argue for no swap at all. What's your take? Lucas: No swap works if you have tight memory accounting and no overcommit. But many applications — including Postgres — do memory-mapped IO that expects swap. Also, if you have no swap and the kernel runs out of anonymous memory, it goes straight to OOM. With a tiny swap, you at least give systemd-oomd a chance to intervene. So I prefer 8 gigs of swap with swappiness 10 and systemd-oomd enabled. Luna: And you need to put that swap on a separate device from the root filesystem if possible. If the swap IO and the database IO share the same disk, you compound the thrash. Lucas: That's a great point. If you have a spare NVMe or even a partition on a separate drive, dedicate it to swap. The latency isolation helps a lot. I've seen setups where swap lives on a small RAID1 of two SATA SSDs just to keep the main NVMe clear for database reads. Luna: Alright. So after the incident, what did the team actually change on that Postgres server? Lucas: Three things. One: set vm.swappiness to 10. Two: enabled systemd-oomd for the postgresql service slice with a 90 percent swap threshold. Three: added the Prometheus alert on pswpout rate. They also added a cron job that runs every five minutes and checks /proc/vmstat for high swap IO and logs a warning. That's a nice belt and suspenders approach until you trust the alerting. Luna: Do you have that cron script handy? Lucas: Sure. It's five lines of bash. You grep for 'pgpgout' in /proc/vmstat, get the second field, compare it to a previous sample you stored in a temp file. If the delta divided by the interval is above a threshold, you log a critical message. The script is on our show-notes page if you want the exact code. Luna: And that script will work on any Linux distribution that has /proc/vmstat, which is all of them. Lucas: Right. And the beauty is it costs almost nothing to run. It's a single stat read and a subtraction. The monitoring industry has convinced us we need agents and collectors and dashboards for everything. Sometimes a cron job and a log file is enough. Luna: I think the deeper point is: the default Linux kernel parameters are tuned for general-purpose workstations, not production servers. The defaults assume you don't care about latency spikes. But in production, you care deeply. Lucas: Exactly. Swappiness is just one example. The same philosophy applies to dirty page ratios, to the scheduler's group migration, to the network buffer sizes. The defaults are reasonable for a laptop. For a server, you need to audit every single one. Luna: So if a listener takes one thing from this episode, what should it be? Lucas: Check your swappiness today. Run 'cat /proc/sys/vm/swappiness'. If it's 60 and you run a database, set it to 10. Then set up the Prometheus alert on pswpout. That five-minute change will prevent the exact scenario we described. And if you want to go further, enable systemd-oomd and give your database service a dedicated swap limit. Luna: And maybe add that five-line cron job while you're at it. Lucas: Definitely. Swap is not evil. It's a tool. But it's a tool that can cut you if you leave it on the default setting.