Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Kernel Same-page Merging for Memory Optimization
Transcript
- Lucas: If you manage Linux servers—especially virtualized or containerized ones—you've probably stared at a memory usage graph and thought, 'How can I squeeze more out of the same hardware?' Luna: It's a common pain point. RAM is often the first resource to hit a ceiling, and throwing more sticks in the box isn't always an option. Lucas: Right. So today I want to talk about a kernel feature that's been around for over a decade but doesn't get enough attention in everyday sysadmin work: Kernel Same-page Merging, or KSM. Luna: KSM… I've seen it mentioned in KVM documentation, but I've never actually tuned it myself. What does it do? Lucas: KSM is a memory deduplication mechanism. It scans the memory pages of running processes, identifies pages that are identical, and merges them into a single copy marked copy-on-write. So if you have fifty VMs running the same guest OS kernel, KSM can merge all those identical kernel pages into one physical page. Luna: That's significant savings. Fifty identical pages become one—you'd reclaim a huge chunk of RAM. Lucas: Exactly. And it's not just for VMs. Containers, database caches, even multiple instances of the same application can share memory. I've seen a 30 to 50 percent reduction in memory usage on heavily virtualized hosts. Luna: Thirty percent is huge. But there must be a catch, right? CPU overhead, latency? Lucas: Yes, there's a trade-off. KSM runs a kernel thread called ksmd that periodically scans memory pages. That consumes CPU cycles. And when a merged page is written to, it must be unshared—that copy-on-write operation adds a small latency spike. Luna: So for latency-sensitive workloads, like trading systems or real-time databases, you'd probably want to keep it off. Lucas: Typically yes. But for development servers, CI/CD runners, or any environment where you have many similar processes, the savings can far outweigh the overhead. The key is tuning. Luna: Let's talk tuning. How do you actually enable KSM on a modern Linux server? Lucas: On most distributions, it's as simple as writing to sysfs. You set /sys/kernel/yeah/ksm/run to 1 to start the scanner, and you can adjust parameters like pages_to_scan and sleep_millisecs to control how aggressively it scans. Luna: So you can pace it. Scan a small number of pages per pass, sleep longer, reduce CPU impact. Lucas: Right. A typical starting point is 100 pages per scan and a 20-millisecond sleep. That keeps ksmd under 1% CPU on most hardware. You can also set merge_across_nodes to 0 if you want to avoid merging across NUMA nodes—that can prevent cross-node memory access penalties. Luna: NUMA awareness is important. Merging a page from node 0 with one from node 1 means every thread accessing that page might incur a remote memory penalty. Lucas: Exactly. In a NUMA system, you'll want to keep memory local. So on multi-socket servers, set merge_across_nodes to 0. Now, the other big knob is full_scans. This counter tells you how many times ksmd has scanned all pages. You can check it to see if the scan is keeping up with memory churn. Luna: What's a reasonable full_scans rate? Should it be increasing steadily? Lucas: If the number is increasing, the scanner is completing full sweeps. That's fine. But if it stalls, it means pages_to_scan is too low or sleep_millisecs is too high—the system is changing memory faster than ksmd can deduplicate. You might need to ramp up pages_to_scan. Luna: What about monitoring actual savings? How do you measure how much memory KSM is merging? Lucas: There's a utility called ksmctl, but the raw data is in the same sysfs directory. The key files are pages_shared and pages_sharing. Pages_shared is the number of unique pages that are being shared. Pages_sharing is the total number of page references to those shared pages—basically the total savings. Luna: So if pages_shared is 1000 and pages_sharing is 5000, then 1000 physical pages are supporting 5000 virtual pages. You're saving 4000 pages of RAM. Lucas: Exactly. You can calculate total savings by multiplying by the page size, which is usually 4 kilobytes. On a host with thousands of VMs, that can easily be gigabytes. Luna: That's compelling. But there's a darker side to KSM, right? Security concerns? Lucas: Yes, and this is where a lot of sysadmins get nervous. KSM's memory deduplication can be exploited via side-channel attacks like Rowhammer or cross vm memory disclosure. If an attacker can influence what a neighbor process writes to memory, they might be able to deduce shared page contents or cause bit flips. Luna: So in a multi-tenant cloud environment, you probably don't want KSM enabled across different customer workloads. Lucas: Correct. That's why cloud providers typically disable KSM. But in a private data center where you control all the workloads, the risk is lower. Some distributions, like Red Hat, actually ship with KSM enabled by default for KVM hosts—they trust the isolation in that context. Luna: What about newer alternatives? I've heard of userspace deduplication or using zswap instead. Lucas: Great question. Zswap compresses memory pages, which can also reduce physical RAM usage, but it's not deduplication. KSM is complementary. KSM works best when you have many identical pages; zswap helps when pages are compressible. Some people run both, but you need to test—they can compete for CPU. Luna: And there's also UMA—User-space Memory Deduplication—but that requires application support, doesn't it? Lucas: Right. KSM is transparent to applications. That's its killer feature. You flip a sysfs knob and suddenly your memory usage drops. No code changes, no container rebuilds. Luna: That's hard to beat for legacy workloads. Lucas: Exactly. So here's a practical decision framework: Use KSM on development servers, staging environments, CI runners, and any homogeneous VM farm. Disable it for latency-sensitive production and multi-tenant hosting. Always test with your specific workload first. Luna: And measure before and after. Don't just flip the switch and assume. Lucas: Right. And you know, a big part of why we can dive into these niche tuning topics is because this podcast is listener-supported. We don't run ads, and that means we can cover whatever we think will help you do your job better. Luna: Yeah, it's a small group of listeners who chip in through buy me a coffee dot com slash fexingo, and that literally funds making this many episodes. Lucas: So if today's conversation gave you something usable, that's exactly why we keep going deep. And back to KSM—one more thing I want to mention is that you can also use KSM in combination with memory overcommit. If you're running VMware or KVM with memory overcommit, KSM can prevent the host from swapping when guests ask for more memory than physically available. Luna: That's a good point. Overcommit without deduplication can lead to thrashing. KSM gives you a safety buffer. Lucas: Exactly. So to wrap up: KSM is a powerful tool, but it's not a set-and-forget. Monitor pages_shared, pages_sharing, and CPU usage. Adjust pages_to_scan and sleep_millisecs based on your workload's memory churn. And always weigh the security implications. Luna: And if you're on a NUMA system, don't forget merge_across_nodes. Lucas: Yes. That one detail can make the difference between a performance win and a hidden NUMA penalty. Next time, we'll talk about another memory management feature: zswap configuration and compression algorithms. Luna: Looking forward to it. Until then, happy merging.