Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Tame Linux Server NFS Performance with Tuning
Transcript
- Lucas: Let's talk about NFS — Network File System. It's one of those protocols that just works, until it doesn't. And when it doesn't, the performance hit can be brutal, especially if you're running something like a media rendering farm or a data pipeline that depends on high throughput over the network. Luna: I've definitely seen NFS become the bottleneck in a cluster. Everyone assumes it's the network or the disks, but sometimes it's just the NFS defaults not being tuned for the workload. Lucas: Exactly. And today I want to walk through a real case. A friend of mine runs a small VFX studio — about thirty artists pulling frames off an NFS share. They were seeing read speeds of around 20 megabytes per second on a 10-gig link. That's terrible. The disks were fine, the network was fine. It was all NFS tuning. Luna: Twenty megabytes per second on 10 gig — that's like two percent utilization. Something was definitely off. Lucas: Right. So let's start with the basics. The first thing to check is the mount options, specifically `rsize` and `wsize`. These are the read and write buffer sizes — the maximum amount of data the client sends or receives in a single NFS request. The default for NFSv3 was 8 kilobytes, which is tiny. For NFSv4, it's often 1 megabyte, but even that can be suboptimal depending on the network and workload. Luna: And what did you find in the VFX case? Lucas: They were on NFSv3 with the default 8-kilobyte `rsize` and `wsize`. I bumped them to 1 megabyte — 1048576 — and the throughput jumped to about 150 megabytes per second on the same hardware. That's a sevenfold improvement just from changing two mount options. Luna: Seven times faster just from buffer sizes. That's the kind of win that makes you wonder how many servers out there are running with ancient defaults. Lucas: And before we get deeper into the tuning — quick honest thing. You know, this show is ad-free, and that's because a small group of listeners chip in monthly through buy me a coffee dot com slash fexingo. It's not a big ask, but it's literally what lets us keep making detailed episodes like this one. So if you get value out of these tuning tips, that's where you can help. Luna: Yeah, it's a low-key thing. And it keeps the show independent. Lucas: Alright, back to the tuning. So once you've set `rsize` and `wsize`, the next thing to look at is the NFS server daemon — `nfsd`. By default, most Linux distributions set the number of `nfsd` threads to 8. That might be fine for a few clients, but for a busy server with multiple concurrent accesses, you want more threads. The rule of thumb is to start with one thread per CPU core, and then monitor. Luna: So for a 16-core server, you'd set 16 threads. And how do you actually check and change that? Lucas: You can check the current thread count by looking at `/proc/fs/nfsd/threads` — cat that file, and it'll show you the number. To change it, you write to the same file: `echo 16 > /proc/fs/nfsd/threads`. But that change is ephemeral. To make it permanent, you set the `RPCNFSDCOUNT` parameter in `/etc/default/nfs kernel server` on Debian-based systems, or add `nfsdcnt=` to the kernel boot options on Red Hat. Luna: And you'd want to monitor the server's CPU utilization to see if the threads are actually being used. If they're all busy, you might need more. Lucas: Exactly. You can also look at `nfsstat` to see the number of NFS requests per second. If the server is handling thousands of operations per second and the threads are maxed out, bump it up. I've seen servers with 64 threads on a 32-core machine for heavy workloads. Luna: What about the network side? I know TCP tuning can matter too. Lucas: It can. But for NFS specifically, the biggest wins are usually on the NFS layer itself. However, there are a couple of sysctl settings that help. One is `net.core.rmem_default` and `net.core.wmem_default` — these set the default socket buffer sizes. The default is often 128 kilobytes, but for 10 gigabit Ethernet, you might want to set them to 1 or 2 megabytes. You can also set `net.core.rmem_max` and `net.core.wmem_max` to allow larger buffers. Luna: And don't forget `net.ipv4.tcp_rmem` and `net.ipv4.tcp_wmem` — those are the TCP buffer autotuning parameters. You can set the min, default, and max there. Lucas: Right. But let's be careful — socket buffer tuning is a dark art. I usually only touch it if I've exhausted the nfs specific options. For the VFX studio, just the `rsize` and `wsize` plus bumping `nfsd` threads from 8 to 16 got them to about 400 megabytes per second. That was enough for their workflow. Luna: So from 20 megabytes to 400 — a twentyfold improvement. That's the kind of tuning story that makes people realize how much headroom is often left on the table. Lucas: Absolutely. And there's one more thing: the NFS export options on the server. The `sync` vs. `async` option on the export can have a big impact. The default is `sync`, which means the server writes data to disk before replying to the client. That's safe, but slower. `async` lets the server reply before the data is fully on disk, which is faster but risks data loss on a crash. If you're serving media files that can be easily regenerated, `async` is usually fine. Luna: Right. And you'd set that in `/etc/exports` — like `/shared 192.168.1.0/24` becomes `async` instead of `sync`. That can give a noticeable boost on writes. Lucas: Yeah. But always test. I've seen `async` cause corruption in databases over NFS, so it's not for everything. For file storage like media, it's usually safe. One more thing: use `no_subtree_check` and `no_root_squash` if appropriate — they reduce overhead. But `no_root_squash` is a security risk, so only use it if you really need it. Luna: You mentioned tools earlier — `nfsstat` and `iostat`. Can you give a quick command example for diagnosing NFS performance? Lucas: Sure. On the client, `nfsstat -c` shows client-side statistics. You want to look at the number of retransmissions — if that's high, your network or server can't keep up. On the server, `nfsstat -s` shows server stats. And `iostat -x 1` on the server can tell you if the disks are saturated. If the disk utilization is low but the NFS threads are maxed, it's likely a tuning issue. Luna: And `tcpdump` or `nload` can help see if it's a network bottleneck. But usually, the NFS layer is the culprit. Lucas: Exactly. So to summarize for the VFX studio: we changed `rsize` and `wsize` to 1 megabyte, increased `nfsd` threads to 16, switched to `async` on the export, and enabled `no_subtree_check`. That took them from unusable to production-ready. If you're running NFS, don't assume the defaults are optimal. Test with your own workload. Luna: And that's the takeaway — benchmark your NFS performance before and after tuning. You might be surprised how much free performance is available. Lucas: Yeah. And if you've got a story about a weird NFS issue, we'd love to hear it. There's always another edge case to learn from. Thanks for listening.