Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Network Performance Tuning with sysctl
Transcript
- Lucas: Alright, let's talk about squeezing more performance out of a Linux server's network stack without touching the hardware. I'm talking about sysctl — the kernel parameter tuning tool that's been around forever but is surprisingly underused. Luna: Sysctl is one of those tools that looks boring until your web server starts dropping connections under load. Then it becomes the most interesting thing in the room. Lucas: Exactly. So today we're going to focus on a specific scenario: a Linux web server handling, say, ten thousand concurrent connections. Out of the box, the default kernel settings are tuned for general-purpose use — not for that kind of throughput. Lucas: And if today's conversation gives you something you can actually use on a Tuesday afternoon, consider throwing a couple of dollars toward keeping this show ad-free. A few bucks a month at buy me a coffee dot com slash fexingo genuinely makes a difference — no pressure, just something to keep in mind. Luna: Yeah, it's a small thing that adds up. Appreciate anyone who chips in. Lucas: So back to sysctl. The first parameter I look at when I'm tuning for high concurrency is net.core.somaxconn. This controls the maximum number of connections that can be queued for a listening socket before the kernel starts rejecting them. Default on most distros is 128. Luna: 128. For a modern web server behind a load balancer that can spike to thousands of connections in a second, that's almost nothing. Lucas: Right. Bumping it to 1024 or even 4096 is common. But you also need to check the application backlog setting — nginx and Apache have their own listen backlog, and sysctl's somaxconn is the kernel-level cap. If your app says 512 but the kernel says 128, the kernel wins. Luna: Good point. I've seen people tweak somaxconn and wonder why nothing changed. They forgot to restart the service so it picks up the new backlog value. Lucas: Next big one: net.ipv4.tcp_tw_reuse. This lets the kernel reuse connections in the TIME_WAIT state for new outbound connections. TIME_WAIT is a normal TCP state that sticks around for about 60 seconds by default. On a busy server making lots of outbound connections — like a reverse proxy or a database client — you can exhaust the ephemeral port range. Lucas: Setting tcp_tw_reuse to 1 allows the kernel to grab a port from TIME_WAIT if it's safe to do so. It's not a silver bullet — there are caveats around NAT and load balancers — but for internal server to server traffic, it's usually safe. Luna: And there's tcp_tw_recycle, which I'd say avoid entirely. It's been removed in newer kernels anyway, but it caused issues with NAT'd connections. Lucas: Yeah, tcp_tw_recycle is deprecated. Don't use it. Stick with tcp_tw_reuse and also lower tcp_fin_timeout. Default is 60 seconds. Cutting it to 15 or 30 can free up sockets faster. Lucas: Another critical one: net.core.rmem_max and net.core.wmem_max, plus the tcp specific buffers like tcp_rmem and tcp_wmem. These control how much memory the kernel allocates for receive and send buffers. Luna: I remember a Black Friday incident at a previous company where the default rmem_max was 212992 bytes — about 208 KB. Under load, packets were getting dropped because the buffer filled up faster than the application could drain it. Lucas: What did you bump it to? Luna: We set tcp_rmem to '4096 87380 16777216' — that's 16 MB max. And the global rmem_max to 16 MB as well. Cleared up the packet drops immediately. Lucas: Classic fix. The three values there are min, default, and max. The default is usually fine for most connections, but the max needs to be high enough for burst traffic. Same pattern for tcp_wmem. Lucas: Let's talk about net.ipv4.tcp_syncookies. It's a defense against SYN flood attacks. By default it's usually set to 1, meaning syncookies are enabled only when the SYN backlog fills up. On a high-traffic server, you might want to keep it on, but be aware it adds a tiny bit of CPU overhead. Luna: And it breaks some TCP options, like large windows. So there's a trade-off. Lucas: Right. Now, how do you actually apply these? The quick way is sysctl -w net.core.somaxconn=4096. But that's temporary. To make it permanent, add the line to /etc/sysctl.conf or a file in /etc/sysctl.d/. Lucas: After editing, run sysctl -p to load the changes. Or sysctl --system on newer distros. Luna: One thing I'd stress: don't blindly copy tuning guides. Some parameters are hardware-specific. For example, net.core.netdev_budget controls how many packets the kernel processes per network interrupt. If you set it too high, you can starve other tasks. Lucas: Absolutely. And there's net.ipv4.tcp_keepalive_time, tcp_keepalive_intvl, and tcp_keepalive_probes. Default keepalive time is 7200 seconds — two hours. For a web server, you might want to drop that to 600 seconds to detect dead connections faster. Luna: But if you set keepalive too aggressive, you generate unnecessary traffic. It's about finding the right balance for your application. Lucas: Let's also mention net.ipv4.ip_local_port_range. Default is usually 32768 to 60999, giving you about 28,000 ephemeral ports. For a server making many outbound connections, you can expand it to, say, 1024 to 65535. Luna: That's a big increase. But make sure your firewall and NAT rules can handle the wider range. Lucas: Good catch. And one more: net.core.optmem_max. This sets the maximum ancillary buffer size per socket. For applications that use lots of control messages — like sendmsg with ancillary data — you might need to bump it from the default 20480 bytes. Luna: So the takeaway: sysctl gives you low-level control over how the kernel handles network traffic. The defaults are safe but not optimal for high-load servers. Lucas: Right. And the best approach is to monitor your server's network performance — look at dropped packets in /proc/net/stat, use netstat or ss to check socket states, and adjust incrementally. Lucas: One thing I'm curious about — there's been talk of a more structured way to manage kernel parameters across kernel upgrades. Do you think we'll eventually see something like a sysctl versioning system? Luna: I hope so. Right now, upgrading the kernel can silently override your custom sysctl settings if the parameter names change. Having a versioned schema would be a game-changer for reproducibility. Lucas: That's a topic for another episode. For now, try adjusting somaxconn and tcp_tw_reuse on a test box and see what happens to your connection handling. You might be surprised at the difference.