Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Tune Linux Server TCP for High Latency Links
Transcript
- Lucas: You run a server in Singapore but half your users are in Brazil. Default TCP settings will absolutely murder your throughput. Let's talk about why, and what you actually tweak. Luna: This is super practical because most sysadmins never touch TCP buffers unless something breaks. What's the concrete scenario you have in mind? Lucas: A friend runs a video transcoding service — small shop, maybe fifty servers. Their biggest customer is a media company with viewers across South America. They had users complaining about buffering even though the server had plenty of bandwidth. Lucas: He ran iperf from Singapore to a test box in São Paulo and saw throughput capped at about 12 megabits per second. Not a bandwidth issue — the link itself could do 200 megabits. It was TCP being TCP. Luna: So the bottleneck was the protocol itself, not the wire. That's the classic high-latency problem. What was the round-trip time? Lucas: Around 280 milliseconds. Singapore to Brazil is not a short hop. And when you have a 280-millisecond RTT with default socket buffers, you run into the bandwidth-delay product calculation in a bad way. Luna: Right, because TCP's flow control depends on the receiver advertising a window. If the window is too small, the sender stalls waiting for ACKs. Lucas: Exactly. The bandwidth-delay product is just bandwidth times latency. Two hundred megabits per second times 0.28 seconds gives you about 56 million bits, or 7 megabytes. That's the minimum buffer you need to keep the pipe full. Lucas: Default Linux socket buffers — at least on older distros — are often set to 128 kilobytes or less for the receive side. So the sender fills that tiny window, waits 280 milliseconds for an ACK, sends another chunk, waits again. You're not streaming, you're dribbling. Luna: So the fix is increasing those buffers. What exactly did he change? Lucas: First, he set net.core.rmem_max and net.core.wmem_max to 16 megabytes each. Those are the absolute max the kernel allows for any socket. Then he tuned the TCP auto-tuning range via net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Lucas: The tcp_rmem and tcp_wmem take three values: minimum, default, and maximum. He went with something like 4096, 131072, and 16777216. The middle value is what most connections start with, and the kernel auto-tunes up to the max based on the connection. Luna: And did that fix the 12-megabit problem? Lucas: It helped, but not entirely. He went from 12 to about 30 megabits. The missing piece was the congestion control algorithm. Default on many Linux systems is still Cubic, which was designed for terrestrial links with moderate loss. Luna: For high-latency links with occasional packet loss, Cubic can overreact and cut the window too much. That's where BBR comes in. Lucas: Exactly. BBR models the bandwidth and RTT instead of using loss as a signal. He enabled it with net.ipv4.tcp_congestion_control = bbr. And he made sure the kernel had BBR built in — which it does since kernel 4.9, but you need the tcp_bbr module loaded. Lucas: After that, his iperf test jumped to about 85 megabits. Not the full 200, but a massive improvement. And real user buffering complaints dropped by a lot. Luna: Eighty-five megabits on a 200-megabit link — there's still some overhead, but that's a 7x improvement. What else could he have tweaked? Lucas: He could have played with tcp_slow_start_after_idle — set it to 0 so that after a period of idle, the connection reuses its previous congestion window instead of resetting to the initial window. That helps for connections that have bursts of transfers. Lucas: Also, tcp_notsent_lowat can help with application write patterns. It controls how much unsent data can queue in the send buffer before the application blocks. For video streaming, you want the send buffer to stay full. Luna: There's also the receive side — tcp_adv_win_scale and tcp_app_win. Those affect how the receive window is calculated and how much buffer is reserved for the application. Lucas: Yeah, he left those at defaults because they interact with auto-tuning, but if you have a specific application pattern, you might tune them. The important thing is to measure before and after. Luna: So the sysadmin takeaway: start with socket buffer sizes, then consider switching to BBR for congestion control, and test with iperf from a representative location. Don't blindly copy settings from a blog post. Lucas: Exactly. And make the changes persistent. Put them in /etc/sysctl.d/90-tuning.conf or similar. Then run sysctl -p to apply. But test one change at a time so you know what helped. Luna: One more thing: if you're using network namespaces or containers, each namespace has its own sysctl for these parameters. Don't forget to set them inside the container if your app runs there. Lucas: Great point. And finally, consider tc qdisc for shaping if you need to prioritize certain traffic. But for pure buffer tuning, the sysctl approach is the first step. Luna: Yeah, it's amazing how many people never touch these settings. Defaults are fine for LAN or low-latency internet, but once you go global, it's a different world. Lucas: And honestly, that's part of why we do this show — we try to surface the stuff that's not in the default config, the things you only learn when you have to troubleshoot a real issue at 2 AM. Luna: Absolutely. And since we keep the show ad-free, we rely on listeners who find this useful to support the effort if they're able. There's a link at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small way to keep these deep dives coming without any sponsor influence. So if today's TCP tuning saves you a headache, consider tossing a coffee our way. Luna: Alright, back to the bits. One last tip: if you apply these sysctl changes to a production server, do it gradually. Not all apps handle huge socket buffers gracefully. Test on a staging server first. Lucas: Wise advice. And if you want the exact numbers from that singapore to brazil case, they're in the show notes. Thanks for tuning in. Luna: See you next episode.