Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose and Fix Linux Server Packet Loss
Transcript
- Lucas: You're pinging your server and you see it. One request gets a reply in 0.3 milliseconds. The next one times out. Then three come back fine. Then another timeout. Luna: That intermittent pattern where you can't quite pin down the cause. It's maddening. Lucas: Exactly. And if you're running anything latency-sensitive — a database, a real-time API, a voice application — that 2 or 3 percent packet loss can tank performance. Today we're going to walk through how to systematically diagnose and fix packet loss on a Linux server. Luna: Great. So where do you start when you suspect packet loss? Lucas: First, confirm it's actually packet loss and not something else — like application-level timeouts or a firewall dropping ICMP. The simplest test is a continuous ping to a stable target like your default gateway. Run 'ping -c 100 <gateway IP>' and look at the summary line. If it says 2 percent packet loss, you have a problem. Luna: And if the gateway is fine but pinging an external host shows loss, that tells you the issue is beyond your first hop. Lucas: Right. The next tool I reach for is mtr — it's basically traceroute combined with ping. You run 'mtr --report <target>' and it'll show you loss percentages at each hop. That's your first clue to where the bottleneck lives. If the loss spikes at hop 2 — your switch — and then stays high, the problem might be that switch port or the cable to your server. Luna: What if mtr shows loss only at the final hop? That could be the remote server's firewall rate-limiting ICMP, not actual packet loss. Lucas: Exactly. So you need to cross-check with a tcp based test. I use iperf3 for that. Run an iperf3 server on the remote machine, connect from your server, and watch the 'retr' column — retransmissions. If you're seeing retransmits at the same rate as your ping loss, it's real. If not, the remote host might just be ignoring ICMP. Luna: Okay, so you've confirmed real packet loss. Now you need to find out if it's your server's network interface, the cable, the switch, or something upstream. Lucas: Right. And you start at the bottom of the OSI model: the physical layer. On your server, run 'ethtool -S <interface>' and look for a few key counters: 'crc_errors', 'rx_fifo_errors', 'tx_fifo_errors', 'rx_frame_errors'. Any of those non-zero suggests a physical problem — bad cable, bad port, or electromagnetic interference. Luna: I had a case where a server in a data center had hundreds of CRC errors. Re-seated the cable and they dropped to zero. It was just slightly loose. Lucas: That's the most common fix. If the cable is fine, check the interface's negotiated speed and duplex with 'ethtool <interface>'. You want to see 'Speed: 10000Mb/s' and 'Duplex: Full'. If you see half-duplex or a mismatched speed — like the switch is set to auto-negotiate and the server is forced to 1 Gbps — you'll get collisions and drops. Luna: And you can force the speed and duplex with ethtool, but you need to make sure both sides match. Lucas: Absolutely. Now let's talk about the kernel side. Even with a perfect physical link, your server can drop packets if it's overwhelmed. Run 'netstat -s' and look for 'packets receive errors', 'packets to unknown port received', and 'receive buffer errors'. The big one is 'receive buffer errors' — that tells you the kernel's socket receive buffer is overflowing. Luna: Which means your application isn't reading data fast enough. Or the buffer is too small for the traffic volume. Lucas: Exactly. You can check the current buffer size with 'sysctl net.core.rmem_max' and 'sysctl net.core.wmem_max'. The default is often 212992 bytes — that's around 200 kilobytes. For a busy server, you might want to bump that to 16 or 32 megabytes. And you need to adjust the application's socket buffer settings too. Luna: Another thing I've seen: the network interface's ring buffer. If the ring buffer fills up before the kernel can process interrupts, packets get dropped at the NIC level. Lucas: That's a great point. Check with 'ethtool -g <interface>'. It'll show you the current and maximum ring buffer sizes. If you see 'RX: 256' and 'TX: 256' and you're dropping packets, try doubling them. For example, 'ethtool -G eth0 rx 512 tx 512'. That gives the NIC more room to store incoming packets before they're copied to kernel memory. Luna: You mentioned interrupt coalescing earlier. Any tips on that? Lucas: Yes. Use 'ethtool -c <interface>' to see the current coalescing settings. Parameters like 'rx-usecs' and 'rx-frames' control how long the NIC waits before raising an interrupt. If you have very bursty traffic, you might want to increase the delay a bit to batch interrupts, reducing CPU overhead. But if your traffic is latency-sensitive, you want lower coalescing — at the cost of more CPU interrupts. Luna: So it's a trade-off. And you can tune it per workload. Lucas: Right. Now, if you've checked the cable, the interface settings, the ring buffer, and the kernel buffers, and you still have packet loss, it's time to look at the switch. But you might not have direct access to the switch. So what do you do? Luna: You can ask your network team to check the switch port for errors. They can look at CRC errors, runts, giants, and collisions on that port. Lucas: Exactly. And if they see errors on the switch side but your server's ethtool counters are clean, the cable between the server and the switch is the likely culprit. Replace it. If both sides show errors, it's probably a faulty port or a bad cable. If neither side shows errors but you still have loss, the issue might be upstream — maybe a congested link further out. Luna: Let's talk about tcpdump for a second. When would you use it to diagnose packet loss? Lucas: I use tcpdump when I want to confirm that packets are actually leaving or arriving at the interface. For example, if your application is sending data but the other side isn't receiving it, you can run 'tcpdump -i <interface> host <target>' on both sides. If the sender sees the packets but the receiver doesn't, the loss is in the network. If the receiver sees them but the application doesn't, it's a socket buffer or application issue. Luna: That's a very clear way to bisect the problem. Lucas: One more tool I want to mention: 'ip -s link show <interface>'. It gives you a quick summary of transmitted and received packets, errors, and drops. I run that as a first check because it's faster than ethtool -S. If you see non-zero drops or errors, then you dive deeper. Luna: And if everything looks clean on your server but you still have symptoms, you might need to involve your network team for a deeper dive. Lucas: Right. And if this kind of systematic troubleshooting has helped you — maybe you saved a late-night server rescue — that's exactly the kind of practical knowledge we try to share here on the show. If you find value in these episodes and want to support keeping us ad-free and independent, you can help at buy me a coffee dot com slash fexingo. It genuinely helps us keep doing deep-dive episodes like this one. Luna: Yeah, listener support is what makes this possible. Back to packet loss — one thing I want to add: don't forget to check for software-based bridging or bonding. I've seen cases where a misconfigured bond interface dropped packets because the hashing policy didn't match the switch. Lucas: Great call. If you're using bonding, check '/proc/net/bonding/bond0' and verify that the mode matches what the switch expects. For LACP, both sides need to be configured the same way. A mismatch can cause packets to be sent down the wrong link and dropped. Luna: So to recap: start with ping and mtr to locate the loss, then use ethtool to check the physical layer, netstat and sysctl for kernel buffers, and tcpdump for endpoint confirmation. Involve the network team for switch-level issues. Lucas: And don't skip the simple stuff — reseat the cable, check the speed and duplex, and look at ring buffer sizes. Most packet loss problems on a single server are solved at layer 1 or layer 2. Thanks for joining us on this episode of Linux Server Admin. We'll see you next time.