Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Network Bonding for Redundancy and Performance
Transcript
- Lucas: Alright Luna, you've got a server with two physical network ports. One cable goes down. The whole box goes offline. That's a bad day if it's your production database or your storage gateway. Luna: Yeah, and I'm guessing the fix isn't just 'plug in two cables and hope Linux figures it out.' Lucas: Exactly. That's where network bonding comes in — also called NIC teaming or link aggregation. You combine two or more physical Ethernet interfaces into a single logical interface called bond0. The kernel handles failover and, depending on the mode, load balancing across the links. Luna: So the server sees one interface, but it's actually backed by two physical ports. And you get redundancy that way. Lucas: Right. The most common setup is active-backup mode — mode 1. One link is active, the other sits idle. If the active cable gets yanked or the switch port dies, the kernel instantly switches traffic to the standby link. No dropped connections, just a slight pause. Luna: And that pause is usually under a second, right? ARP resolution might cause a hiccup but TCP sessions survive. Lucas: Exactly. For most services it's transparent. But if you want more than just failover — if you want to actually use both links for throughput — you need a different mode. Mode 0 is balance-rr, round-robin, which sends packets alternating across the interfaces. But that has strict requirements: you need a switch that supports EtherChannel or you'll get out-of-order packets. Luna: Which causes TCP retransmissions, so you actually lose performance. Not great. Lucas: Precisely. The more practical load-balancing modes are balance-xor — mode 2 — and 802.3ad — mode 4. Balance-xor uses a hash of the source and destination MAC addresses to decide which interface to use. It's simple, no switch configuration needed beyond the same VLAN on both ports. Luna: But mode 4, 802.3ad, is the one that requires LACP on the switch side. That's Link Aggregation Control Protocol — the industry standard for dynamic link aggregation. Lucas: Yep. With 802.3ad, the switch and the server negotiate which links are part of the bundle. It gives you better load distribution because the hash can also use IP and port information if you set xmit_hash_policy to layer3+4. And it's full fault tolerance — if one link drops, traffic redistributes to the others. Luna: So for a virtualization host with multiple VMs, 802.3ad is the go-to. Each VM's traffic gets hashed to a different physical link, and overall throughput can approach the sum of all links. Lucas: Right. But there's a catch: you need a switch that supports LACP and you have to configure the port-channel on the switch. That might be a different team — network team versus server team — so in some orgs, active-backup or balance-xor is preferred because no switch config change is needed. Luna: Alright, let's walk through the actual setup. I'm on Ubuntu Server 24.04. What packages do I need? Lucas: The kernel bonding module is usually built-in. You just need the ifenslave package — that gives you the utilities to enslave interfaces to the bond. On Debian-based systems: 'apt install ifenslave'. On RHEL, it's included in the base, but you might need to load the bonding module: 'modprobe bonding'. Luna: Then edit /etc/network/interfaces — assuming you're not using netplan or NetworkManager. Let's say eth0 and eth1 are the physical interfaces, and you want bond0 in active-backup mode. Lucas: Right. Here's the config. First, the bond interface itself: 'auto bond0', 'iface bond0 inet static', address, netmask, gateway. Then 'bond-slaves none' and 'bond-mode 1', 'bond-miimon 100'. The miimon parameter tells the driver to check link status every 100 milliseconds. Luna: Then for each physical interface: 'auto eth0', 'iface eth0 inet manual', 'bond-master bond0'. Same for eth1. And you don't assign IP addresses to the slaves. Lucas: Exactly. After that, 'systemctl restart networking' — or ifup bond0 if you prefer. Then check the bond status with 'cat /proc/net/bonding/bond0'. That shows you the bonding mode, the MII status, the active slave, and which interfaces are up. Luna: And you can test failover by pulling a cable. The proc file will update to show the active slave changed. Lucas: Yeah. One thing to watch: if you're using balance-xor or 802.3ad, you need to set the bond's MAC address explicitly sometimes. By default, bond0 uses the MAC of the first slave. If that slave goes down, the bond may change MAC, which can confuse your switch's ARP table. Luna: So you'd set a static MAC on the bond interface. Or use 'fail_over_mac=1' in the bonding options to keep the active slave's MAC. Lucas: Exactly. Another thing: on systems with NetworkManager, you'd create a team or bond connection via nmcli. But the kernel-level bonding works the same way underneath. Luna: What about performance monitoring? How do you know if the load is balanced well? Lucas: You can check 'cat /proc/net/dev' to see packet counts per interface. If you're in balance-xor, you want to see similar numbers on both links. If one interface is saturated and the other is idle, your hash is not distributing well — maybe you have one big flow, like a single backup job to one IP. Luna: Because the hash is per-flow, not per-packet. A single TCP connection will always go to the same interface. So you need many concurrent flows to utilize both links. Lucas: Right. That's why 802.3ad with layer3+4 hashing helps — it uses source and destination IP and port, so different connections from different VMs or clients get spread out. But a single huge file transfer still pins one link. Luna: So bonding isn't a magic bandwidth multiplier. It's about redundancy first, and aggregate throughput for many flows second. Lucas: Exactly. For a storage server with lots of NFS clients, bonding makes sense. For a single high-throughput flow, you might need link aggregation at the switch level or just buy a faster NIC. Luna: One more mode worth mentioning: balance-alb, mode 6. That's adaptive load balancing. It doesn't require any switch config — it handles both inbound and outbound load balancing by ARP negotiation. Lucas: Right. balance-alb is great when you can't touch the switch. It works by the kernel intercepting ARP replies and rewriting the source MAC to different slaves so that the switch sends traffic to different ports. Outbound is balanced by a hash. It's not as robust as 802.3ad but very practical. Luna: But it has a downside: it can confuse some switches or routers if they see MAC addresses changing on the same physical port. And it doesn't work with all NICs. Lucas: Yeah, you need NICs that support the specific hardware features for packet rewriting. Intel and Broadcom usually work. Still, it's a good backup plan. Luna: Speaking of backup plans — honestly, if today's walkthrough saved you from a late-night outage or just gave you a clean config to reference, that's exactly the kind of thing that makes listener support worth mentioning. Lucas: Yeah, and the way that support works is really simple: if it was worth a coffee to you, head over to buy me a coffee dot com slash fexingo. That's it. Helps keep the show ad-free and focused on the nuts and bolts. Luna: And we really appreciate it. Now, back to bonding — one thing we didn't cover: what about VLANs on top of a bond? Lucas: Good question. You can absolutely create VLAN interfaces on top of bond0. For example, 'iface bond0.100 inet static' with vlan raw device bond0. The bond handles the physical redundancy, and the VLAN tag is added at the bond level. Works great with 802.3ad on trunk ports. Luna: So you can have multiple VLANs riding over the same bonded links, and each VLAN gets the failover and load balancing. Lucas: Exactly. That's common in data centers — one bond per host, carrying management, storage, and VM traffic on different VLANs. Just make sure your switch trunk port allows those VLANs. Luna: Alright, I think we've covered the essentials. Next time someone asks me about server network reliability, I'm sending them to this episode. Lucas: Good. And remember, test your failover before you need it. Pull that cable in a maintenance window, not during a crisis.