Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Load Balancing with HAProxy
Transcript
- Lucas: So today I want to talk about load balancing Linux servers with HAProxy. Not as an abstract concept — I mean a real, concrete setup you could put in front of a pair of web servers this afternoon. Luna: Good. Because I think a lot of people use nginx for this, or they just rely on their cloud provider's load balancer. What makes HAProxy worth the extra layer? Lucas: Right. So HAProxy is purpose-built for load balancing. It's been around since something like 2006, and it's incredibly efficient at layer 4 — that's TCP — and layer 7, which is HTTP. For pure throughput and low latency, it beats nginx in most benchmarks. Plus it has a really clean stats interface and a Unix socket for runtime management. Luna: The stats socket is one of my favorite things. You can disable a server, put it into maintenance mode, all without restarting the process. Lucas: Exactly. Let's start with a basic scenario. You have two Linux servers running Apache or nginx, serving the same web app. They're on 192.168.1.10 and 192.168.1.11 on port 80. You want to put HAProxy on a third machine — say 192.168.1.5 — and have it distribute traffic between them. Luna: So the HAProxy server becomes the single point of entry. Clients hit that IP, HAProxy decides which backend gets the request. Lucas: Right. And the config is surprisingly small. You create /etc/haproxy/haproxy.cfg with a global section, a defaults section, a frontend, and a backend. In the frontend you bind to *:80 and tell it to use a certain backend. In the backend you list your two servers with the roundrobin algorithm and health checks. Luna: Health checks are key. Without them, if one server dies, HAProxy will still send traffic there and you get timeouts. Lucas: Exactly. So you add 'option httpchk GET /health' — that defines a health check endpoint. HAProxy will hit that URL every, say, two seconds, and if it gets a non-200 response, it marks the server as DOWN and stops sending traffic there. When the server comes back, HAProxy automatically re-enables it. Luna: I've seen setups where they check a more specific path, like /healthz, and the app actually checks its own dependencies — database, cache — before returning 200. Lucas: That's a great pattern. It's called a health endpoint with deep checks. So your backend section looks something like: backend webservers, balance roundrobin, server web1 192.168.1.10:80 check inter 2000 rise 2 fall 3, server web2 192.168.1.11:80 check inter 2000 rise 2 fall 3. That's it. Luna: Inter 2000 means check every two seconds. Rise 2 means it needs two successful checks to be marked up. Fall 3 means three failures before it's marked down. Lucas: Yeah. And if you want to do a zero-downtime deployment, you can connect to the HAProxy stats socket, disable web1, do your update on web1, re-enable it, then disable web2, update that one, re-enable. No traffic lost. Luna: How do you connect to that socket? I remember it's a Unix socket, not a TCP port. Lucas: In the global section you add 'stats socket /var/run/haproxy.sock mode 600 level admin'. Then you can use socat to talk to it: echo 'disable server webservers/web1' | sudo socat stdio /var/run/haproxy.sock. It responds immediately with nothing on success, or an error if something's wrong. Luna: That's slick. So you can script that in Bash. For a rolling deploy, you could loop through each server, disable, update, wait for health, then re-enable. Lucas: Exactly. And HAProxy supports many more algorithms than roundrobin. There's leastconn, which sends new connections to the server with the fewest active connections — good for long-lived connections like WebSockets. There's source, which hashes the client IP so the same client always hits the same server — useful for sticky sessions without cookies. Luna: But roundrobin is fine for most cases. Especially if your requests are uniform and fast. Lucas: Right. Now let's talk about layer 7 — HTTP mode. If you add 'mode http' in the defaults, HAProxy can inspect headers, cookies, and paths. You can route traffic based on the URL: for example, /api/ goes to another. Luna: So you could use HAProxy as an API gateway too, not just a simple load balancer. Lucas: Yes. You use ACLs — access control lists. For example: acl is_api path_beg /api, then use_backend api_servers if is_api. The default backend could be your web servers. That's a common pattern for microservices. Luna: I've also seen people terminate SSL at HAProxy. You can offload the TLS handshake and then pass plain HTTP to the backends. Lucas: That's a big performance win because it reduces the CPU load on your app servers. You add 'bind *:443 ssl crt /etc/ssl/certs/yourcert.pem' in the frontend. Then the backend servers just listen on port 80. But you should probably use a separate internal network for that. Luna: Security in depth. Don't let plain HTTP leave the data center if you can help it. Lucas: Another thing: HAProxy has a built-in stats page. Enable it by adding a listen section: listen stats, bind:8080, stats enable, stats uri /, stats auth admin:password. Then you can open a browser and see live metrics — connections per server, status, bytes in/out. Luna: That's really useful for monitoring. But for production you'd probably lock that down to a specific IP. Lucas: Absolutely. Add a 'bind:8080' with an ACL restricting source IP. Or you can use the stats socket and parse the output with scripts. Luna: I've done that — 'echo show stat | socat...' gives a csv like output. You can pipe it to awk to get specific fields. Lucas: Let's talk about high availability for HAProxy itself. If you only have one HAProxy, that's a single point of failure. The classic solution is Keepalived with a virtual IP — VRRP. Two HAProxy servers share a virtual IP, and if the master dies, the backup takes over. Luna: That's a good setup. Keepalived monitors the haproxy process. If it fails, it lowers the priority and the other node grabs the VIP. Lucas: Yeah. The config for Keepalived is pretty straightforward. You define a vrrp_instance, assign a virtual IP, and set a priority. The one with higher priority becomes master. You can also run a script that checks if haproxy is running and adjusts priority accordingly. Luna: So you have active-passive failover for the load balancer itself. Combined with active-active for the backends, you're in good shape. Lucas: Exactly. Now, I want to mention something practical about tuning. By default, HAProxy has a maxconn setting — maximum concurrent connections. In the global section you can set 'maxconn 4000'. But you also need to increase the system's file descriptor limit. ulimit -n 65535 or set it in systemd. Luna: That's a common gotcha. HAProxy says it can handle tens of thousands of connections, but only if the OS allows it. Lucas: Right. And one more thing: timeouts. In the defaults section, set 'timeout connect 5000', 'timeout client 50000', 'timeout server 50000'. These prevent hung connections from eating up resources. 50 seconds is reasonable for most web apps. Luna: You can also set 'timeout http-request 10000' to guard against slow loris attacks. Lucas: Good point. So the full config for a basic setup is maybe 30 lines. Let me read a minimal one: global, daemon, maxconn 256, stats socket /var/run/haproxy.sock mode 600 level admin. Defaults: mode http, timeout connect 5000, timeout client 50000, timeout server 50000, option httpchk GET /health. Frontend main: bind *:80, default_backend webservers. Backend webservers: balance roundrobin, server web1 192.168.1.10:80 check inter 2000 rise 2 fall 3, server web2 192.168.1.11:80 check inter 2000 rise 2 fall 3. Luna: And you'd start it with 'systemctl start haproxy' and enable it. Then test by hitting the HAProxy IP from a browser or curl. Lucas: Exactly. And you can watch the traffic distribution in real time with the stats page or by tailing the logs. HAProxy logs to syslog by default, usually in /var/log/haproxy.log if you configure rsyslog. Luna: I want to ask about something we touched on earlier — the stats socket. Can you give a quick real-world example of using it in a deploy script? Lucas: Sure. Let's say you have a Bash script that does: for server in web1 web2; do echo "disable server webservers/$server" | socat stdio /var/run/haproxy.sock; echo "Updating $server..."; ssh $server 'systemctl reload apache2'; echo "enable server webservers/$server" | socat stdio /var/run/haproxy.sock; sleep 5; done. That's a zero-downtime deploy in about 10 lines. Luna: Beautiful. And you can add a check to see if the server comes back healthy before moving on. Lucas: You could parse 'show stat' for that server's status field. If it's not UP after a timeout, abort the deploy. Luna: Alright, this has been really practical. I think listeners can take this and set up a basic HAProxy this week. Lucas: If today's tech conversation gave you something usable, you know, a couple of dollars a month genuinely keeps these shows going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. It helps us stay ad-free and focused on content like this. Luna: Yeah, it really does make a difference. Every little bit helps keep the server running and the episodes coming. Lucas: So coming back to the practical side — one thing I didn't mention is that HAProxy also supports HTTP/2. If you're running modern web apps, you can enable it with 'bind:443 ssl crt... alpn h2,http/1.1'. That allows clients to negotiate HTTP/2 over TLS. Luna: Good to know. And it works seamlessly with the same backend config. Lucas: Right. So to wrap up: HAProxy is a powerful, lightweight load balancer that every Linux sysadmin should have in their toolkit. With a minimal config, you get health checks, runtime management via socket, stats, and failover options. Whether you're balancing two web servers or a hundred, it scales beautifully.