Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Zero Downtime Deployments With Blue-Green Servers
Transcript
- Lucas: You're pushing a new version of your API to production on a Friday afternoon. Tests passed. Staging looked fine. You deploy, and for about four seconds, every user who reloads gets a 502 Bad Gateway. Nobody died, but your boss notices. The question is: why does a deploy have to cause even a blip? Luna: Right — and most people accept it because they're doing in-place updates. Stop the service, replace the binary, restart. That gap is downtime. Lucas: Exactly. And in-place is fine for a low-traffic internal tool. But if you're running customer-facing anything, there's a pattern that eliminates that gap entirely: blue-green deployment. The idea is dead simple — you run two identical production environments, call one blue and one green. Only one serves live traffic at a time. Luna: So when you deploy, you push to the idle environment, test it, then flip the router. No downtime because the live one never stops. Lucas: That's the whole play. Let me walk through it concretely. Say you've got a Node.js API behind an Nginx reverse proxy. Your blue environment is live on port 3001. Green is idle on 3002. Nginx proxies all requests to 3001. Your deploy script builds the new container, starts it on 3002, runs a health check — maybe curl localhost:3002/health — and if it returns 200, you update the Nginx config to point upstream to 3002, then reload Nginx. Luna: Reload, not restart. Because nginx -s reload doesn't drop connections — it spawns new worker processes with the new config and gracefully shuts down the old ones. Lucas: Which is the key to keeping the switch atomic. The old workers finish handling any in-flight requests before they die. So a user who already had a connection keeps talking to the old backend for that one request. The next request lands on the new backend. Zero dropped connections. Luna: What about the case where the health check passes but the new version has a subtle bug that only shows under real traffic? Lucas: That's where the green field earns its keep. You don't destroy the old environment. You keep it running. If you see errors, you flip Nginx back to the old upstream. The rollback is a single config edit and a reload. No rebuilding, no re-deploying. You're back on the known-good version in under a second. Luna: So the old environment becomes a safety net. How long do you keep it around? Lucas: Depends on your confidence and your traffic pattern. Some teams keep the previous version for an hour. Some keep it for a full day, especially if they do a phased rollout where only a percentage of traffic goes to green first. That's actually a common variation — you can use the load balancer's weight settings to send ten percent of requests to green, watch for errors, then ramp to a hundred. Luna: That's basically a canary release built on top of the same infrastructure. Lucas: Exactly. Blue-green is the foundation. Canary is a refinement. But there's a friction point that trips people up: database schema changes. If your new code expects a column that doesn't exist in the old schema, and both environments share the same database, you've got a problem. Luna: That's the big one. How do you handle migrations when both versions might touch the same tables? Lucas: There are two common approaches. First: make the migration backward-compatible. Add the column as nullable with a default value. Deploy the migration first, while blue is still live. The old code ignores the new column. Then deploy the new code that uses it. If you need to roll back, you reverse the code deploy first, then drop the column later. Luna: That's the read-only pattern. What if your change is more fundamental — say, renaming a table or splitting one column into two? Lucas: That calls for the dual-write pattern. For a period, your application writes to both the old and new schema — maybe via a trigger or application-level dual-write. During that window, both blue and green can operate. Once you've verified green is stable, you backfill historical data and cut over reads. It's more complex, but it's the only way to keep zero downtime with schema changes. Luna: So you're essentially doing a live migration in parallel with the deploy. That adds a lot of testing surface. Lucas: It does. And honestly, that's where a lot of teams decide blue-green isn't worth it for every deploy. They reserve it for major releases and use in-place deploys for hotfixes. Which is fine — you don't need a surgical tool for every cut. Luna: What about infrastructure cost? You're running double the servers during the switch window. Lucas: That's the trade-off. You pay for idle capacity during the deploy window. But for most setups, that's a marginal cost — especially if you're using cloud instances that you can spin down after the old environment is verified. Some teams even run blue and green on the same server, just different ports, so the extra cost is negligible. The real value is the risk reduction. Luna: I've also seen people use DNS switching instead of a load balancer. Set a low TTL, update the A record to point to the new server's IP. Lucas: That works, but DNS propagation is not atomic. Even with a TTL of sixty seconds, some clients cache longer. You get a gray period where some users hit the old server and some hit the new one. For a true zero-downtime switch, you want the flip to happen at the load balancer or reverse proxy level, where you control it directly. Luna: Makes sense. So the ideal setup is: two application instances, a shared database with backward-compatible migrations, and a load balancer you can reconfigure instantly. Lucas: And a script that automates the whole sequence. Let me give you the shell of one — literally a shell script. You'd have variables for the current upstream port, the new upstream port, the health check endpoint. The script starts the new service, polls the health endpoint with a timeout, and if it passes, it sed-replaces the upstream line in the Nginx config and reloads. If the health check fails, it kills the new service and exits with an error. Luna: That's maybe twenty lines of bash. Totally doable. Lucas: Yeah. And you can wrap it in a CI/CD pipeline step. The important thing is that the health check is meaningful — not just 'is the process running?' but 'can it connect to the database and return a valid response?' You want to catch a broken state before traffic hits it. Luna: Let's talk about the edge case nobody thinks about: what about long-lived WebSocket connections? Your Nginx reload drops those if they're not handled properly. Lucas: That's a real gotcha. WebSocket connections are persistent. When you reload Nginx, the old worker processes finish their current HTTP requests and die, but WebSocket connections are not HTTP transactions — they're long-lived TCP tunnels. The default Nginx reload behavior will terminate those connections. The fix is to use the 'nginx -s quit' signal with a longer timeout, or implement a WebSocket-aware health check that drains connections gracefully. Some teams handle this by having the client reconnect on disconnect, which works for chat apps but not for real-time financial data. Luna: So if your app uses WebSockets, blue-green requires a bit more engineering. Not a deal-breaker, but you have to test it. Lucas: Absolutely. And honestly, the same applies to any stateful connection. That's why the pattern works best for stateless HTTP APIs. For stateful services, you often need a different strategy — like session replication or sticky sessions with a drain mechanism. Luna: Sticky sessions kind of defeat the purpose of blue-green, though, because you can't atomically drain all sessions from one environment. Lucas: Right. If you have sticky sessions, a blue-green switch means users who were sticky to the old server will either lose their session or have to re-authenticate. Some teams accept that for a brief window. Others use a session store like Redis that's shared between environments, so session data survives the switch. That's more infrastructure, but it makes the transition seamless. Luna: There's another subtlety: log aggregation and monitoring. When you flip to green, your monitoring dashboards need to know the new environment is now production. Otherwise, you might be looking at blue's metrics and wondering why traffic dropped to zero. Lucas: That's a fantastic point. Your deploy script should also update a monitoring tag or a label in your metrics system. Something like environment=production should point to the active color. Some teams use a simple file, like /etc/environment-color, that their monitoring agent reads. After the flip, they update that file and the agent picks up the change. Luna: So the deploy script doesn't just flip Nginx. It also updates the monitoring config, runs a smoke test against the new production URL, and then optionally tears down the old environment after a cooldown period. Lucas: Exactly. And the cooldown period is crucial. I've seen teams destroy the old environment five minutes after the flip, only to discover that a batch job that runs on a fifteen-minute schedule was still pointing to the old upstream. Keep the old environment running for at least one full batch cycle. Luna: So what's the minimum viable blue-green setup for someone listening who's never tried it? Say they have one server and one API. Lucas: If you have one server, you can still do it. Run two instances of your app on different ports. Use Nginx or Caddy as a reverse proxy. Write a script that does the port switch and reload. That's it. You don't need Kubernetes. You don't need a second server. The pattern scales down to a single $5 VPS. Luna: And the payoff is that you never have to say 'sorry for the brief outage' again during a deploy. Lucas: Right. And honestly, if that script saves you one late-night rollback panic, it's worth the hour it takes to set up. If today's conversation gave you something usable, consider supporting the show — it keeps us ad-free and focused on practical engineering. The link is buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small thing that makes a big difference. We put a lot of work into making these episodes concrete, and listener support is what lets us keep doing that. Lucas: So back to the practical side — let's talk about one more edge case: the database migration rollback. If you deploy a backward-compatible migration and then flip to green, and something goes wrong, you flip back to blue. But the migration already ran. Blue's code expects the old schema. If the migration added a column, that's fine — blue ignores it. But if the migration dropped a column, you've lost data. Luna: So the rule is: never drop a column in the same deploy that introduces the code change. Drop it in a separate, later deploy after you're confident the new code is stable. Lucas: Exactly. That's the safest approach. And it's why blue-green forces you to think about your database changes more carefully. It's actually a good discipline — it makes you write migrations that are reversible and backward-compatible. That's a net positive for your whole engineering team. Luna: It's almost like the constraint of zero downtime makes you a better engineer. Lucas: I think that's true of a lot of operational patterns. The constraint forces you to design for resilience. Blue-green is just one tool in that toolbox, but it's a surprisingly accessible one. You don't need a massive DevOps team. You need a load balancer, a script, and a willingness to test the rollback. Luna: And that's the note I'd leave listeners with: test the rollback. Deploy to green, then deliberately flip back to blue. Make sure your script works both ways. If you only test the forward path, you'll discover the rollback bug at 3 a.m. on a Saturday. Lucas: Exactly. Blue-green gives you a safety net, but only if you practice using it. Run a fire drill. Deploy a dummy change that intentionally fails the health check, and verify that your script aborts and doesn't flip traffic. That kind of testing is what separates a pattern from a prayer.