Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Needs Canary Releases
Transcript
- Lucas: So you've got a new build. Your staging tests pass. Your unit tests are green. Your integration suite is clean. You push to production — and within minutes, the error rate spikes, users can't check out, and someone's pager goes off. Luna: Right, and we've talked about a lot of ways to catch that earlier — shift-left, better test data, flakiness reduction. But there's one tactic we haven't touched on that directly addresses that moment of deployment. Lucas: Exactly. I'm talking about canary releases. The idea is simple: instead of shipping a change to every user at once, you route a small percentage of real traffic — say two percent — to the new version. If nothing breaks, you gradually increase that percentage until it's serving everyone. Luna: So it's like a coal mine canary. If the canary dies, you stop. But instead of a bird, you're watching metrics like error rate, latency, and throughput. Lucas: Exactly that. The name comes from the old mining practice — miners would bring a canary into the tunnel because the bird would show signs of toxic gas before humans would. In software, the 'toxic gas' is a regression that your pre-production tests missed. Luna: And they almost always miss something. I think the stat from the 2023 State of DevOps Report was that teams with mature canary practices have 15 times lower change failure rates than those that don't. Lucas: I've seen that number. And it makes intuitive sense — you're literally testing in production, but with a safety net. Let me give you a concrete example. There's a payment processing company — I'll call it PayFlow — that processes tens of millions of transactions a day. Luna: Okay. Lucas: They had a feature that recalculated tax on refunds. The logic looked correct in staging. Unit tests covered every branch. But when they pushed to production, a subtle data-corruption bug meant that for a subset of multi-currency transactions, the refund amount got stored in the wrong currency field. Luna: Oh, that's nasty. If it had gone to all users, they'd have had to reconcile millions in accounting errors. Lucas: Exactly. But they were running a 2 percent canary. Within five minutes, their automated monitoring detected that the 'refund discrepancy' metric — which they'd instrumented specifically for this — had jumped. The pipeline auto-rolled back. The bug affected exactly 340 transactions before the canary was killed. Luna: Three hundred forty out of potentially millions. That's the power of a small blast radius. Lucas: Right. And they fixed the bug in an hour, re-ran the canary, and shipped safely the same day. Now, the key question is: how do you actually set up a canary? There are a few components. Luna: Break it down for me. Lucas: First, you need a way to split traffic. That can happen at the load balancer level, the API gateway, or via a service mesh like Istio or Linkerd. You configure a routing rule that sends, say, 2 percent of requests to the new version and 98 percent to the old one. Luna: And that traffic split needs to be deterministic enough that you can correlate the metrics. You don't want the same user bouncing between versions on consecutive requests. Lucas: Great point. That's why sticky sessions based on user ID or a hash of the session cookie are common. The same user always hits the same version during the canary window. Luna: Second component: observability. You can't know if the canary is healthy without real-time metrics. You need error rate, latency percentiles — especially p99 — throughput, and any business-specific counters like 'successful checkout' or 'refund processed'. Lucas: And third: automated rollback. The canary should die on its own if a metric crosses a threshold. If error rate jumps by more than 1 percent, or p99 latency increases by 200 milliseconds — kill the canary, route all traffic back to the old version, and alert the team. Luna: I think that's actually the hardest part for most teams. Setting the thresholds so you don't get false positives. If your rollback triggers on every small blip, you'll never ship anything. Lucas: True. But you can start conservative — very sensitive thresholds — and relax them as you gain confidence. The cost of a false rollback is a few minutes of delayed deployment. The cost of a false negative is a full outage. Luna: And there's a cultural dimension too. Canary releases require developers to accept that their code might not go to 100 percent of users immediately. Some teams push back because they want to see their feature 'live' right away. Lucas: That's a real friction point. But the teams I've seen that adopt canaries as the default deployment method — not just for risky changes, but for everything — they develop a muscle. They start treating partial rollout as normal, and they get faster at detecting and fixing problems. Luna: Let's compare canaries to blue-green deployments for a second. In blue-green, you spin up a whole new environment — the 'green' — and then flip a switch to route all traffic to it. That's a binary cutover. You can roll back by flipping back, but during the rollout, 100 percent of users are on the new version. Lucas: Right. Blue-green is great for infrastructure changes where you need a clean environment, but it doesn't give you the gradual exposure that a canary does. If you flip to green and it's broken, everyone sees it. Canaries give you that early warning signal. Luna: And feature flags are a different tool altogether. With a feature flag, you can turn a feature on for specific users or percentages without deploying new code. But feature flags don't test the deployment process itself — they test the feature toggle logic. Lucas: Exactly. Canaries test the whole stack: the new binary, the new config, the new database migrations. Feature flags can be part of a canary strategy — you might use a flag to control who gets a new feature within the canary group — but they're not a substitute. Luna: So if someone on our team wants to start doing canary releases next sprint, what's the minimum viable setup? Lucas: I'd say three things. One: a load balancer or API gateway that supports percentage-based routing. Two: a dashboard with your top three metrics — error rate, latency, and one business metric like 'successful order'. Three: a script that watches those metrics and, if a threshold is exceeded, changes the routing rule to send 100 percent back to the old version. Luna: You can do that with open-source tools. HAProxy or NGINX for routing, Prometheus for metrics, and a simple Python script for the rollback logic. You don't need a commercial tool. Lucas: Not at all. And once you have that, you can start with tiny canaries — 1 percent, 2 percent — and increase the percentage as you gain confidence. Over time, you'll find that your deployment anxiety drops significantly. Luna: I want to loop back to something you said about the cultural shift. Because I think that's the real barrier for most organizations. It's not the technology — it's the mindset. Lucas: Absolutely. I've seen teams where the lead engineer insists on pushing to 100 percent because 'it's just a small change'. And then that small change takes down the checkout page for 15 minutes. The canary would have caught it in 30 seconds. Luna: One thing that helps is making the canary the default pipeline step. If every build goes through a canary — even the ones you're confident about — then it becomes part of the process, not an extra gate. You stop arguing about whether to do it and just do it. Lucas: That's a great practical takeaway. And it aligns with something I've seen at companies that do this well: they measure 'canary duration' and 'canary success rate' as team KPIs. If your canary success rate is 99 percent, you know you're shipping clean code. Luna: You know, speaking of shipping clean code — this whole conversation about canaries and deployment safety is exactly why we keep this podcast ad-free. We want to give you actionable, no strings attached advice that you can actually use on Monday morning. Lucas: That's the idea. And if you find value in that — if today's episode gave you something you'll take to your team — there's a simple way to support the show. It's buy me a coffee dot com slash fexingo. No subscription, no perks, just a way to keep this independent and ad-free. Luna: Yeah. We don't do ads, we don't do sponsors, and that's deliberate. So if you want to help keep it that way, that link is there. And now back to canaries — because there's one more nuance I want to get into. Lucas: What's that? Luna: Progressive exposure. You don't have to just go from 2 percent to 100 percent in one step. You can have multiple stages: 2 percent, then 10 percent, then 25, then 50, then 100. Each stage gives you a chance to validate at higher traffic volumes. Lucas: Right. And each stage should have a cooldown period — say, five minutes — where you observe the metrics before proceeding. That's especially important for catching issues that only manifest under load. A 2 percent canary might not trigger a database connection pool exhaustion, but 25 percent might. Luna: Exactly. So the full pipeline looks like: build, run pre-production tests, deploy to canary group at 2 percent, monitor for five minutes, if healthy increase to 10 percent, monitor, then 25, then 50, then 100. Lucas: And if at any stage the metrics go red, the pipeline aborts and rolls back. That's the ideal. Some teams even automate the progression using a tool like Spinnaker or Argo Rollouts. Luna: Spinnaker actually has a built-in 'canary analysis' step that compares the canary's metrics to the baseline using a statistical test. It can automatically promote or roll back based on the probability that the canary is worse. Lucas: That's a level of sophistication most teams don't need on day one. But it's good to know it exists. For most teams, a simple script with a for loop and some sleep commands is enough to get started. Luna: So the homework for listeners: go look at your deployment pipeline. If you're pushing to 100 percent of users in one shot, pick one service — maybe an internal API that's not customer-facing — and set up a 2 percent canary this week. Lucas: And if you don't have the routing infrastructure yet, set up a second staging environment that mirrors production, and use that as a manual canary before you flip the switch. It's not as good as real production traffic, but it's a step. Luna: I think that's the theme of this whole series: start small, reduce risk, and iterate. Canary releases are one of the highest-leverage practices you can adopt. The ROI is almost immediate. Lucas: Agreed. So next time you're about to hit deploy on that Friday afternoon change — think about the canary. It might save your weekend.