Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Production Traffic Safely with Canary Deployments
Transcript
- Lucas: Most companies treat production like a minefield they want to cross as quickly as possible, usually by deploying everything at once on Tuesday morning. But there is a growing movement of engineering teams that prefer to walk slowly, testing their code against real user traffic while keeping the blast radius small. This approach is called a canary deployment, and it fundamentally changes how we think about software quality assurance. Luna: It feels counterintuitive to test live systems when you have so many pre-production environments available, doesn't it? Lucas: Exactly. The problem with staging environments is that they never perfectly mirror reality. You miss the network latency of a user on a congested mobile connection, or the weird edge case where a customer’s credit card issuer declines a transaction for a reason not listed in your documentation. By routing just one percent of your traffic to the new version, you get those signals immediately without risking a total outage. Luna: So you are essentially buying insurance against bad code by paying with computational resources and observation time. Lucas: That is a great way to put it. Let’s look at how Stripe handles this with their payment routing infrastructure. When they update the logic that decides which bank gateway processes a transaction, they don’t just flip a switch. They deploy the new code to a single data center first, then route a tiny fraction of requests through it. If the success rate stays above ninety-nine point nine percent, they gradually increase that percentage. Luna: And if the success rate drops even slightly, they roll back instantly before most users notice anything changed. Lucas: Right. And here is the critical part that people often miss: you cannot just watch the overall error rate. If you only look at the global error metric, a bug affecting one percent of users might be completely drowned out by normal noise. You need to isolate the metrics specifically for the canary group. This requires sophisticated observability tools that can tag requests based on which version of the code processed them. Luna: Is that tagging process expensive in terms of performance overhead for the application? Lucas: It adds some complexity to your tracing infrastructure, but modern distributed tracing systems handle it efficiently. The cost of a few extra milliseconds per request is negligible compared to the cost of fixing a bug after it has hit ten million users. I remember reading an internal blog post from Netflix discussing how they use chaos engineering principles within their canary tests to ensure the system degrades gracefully. Luna: Chaos engineering in a canary? That sounds like it could cause more problems than it solves. Lucas: Not necessarily. They inject failures into the canary instance intentionally to see if the automated rollback triggers correctly. It is a stress test of your recovery process, not just the code itself. If the system fails to roll back when you expect it to, you catch that operational failure during the canary phase rather than during a real crisis at three in the morning. Luna: So the canary is testing both the software and the team's ability to respond to incidents. Lucas: Precisely. Now, let me ask you about database migrations. How do you handle schema changes when only a subset of users are running the new code? Luna: That is always the trickiest part because you can have two versions of the application talking to the same database simultaneously. Lucas: You are right. The safest pattern is backward-compatible migrations. You add new columns before you remove old ones, and you allow both versions to read and write to the new structure until you are ready to promote the full rollout. If you try to drop a column while half your fleet is still using it, you will crash the older instances. Luna: Backward compatibility seems like a discipline that slows down development but saves lives during outages. Lucas: It absolutely does. And this brings us to the financial side of the equation. What is the actual return on investment for setting up robust canary deployment pipelines? Luna: I would argue it is hard to quantify directly, but the cost of downtime is usually much higher than the engineering time spent building the pipeline. Lucas: Agreed. A single hour of downtime for a major e-commerce platform can cost millions in lost revenue and brand damage. By reducing the mean time to detection from hours to minutes, canary deployments protect that revenue stream. It also reduces the mean time to recovery because the rollback is automated and immediate. Luna: Do smaller companies struggle with this because they lack the infrastructure maturity? Lucas: They do. Many startups start with manual deploys because they are moving fast. But as they scale, the cognitive load of managing releases becomes unsustainable. Automating the canary process allows a small team to manage complex rollouts without needing a large site reliability engineering department. Luna: It shifts the burden from human vigilance to system design. Lucas: Exactly. Human vigilance is unreliable; automated checks are consistent. Speaking of consistency, if today's tech conversation gave you something usable, consider supporting the show directly. Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. It keeps these episodes ad-free and helps us keep digging into these technical details. Luna: Yeah, and it helps us bring more deep dives like this to listeners who care about how things actually work under the hood. Lucas: Let’s circle back to the metrics. When you are monitoring a canary, what is the most common mistake teams make? Luna: I think it is focusing too much on happy path scenarios and ignoring the tail latencies. Lucas: Spot on. If the average response time is fine, but the top five percent of requests are taking ten times longer, that is a sign of resource contention or a locking issue. Those bugs often manifest only under specific load conditions that your canary traffic might trigger. You have to monitor the distribution of latencies, not just the average. Luna: So you are looking for outliers in the data, not just the central tendency. Lucas: Yes. And another common pitfall is not defining clear success criteria before you start the deployment. If you don’t know what constitutes a failure ahead of time, you might hesitate when you see a red flag, allowing the bad code to propagate further than intended. Luna: Decision paralysis is a real risk when you have so much data flowing in real-time. Lucas: It is. That is why automation is key. The system should automatically halt the rollout if certain thresholds are breached, removing the emotional component from the decision. You trust the data, not your gut feeling. Luna: Trust the data, automate the rollback, and move on. Lucas: That is the mantra. Now, let’s talk about feature flags. How do they interact with canary deployments?