Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Production Without Breaking It
Transcript
- Lucas: There is a pervasive myth in engineering management that production is the one environment you must never touch with untested code. It feels safe, right? You have your pristine staging server, your mocked databases, and your controlled user flows. But here is the problem: staging is a lie. It cannot replicate the chaotic, messy reality of live network conditions, concurrent user behavior, or the sheer volume of data that accumulates over years. Luna: Exactly. I remember working on a payment gateway where our staging environment looked perfect until we hit five thousand concurrent transactions, and then everything just... froze. Staging doesn't scale like that. Lucas: And that freeze happened because staging lacks the friction of the real internet. Today, we are talking about how to test production safely using techniques like canary deployments, feature flags, and shadow traffic. The goal isn't to break things; it is to learn from them before they break for everyone else. If today's deep dive into production safety gives you something usable for your team, consider supporting this show by visiting buy me a coffee dot com slash fexingo. Even a couple of dollars a month helps keep these technical deep dives ad-free and sustainable. Luna: That makes sense, especially when the alternative is waking up to a PagerDuty alert at three in the morning. So, let's start with the basics: what does a canary deployment actually look like in practice? Lucas: A canary deployment is essentially a gradual rollout. Instead of pushing code to one hundred percent of your servers at once, you push it to maybe one percent. You monitor that small slice closely for errors, latency, or unusual resource consumption. If the metrics hold steady, you slowly increase the percentage to five percent, then ten, all the way up to full release. It is like dipping a toe in the water before jumping in. Luna: But how do you decide which traffic goes to the canary? Is it random, or do you target specific user segments? Lucas: It can be both. Random sampling is good for catching general instability. But targeting specific user segments is often smarter. For example, you might route only internal employees or beta testers to the new version first. This acts as a filter for obvious bugs before the general public sees anything. It reduces the blast radius significantly if something goes wrong. Luna: Right, because internal users are more forgiving and faster to report issues. But what if the bug only appears under heavy load? That won't show up with just a few internal testers. Lucas: That is where shadow traffic comes in. Shadow traffic involves sending a copy of live production requests to your new code path while the original code still handles the actual response. The new code processes the request but discards the result. It is purely observational. You get to see how the new logic behaves with real data, real complexity, and real timing, without affecting any end user. Luna: So you are comparing the output of the new code against the old code in real time? That sounds computationally expensive. Lucas: It is, but modern infrastructure makes it manageable. You don't need to run the entire backend twice. Often, you just need to validate the final output or check for specific exceptions. The key is automation. You want scripts that compare the shadow results against the original results and flag discrepancies automatically. If the divergence exceeds a certain threshold, you stop the test immediately. Luna: Automatic divergence detection is clever. It turns subjective code review into objective data analysis. But what happens if you see those discrepancies spike? Lucas: Then you trigger an automated rollback. This is the critical part. You cannot rely on a human to notice a slow creep in error rates at two in the morning. Your monitoring system needs to watch key performance indicators like latency, error rates, and throughput. If any of those cross a predefined threshold, the system should automatically roll back the change or divert traffic away from the canary instance. Luna: So the safety net isn't just observation; it's immediate action. That requires a lot of trust in your monitoring stack. Lucas: Absolutely. If your monitoring is flawed, your rollback will be too. You need granular visibility. Not just 'is the server up?', but 'is the database query taking fifty milliseconds longer than usual?' or 'are we seeing a spike in five hundred internal server errors?' These subtle signals are often the first warning signs of trouble. Luna: I've seen teams skip that granularity because setting it up takes time. They just wait for users to complain. Is that really worth the risk? Lucas: It is a false economy. The cost of fixing a bug in production is exponentially higher than fixing it in staging. And the reputational damage from a widespread outage is hard to quantify but very real. Investing in robust monitoring pays for itself the first time it catches a bug before it impacts customers. Luna: Fair point. But not every company has Netflix-level resources for custom monitoring tools. How do smaller teams implement this? Lucas: They can leverage managed services. Most cloud providers offer built-in canary deployment capabilities. AWS CodeDeploy, Azure DevOps, and Google Cloud Run all have native support for gradual rollouts. You don't need to build the orchestration layer from scratch. You just need to configure the thresholds and define your health checks. Luna: That lowers the barrier to entry significantly. So it's less about building tools and more about configuring existing ones correctly. Lucas: Exactly. The tooling is commoditized now. The differentiator is the process and the discipline. Do you have clear criteria for promoting a canary? Do you know exactly which metrics matter for your specific application? If you are running an e-commerce site, checkout latency is king. If you are running a media platform, video buffering times might be the key metric. Luna: Context matters. A generic 'system healthy' check is useless if the specific function you changed is broken. You need domain-specific metrics. Lucas: Precisely. Generic health checks tell you the server is alive. Domain-specific metrics tell you if the business logic is working. You need to tie your monitoring alerts to user journeys, not just server uptime. Luna: That shifts the focus from infrastructure to user experience. Which feels more aligned with modern product development goals. Lucas: It does. And it encourages closer collaboration between engineering and product teams. Product managers need to understand what those metrics mean. Engineers need to understand which features are most critical to the user. When you align on those priorities, testing becomes a shared responsibility rather than a gatekeeping exercise. Luna: I like that. It removes the 'us versus them' dynamic between dev and ops. Everyone is focused on the same outcome: a stable, happy user base. Lucas: Right. And it allows for faster iteration. You aren't waiting for massive monthly release cycles anymore. You can ship small changes daily, with confidence that if something breaks, you will know instantly and can revert quickly. Luna: Speed without stability is chaos. But stability without speed is stagnation. This approach seems to balance both. Lucas: It does. The key is embracing failure as a learning opportunity, not a disaster. When you test in production with controls, a minor glitch is just data. It tells you what didn't work so you can fix it before it scales. It transforms fear into curiosity. Luna: Curiosity is a great mindset for engineering. It turns debugging into discovery rather than blame assignment. Lucas: Exactly. And it builds a culture of transparency. When everyone knows that failures are caught early and handled gracefully, people are less afraid to try new things. Innovation thrives in environments where the cost of experimentation is low. Luna: So, if you're listening and you're still doing big bang releases, maybe it's time to start small. Even just routing one percent of traffic to a new version could save you a weekend. Lucas: Start small, measure carefully, and be ready to pull the plug. That is the essence of safe production testing. It's not about being fearless; it's about being prepared.