Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Ship Code Without Breaking Production
Transcript
- Lucas: So here is the thing about deploying software today. You are not just pushing code anymore. You are pushing risk. And if you do it all at once, you are basically rolling dice with your customer experience. Luna: Exactly. It feels like every major outage we hear about started with a confident full rollout that nobody bothered to test in a real-world scenario first. Lucas: Right. And most teams still treat testing as a gate you pass through before you go live. But in modern distributed systems, the production environment is the only place that truly matters. The rest is just rehearsal. Luna: So instead of asking whether we tested it enough, we should be asking how safely we can introduce it. Lucas: That is the shift. And the tool for that shift is called a canary release. It sounds like aviation terminology because it is. You deploy to a tiny slice of users first. Maybe one percent. You watch. If it holds up, you expand. If it breaks, you roll back before anyone notices. Luna: But does everyone actually have the data infrastructure to monitor that one percent effectively? Or are they just guessing? Lucas: Most don't. That is why this fails. I looked at a case study from a mid-sized fintech firm last week. They cut their critical production incidents by ninety percent in six months. Not by writing more unit tests. By changing how they shipped. Luna: Ninety percent is huge. What was the mechanism? Did they hire more QA people? Lucas: No. They hired fewer. They stopped trying to predict failure and started designing for containment. They set up automated rollback triggers based on latency spikes and error rates. If the canary group sees a two hundred millisecond increase in response time, the system rolls back automatically. No humans involved. Luna: That is scary but brilliant. It removes the ego from the equation. You cannot argue with a broken metric. Lucas: It also changes the culture. When you know a bad deploy will be caught in seconds, you stop being afraid to ship. Paradoxically, that makes you safer. Fear causes mistakes. Confidence with guardrails causes speed. Luna: I wonder if smaller companies can pull this off without enterprise-grade observability stacks. Lucas: They can. You do not need Splunk or Datadog right away. You just need basic health checks and a simple dashboard. The principle is what matters. Isolate the blast radius. Luna: Okay, so let's say you are running this canary. How do you decide when to promote it to the whole user base? Is there a magic number? Lucas: There is no magic number. There is an error budget. Every service has a target availability. If you are at ninety-nine point nine percent uptime, you have some budget left. You spend it on risky deploys. If you burn through it, you freeze. You stop shipping until you stabilize. Luna: So it is a finite resource. You have to choose between speed and reliability carefully. Lucas: Exactly. And that forces engineering leaders to make explicit trade-offs. Instead of saying 'we want both,' they have to say 'we are willing to accept five minutes of downtime this month for this new feature.' Luna: I feel like most product managers would panic at that conversation. They want zero risk. Lucas: Zero risk is zero progress. The market punishes stagnation faster than it punishes minor bugs. As long as you catch those bugs early, you are fine. Luna: Speaking of catching bugs early, how do you handle database migrations in a canary setup? That always feels like the fragile part. Lucas: Database migrations are the hardest part. You cannot roll back a schema change easily. So you decouple the deploy from the migration. You add columns first. You write to the old and new columns. You backfill. Then you switch reads. Then you drop the old column. Each step is a separate canary deploy. Luna: That takes forever. Does the business ever get impatient with that pace? Lucas: Sometimes. But the alternative is waking up at three in the morning to fix a corrupted table. I would rather take the slow path. Luna: Yeah, nobody wants to be woken up. But it does require a lot of discipline from the dev team. Lucas: It requires treating infrastructure as code. If your deployment process is manual, you will never scale this. You need scripts that define exactly what happens at each stage. Luna: Do you see this becoming standard across all tech companies within the next few years? Lucas: It already is for mature organizations. For legacy enterprises, it is still a struggle. They are stuck in waterfalls. But the pressure is mounting. Customers expect instant updates. Luna: What about security? Can a canary release hide a vulnerability? Lucas: Yes. That is the danger. If your canary group is too small, a sophisticated attacker might slip past detection. You need diverse traffic in that group. Real users, not just synthetic tests. Luna: So randomness is actually a feature here. You want unpredictable exposure to catch edge cases. Lucas: Precisely. You want the worst-case scenario to happen to one person, not everyone. Luna: I think the biggest barrier is trust. Trusting the automation to kill the process if things go wrong. Lucas: Trust is earned through repetition. You run drills. You intentionally break things in staging to prove the rollback works. Then you trust it in production. Luna: That is a great mindset. Treat failure as data. Lucas: Always. Failure is just feedback with consequences. Make the consequences small, and the feedback valuable. Luna: Before we go deeper, I want to mention something quick. If today's tech conversation gave you something usable, a couple of dollars a month is genuinely what keeps these going. Buy me a coffee dot com slash fexingo. It helps us keep the lights on and the research deep. Lucas: Yeah, we appreciate it. It lets us focus on quality over clicks. Anyway, back to the canary model. How do you measure success beyond just uptime? Luna: You look at business metrics too. Conversion rates, session length, revenue per user. Technical health is half the story. Lucas: Right. If your app loads faster but people buy less, you failed. The canary gives you that granular view. You can see if a UI change hurt sales before you roll it out globally. Luna: So it is really A/B testing with safety rails. You get the best of both worlds. Lucas: Exactly. You learn and you protect. Most teams pick one. The smart ones do both. Luna: Do you think AI will eventually automate the decision to roll back entirely? Lucas: Maybe. But I doubt it. An algorithm can detect a spike. It cannot understand context. Maybe the spike is because of a viral tweet. Rolling back then would be stupid. Humans need to be in the loop for nuance. Luna: Good point. Automation handles the noise. Humans handle the signal. Lucas: Or vice versa. Sometimes the machine sees a pattern the human misses. It is a partnership. Luna: I am curious. Have you seen any companies fail spectacularly by ignoring this advice recently? Lucas: Almost daily. But the big ones usually cover it up quickly. The small ones just die quietly. You rarely hear about them. Luna: That is the silent graveyard of startups. Bad tech debt collecting interest. Lucas: Exactly. So the question is not whether you can afford to test properly. It is whether you can afford not to. Luna: And maybe the answer is simpler than we think. Just start small. One percent. See what happens. Lucas: Start small. Measure everything. Roll back fast. Repeat until it becomes boring. That is when you know you have mastered it.