Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Fails in Production
Transcript
- Lucas: Luna, have you ever pushed code with a fully green test suite, only to have it blow up in production within an hour? Luna: More times than I'd like to admit. It's that sinking feeling when your CI pipeline says everything's fine, but real users are hitting errors. Lucas: Exactly. And I think there's a dangerous assumption a lot of teams make: if all automated tests pass, the software is ready. But the gap between a green suite and production readiness can be enormous. Luna: So what's causing that gap? Is it the tests themselves, the environment, or something else? Lucas: It's a combination, but let's focus on one vivid example. In early 2025, a major online payments platform — one you've definitely used — rolled out a backend update that passed hundreds of unit and integration tests. Within thirty minutes of deployment, a data-race condition under peak load caused payment confirmations to be silently dropped. The test suite had never exercised that exact concurrency pattern. Luna: Right, because their tests ran in a low-concurrency environment. They didn't simulate the real-world traffic spike. Lucas: Precisely. And that's the core issue: most test automation validates isolated correctness — does this function return the right value? Does this API respond with 200? But production failures often emerge from emergent behavior under load, from timing issues, from interactions between services. Luna: So the green suite gives a false sense of security. It's like checking that each brick is solid but never testing whether the wall holds up in a storm. Lucas: That's a great analogy. And the really tricky part is that teams often measure test coverage as a proxy for quality. They say, we have 90 percent line coverage, we're good. But coverage just tells you what code got executed, not under what conditions. You can have 100 percent line coverage and still miss the bug that brings down your system. Luna: I've seen that. A team I worked with celebrated hitting 95 percent coverage. Then a production incident revealed they'd never tested the scenario where a third-party API returned a corrupted payload. The code path was covered, but the test data was too sanitized. Lucas: Exactly. That's the 'happy path bias' — tests tend to verify expected inputs, not the weird, broken, or malicious ones. And even when you do write negative tests, they're often static. They don't evolve as the system grows. Luna: So what should teams do differently? I mean, we can't test every possible production scenario in CI. That would take forever. Lucas: No, you can't. But you can shift some of your testing into production itself, in a controlled way. That's where techniques like synthetic monitoring and canary deployments come in. Luna: Synthetic monitoring — that's where you run automated transactions against the live system, right? Like a bot that goes through a checkout flow every few minutes. Lucas: Yes. And the key insight is that synthetics run in the actual production environment, under real network conditions, hitting real databases. If a deployment breaks a critical path, you'll see it in minutes — not after a customer complains. The payments platform I mentioned now runs a suite of synthetic transactions every sixty seconds across different regions. Luna: That makes sense. But synthetics alone won't catch every data race either. They run a limited number of scenarios. Lucas: Right, they're not a silver bullet. That's why you layer on canary deployments. Instead of routing all traffic to the new version at once, you send a small percentage — say one percent — of real users to it. You monitor error rates, latency, and business metrics. If something looks off, you roll back before the blast radius grows. Luna: So you're using real production traffic as a test. That's powerful, but it requires good observability. You need to know what 'normal' looks like. Lucas: Exactly. And that's its own discipline. But when done right, you catch issues that no amount of pre-production testing would find. The payments platform now deploys with a two-phase canary: two percent for ten minutes, then ten percent for another ten minutes, before full rollout. They've caught two critical bugs in the last year this way. Luna: So the safety net isn't just in your CI pipeline — it's in how you release and monitor. That changes the whole mindset from 'test before deploy' to 'test during deploy'. Lucas: Yeah. And I want to emphasize something: I'm not saying stop writing unit tests or integration tests. Those are still essential for catching logic errors early. But teams need to recognize the limits of those tests and invest in production-verification techniques as a complement. Luna: Before we go deeper, I want to pause for a quick note. This podcast is listener-supported and we keep it ad-free by design. If today's episode was useful to you and you want to help us keep producing these conversations, you can do that at buy me a coffee dot com slash fexingo. Lucas: Even a few dollars from listeners who get value from the show makes a real difference. It keeps us independent and focused on substance over sponsorship. So thank you to anyone who chips in. Luna: Now, back to the topic — one technique I've seen work well is combining canary releases with feature flags. You can toggle the new code on for a small group without doing a full deploy. Lucas: That's a great point. Feature flags give you even finer control. You can enable a feature for internal users first, then a subset of beta customers, and ramp up gradually. And if something breaks, you flip the flag off — no rollback needed. Luna: So the test suite still catches the simple bugs, but the real verification happens in production with real traffic. That shifts the conversation from 'does it work?' to 'does it work for real people?'. Lucas: Exactly. And there's another dimension: test data management. A lot of teams run tests against synthetic data that's too clean. They never see the weird edge cases that exist in production — like a user with a hyphenated name that breaks a substring operation, or a timestamp from a timezone the code doesn't handle. Luna: So using production data — anonymized, of course — in your test environments could expose those issues earlier. Lucas: Yes, but that comes with compliance and privacy challenges. An alternative is to use production traffic replay. You capture real requests in production and replay them against a staging environment. That gives you realistic load and realistic data without exposing PII. Luna: Traffic replay is powerful. I know some companies use it to test database migrations. They record actual queries and replay them against the new schema to catch regressions. Lucas: Right. And that brings us back to the core theme: your test suite is not a quality guarantee. It's a safety net with specific holes. The more you understand where those holes are, the better you can patch them with complementary practices. Luna: So let's wrap up with a takeaway. If you're an engineer or QA lead listening, what's the one thing you should examine tomorrow? Lucas: Look at your last three production incidents. Ask: would any of them have been caught by our existing automated tests? If the answer is no for more than one, you have a gap. Then ask: could synthetic monitoring or canary analysis have caught them? That tells you where to invest next. Luna: That's a concrete starting point. And maybe the most important lesson is to stay humble about what your test suite can guarantee. Lucas: Absolutely. The goal isn't to eliminate all bugs — that's impossible. The goal is to minimize the time between a bug being introduced and it being detected. Shifting some of that detection into production, safely, is how you get there. Luna: Well said. Thanks for digging into this, Lucas. Lucas: Thanks, Luna. And thanks to all of you for listening.