Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How A-B Testing Finds Bugs Before Users Do
Transcript
- Lucas: So you’ve probably heard the statistic that most users never report a bug — they just leave. But what if you could catch a bug in production, silently, before it ever touches most of your audience? Luna: That sounds like the dream. Are we talking about canary releases again, or something different? Lucas: Related, but distinct. I’m talking about A/B testing as a bug-finding mechanism. We usually think of A/B tests as tools for product decisions — does a red button convert better than a blue one? But the same infrastructure can catch regressions that your pre-production suite missed. Luna: Right, because you’re comparing a control group against a treatment group in real traffic. If the treatment group shows a statistically significant drop in a key metric, that’s a red flag. Lucas: Exactly. And the beauty is that you’re already running these experiments for product reasons. So you get a safety net for free. Let me give you a concrete example from a major e-commerce platform — I’ll call them ShopFast — where this saved them. Luna: I’m listening. Lucas: ShopFast had a team that refactored the checkout page’s payment module. They ran a full regression suite — unit tests, integration tests, even a few end to end tests. All green. But they also decided to ship the change behind a feature flag and run a 50-50 A/B test for a day before rolling out fully. Luna: Smart. So what happened? Lucas: Within four hours, the treatment group showed a 3.2 percent drop in conversion rate — statistically significant at the 99 percent confidence level. The cause? A race condition in the new payment API call that only manifested under real-world network latency. Their test environment used a mock that responded in under 50 milliseconds every time. Luna: Ouch. So the mock was too perfect. That’s a classic pitfall. Lucas: Exactly. And without the A/B test, that bug would have hit 100 percent of users. They would have lost an estimated four hundred thousand dollars in revenue over the weekend before they caught it via monitoring. Luna: So the A/B test acted as a canary in the coal mine. But how do you set this up without false positives? I mean, if you run enough A/B tests, you’ll eventually see a statistically significant difference by chance. Lucas: Great question. The key is to treat the experiment as a hypothesis test with a pre-specified threshold. ShopFast used a two-sided test with a p-value threshold of 0.01 — stricter than the usual 0.05 — because they were specifically looking for regressions, not trying to prove an improvement. Luna: So they’re okay with a slightly higher false negative rate in exchange for fewer false positives. That makes sense for a safety net. Lucas: Right. And they also set a minimum detectable effect of one percent. Any drop smaller than that, they treat as noise. That prevents overreacting to tiny, meaningless fluctuations. Luna: But what about things like the novelty effect? Users might behave differently just because something is new. Lucas: That’s a real concern. The standard mitigation is to run the experiment long enough for the novelty to wear off — typically at least one full business cycle, which for most e-commerce sites is one week. ShopFast’s four-hour window was actually risky; they got lucky the effect was large enough to show up quickly. Luna: So you need to balance speed against reliability. If you’re looking for a regressive bug, you probably want to catch it fast, but not at the cost of a false alarm that wastes everyone’s time. Lucas: Exactly. And there’s another hidden trap: sample ratio mismatch. That’s when the actual split of users between control and treatment deviates from the intended 50-50. It can happen due to caching, load balancers, or even bugs in the assignment logic itself. Luna: And if you don’t check for that, you might think you have a regression when really you just have an imbalance in user segments. Lucas: Right. So ShopFast added an automated check that runs every hour during an experiment — it compares the observed split against the expected split using a chi-squared test. If the p-value drops below 0.001, the experiment is automatically paused and the team gets paged. Luna: That’s a high bar. So they’re essentially saying, if the split is off by more than a fraction of a percent, something is broken. Lucas: Yeah. And it’s caught real issues. Once, a caching layer was serving the same version to all users, so the treatment group was actually empty — but the metrics still looked fine because the control was getting all the traffic. Without the split check, they’d have thought the experiment was working. Luna: So you need both the metric comparison and the split integrity check. That’s a good framework. Lucas: Now, the question is: how do you integrate this into your existing QA pipeline? You don’t want to run A/B tests for every single change — that would be overwhelming. So you need to decide which changes get the treatment. Luna: I’d imagine anything that touches a revenue-critical path — like checkout, signup, payment. Lucas: Exactly. ShopFast has a tiered system. Tier one changes — anything in the checkout funnel — automatically get a 24-hour A/B test with a one percent minimum detectable effect. Tier two changes, like a UI tweak on the product page, get a 6-hour test with a two percent threshold. Everything else just goes through standard canary. Luna: So they’re not testing everything, but they’re testing the highest-risk changes with a safety net. Lucas: Right. And they also have a principle that no experiment meant for bug detection ever runs without a corresponding monitoring dashboard. They track not just the primary metric — conversion — but also secondary metrics like page load time, API error rate, and client-side JavaScript errors. Luna: Because a regression might show up in a secondary metric first. If page load time spikes, conversion might follow later. Lucas: Exactly. And they’ve trained their teams to look at those metrics holistically. One time, a change caused a 200-millisecond increase in load time but no drop in conversion — yet. They caught it, rolled back, and prevented a potential future loss. Luna: That’s proactive. So what’s the one takeaway you want listeners to remember? Lucas: If you already have A/B testing infrastructure, you’re leaving money on the table if you don’t use it as a regression detection tool. It’s not a replacement for unit tests or integration tests — but it’s a powerful complement that catches things your mocks and stubs never will. Luna: And if you don’t have A/B testing yet? Lucas: Start small. Pick one critical user flow — like signup or checkout — and run a 50-50 split with a simple feature flag. Track one primary metric and two secondary ones. You’ll learn more about your system’s behavior in a week than a month of synthetic testing. Luna: That’s a practical on-ramp. And hey, speaking of practical — if today’s tech conversation gave you something usable, you can support the show and keep it ad-free by buying us a coffee. That’s buymeacoffee dot com slash fexingo. Lucas: Yeah, it really helps us keep bringing deep dives like this without any sponsor interruptions. And we genuinely appreciate every single contribution. Luna: Alright, back to the topic. So we talked about using A/B tests to catch regressions. But what about using them to validate test suites themselves? That’s a whole other layer. Lucas: That’s a great point. Some teams run ‘negative A/B tests’ where they intentionally introduce a known bug into the treatment group. If the experiment doesn’t detect a drop, they know their monitoring or their test thresholds are too loose. Luna: Chaos engineering meets A/B testing. I love that. Lucas: Exactly. It’s a way to validate your validation. And it builds confidence that when a real regression happens, your safety net will catch it. Luna: So the key is to treat A/B testing not as a product feature, but as a core reliability tool. Lucas: Exactly. And the best part is that it scales with your traffic. The more users you have, the faster you can detect small regressions. So it’s an investment that pays back more as you grow. Luna: Alright, I think we’ve given people a solid blueprint. Next time, maybe we’ll talk about how to automate the analysis so you’re not staring at dashboards all day. Lucas: That sounds like a plan. Until next time.