Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How End-to-End Testing in Production Catches Real User Bugs
Transcript
- Lucas: Let me ask you something, Luna. When was the last time you ran an end to end test in production, not staging, but the actual live environment? Luna: Honestly? Never. And I think most QA engineers would say the same. Production is sacred — you don't poke around in there. Lucas: Right, that's the conventional wisdom. But here's the thing — staging environments are almost always cleaner than production. They don't have real user data, real network latency, real browser extensions, real third-party API quirks. So you end up testing against a sanitized version of reality. Luna: And then the bug ships anyway. I've definitely had that moment — everything green in staging, then five minutes after deploy, something breaks. Lucas: Exactly. That's the gap a growing number of teams are trying to close with what's called production testing — specifically, running end to end tests against live production traffic without impacting real users. Luna: So how do you run tests in production without breaking things? You can't just fire off random clicks on a live checkout flow. Lucas: You're right — you don't touch real user data or real checkout flows. The key is shadow traffic. You capture a copy of real production requests and replay them against a separate test instance running in production, but isolated. The test service processes the request and checks the response, but it never actually commits the transaction or stores real data. Luna: So it's like a read-only replica of production that mirrors user activity. That way you get realistic load and data patterns without risk. Lucas: Exactly. Some teams take it a step further and run synthetic user flows — automated scripts that simulate a complete user journey, like logging in, searching for a product, adding it to the cart, and starting checkout — but they hit a test account, not a real customer. The test never completes the purchase. Luna: And because it's in production, it hits the real CDN, the real load balancer, the real database replicas. That's something staging can never replicate. Lucas: Right. And that's where you catch the really nasty bugs. Race conditions that only appear under production load. Config drift between staging and production — someone updated an environment variable in production but forgot staging. Or third-party API changes that your mock server never picked up. Luna: I've seen a team at a mid-size e-commerce company lose a whole day debugging why their staging tests passed but production checkout failed. Turned out the production payment gateway had been updated to require a new field, but the staging mock still used the old schema. Lucas: That's exactly the kind of thing production testing catches. And it's not just about catching bugs — it's about confidence. If your end to end tests run in production on every deploy, you know the system works in the real environment, not just a simulation. Luna: But there are risks. What about data contamination? If a test accidentally writes to a production database, you could end up with garbage data or corrupted records. Lucas: That's why guardrails are critical. First, you use feature flags to gate test traffic — the test runs only if a specific flag is enabled, and that flag is tied to read-only permissions. Second, you set up a separate database or use database transactions that auto-rollback. Third, you monitor test execution in real time with alerting if anything anomalous happens. Luna: So you need a pretty mature observability setup. Alerts, dashboards, maybe even a dedicated 'test in production' dashboard that tracks test traffic versus real traffic. Lucas: Absolutely. And you also need a rollback automation — if a production test triggers an error spike or a latency increase, the system automatically disables the test flag and routes traffic away. Some teams even use canary deployments where the new code version only receives test traffic first, then real traffic after validation. Luna: That's basically what Netflix does with their Chaos Monkey — but for testing, not resilience. They run tests in production all the time. Lucas: Exactly. And it's not just Netflix. A lot of SaaS companies now run production end to end tests as part of their CI/CD pipeline. They deploy to a canary, run the tests, and only promote to full production if tests pass. Luna: But that requires your tests to be fast. End to end tests are notoriously slow — sometimes minutes per test. If you're blocking a deploy on a 10-minute test suite, developers get frustrated. Lucas: True. So teams optimize by running only critical path tests in production — login, search, add to cart, payment initiation. They leave the deep functional tests for staging. Luna: So it's a risk-based approach. You test the highest-impact flows in production and everything else in lower environments. Lucas: Right. And over time, as you gain confidence, you expand the set of production tests. But you always keep the safety mechanisms in place. Luna: I want to ask about synthetic monitoring — is that basically the same thing as production end to end testing? Lucas: They're related but different. Synthetic monitoring is typically a continuous, lightweight check — like, 'can the homepage load in under two seconds?' — that runs every few minutes. Production end to end testing is more thorough — it's a full user flow, run on demand as part of a deploy, with assertions about state and data. Luna: So synthetic monitoring tells you the site is up. Production testing tells you the checkout works. Lucas: Exactly. And the really mature teams combine both. Synthetic monitoring gives you always-on visibility, and production tests give you deploy-time validation. Luna: What about the test data itself? If I'm running a test that logs in, I need a real account in production. Isn't that a security concern? Lucas: It can be. The best practice is to create dedicated test accounts with limited permissions — they can't see real user data, they can't place real orders, and they're automatically cleaned up after the test. Some teams even generate synthetic identities that look realistic but are fake. Luna: So you're essentially running a controlled experiment inside production. It's like a scientific method for deployment safety. Lucas: That's a great way to put it. And the more teams adopt this, the more they realize that staging is a necessary but insufficient step. You need production testing to catch the environment-specific issues. Luna: And you also get a side benefit: you're constantly validating your monitoring and alerting systems, because you're generating real traffic with known outcomes. Lucas: Yes — that's a huge bonus. Your test itself becomes a test of your observability. If a test passes but your monitoring doesn't record it, you know you have a blind spot. Luna: So what's the first step for a team that wants to start production testing? They're probably nervous about breaking things. Lucas: Start small. Pick one critical flow — say, user login — and implement it as a read-only test that runs in a canary environment. Use feature flags so you can disable it instantly. Monitor the results for a week. If everything looks good, add another flow. You build confidence gradually. Luna: It also helps to start during low-traffic hours. That way if something goes wrong, the blast radius is smaller. Lucas: Definitely. And this is actually a great moment to mention something: if you're enjoying these deep dives into practical testing strategies, and you find them useful in your day-to-day work, listener support is what keeps the show ad-free and independent. You can help at buy me a coffee dot com slash fexingo. It's a small gesture that makes a big difference for us. Luna: Yeah, and it really does help us keep bringing you specific, actionable content like this. We love digging into the details that actually change how teams work. Lucas: So back to the practical side — once you've got a few production tests running, the next step is to integrate them into your CI/CD pipeline as a quality gate. If the production test fails, the deploy automatically rolls back. Luna: That requires trust in the test's reliability. You don't want a flaky test blocking every deploy. Lucas: Right — flakiness is the enemy. That's why you need to run each production test multiple times before promoting it to a gate. If a test fails more than 1 in 100 times, it's not ready to block deploys. Luna: So you treat the test itself as code that needs to be reliable. You monitor its pass rate and retire tests that are too flaky. Lucas: Exactly. And over time, you build a suite of production tests that are both fast and stable. That's when you really start seeing the benefits — fewer production incidents, faster recovery times, and more confidence in every deploy. Luna: I think the biggest takeaway for me is that production testing flips the mindset from 'don't touch production' to 'production is the ultimate testing ground, but with guardrails.' It's a more mature approach to reliability. Lucas: Absolutely. The goal isn't to break production — it's to prove that the system works in production before your users do. And with the right safeguards, it's not as risky as it sounds.