Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How End-to-End Testing Misses Real User Pain Points
Transcript
- Lucas: So back in 2024, a major airline — I'm not naming names, but think one of the big US carriers — spent six months building an end to end test suite for their new booking system. Hundreds of tests, all green in CI, every single sprint. They launched, and within a week, customer support was flooded with complaints about the search results being slow and error messages that made no sense. Luna: Wait — their tests didn't catch slow search or confusing errors? That's like the whole point of end to end testing. Lucas: Right, and that's exactly the blind spot. Their E2E tests were validating the happy path — they'd script a user searching for a direct flight on a specific date, clicking the first result, and booking it. But real users search for complex itineraries, they hit edge cases like 'no flights found', they get interrupted and come back. The tests were passing because they were too rigid and too shallow. Luna: So the tests were essentially checking that the system works under ideal conditions — which it did. But real-world conditions broke it. Lucas: Exactly. And this is a pattern I see everywhere. Teams pour resources into E2E tests because they feel comprehensive. But they often miss the actual user pain points because the tests are designed by engineers who already know how the system should behave. There's a term for this — 'happy path bias'. Luna: Happy path bias. I like that. So how do you counteract it? You can't just tell testers to 'think of more edge cases' — that's not systematic. Lucas: No, you need a different approach entirely. One thing that works is user-journey mapping before you write a single test. Sit down with actual customer support transcripts, session recordings, maybe even call recordings if you have them. Map out the journeys that real users take — including the frustrated ones. Then design your tests around those journeys, not around the system's expected behavior. Luna: So you're saying E2E tests should be derived from user data, not from the spec document. That's a shift. Lucas: It's a big shift. And it's not just about test design. Even if you map the right journeys, your E2E tests still run in a controlled environment — they don't see production variability. The airline's tests ran against a test database with five flights and three users. Production had millions of flights and thousands of concurrent users. The slow search? Caused by a database query that worked fine with small datasets but blew up under load. No test caught it because the test data wasn't realistic. Luna: So test data entropy — we talked about that in episode 102. But even with realistic data, you can't simulate every production scenario in staging. That's why some teams are moving toward 'testing in production' — not as reckless as it sounds. Lucas: It's not reckless if you do it right. Feature flags, gradual rollouts, observability. The idea is you combine E2E tests with production monitoring. So your E2E tests give you a baseline — 'the booking flow works for this known scenario'. But you also have real user monitoring that tracks actual performance and error rates. When a real user hits a slow search, you catch it immediately, not after a week of complaints. Luna: That's the observability piece. And it's interesting because it changes the role of QA from 'finding bugs before release' to 'ensuring the system behaves well in the wild'. That's a different mindset. Lucas: Yes, and it's more aligned with what users actually care about. They don't care that your test suite has 95% pass rate. They care that the page loads in under two seconds and the error message tells them what to do. I've seen e-commerce companies where the checkout flow passes every E2E test, but real users abandon the cart because the payment form doesn't handle spaces in credit card numbers — the test always enters the number without spaces. Luna: Oh, that's a classic. And the fix is trivial — strip spaces on the backend — but the test never caught it because the test data was too clean. So how do you systematically find those gaps? Is there a methodology? Lucas: There's a few. One is 'exploratory testing sessions' where testers deliberately use messy data. Another is using production traffic replay — tools that record real user requests and replay them against a staging environment. That gives you realistic data and realistic user behavior. But honestly, the simplest thing is just to look at your production monitoring dashboards and ask: 'Where are users struggling?' then write tests for those specific struggles. Luna: So it's a feedback loop: production data informs test design, tests catch issues before they hit production again. That makes sense. But it requires a culture where QA and operations actually talk to each other. Lucas: Right, and that's often the harder part. I've seen teams where QA doesn't have access to production monitoring tools, or ops doesn't share error logs. But if you can break those silos, you get a much more effective testing strategy. And it doesn't mean you throw away your E2E tests — you just supplement them with real-world data. Luna: Let's talk about a concrete example. I was reading about a fintech company that used this approach. Their E2E tests for the payment flow passed perfectly. But production monitoring showed that users on slow networks were timing out during the three-second payment processing window. The test environment had a fast local network, so it never triggered. Lucas: Great example. So what did they do? They added network throttling to their E2E tests — simulated a 3G connection. Suddenly a bunch of tests failed. They optimized the payment flow to handle slower networks, and the timeout rate dropped by 60%. That's a win driven entirely by combining test automation with production observability. Luna: And that's the kind of story that makes me think — if today's episode gave you something usable, something you can apply at work, that's exactly the kind of value we aim for here. And listener support is what keeps this show ad-free and focused on the substance. If it was worth a coffee to you, there's a link — buy me a coffee dot com slash fexingo. No pressure, just if it helped. Lucas: Yeah, it's a small way to keep the conversation going without any ads or sponsors. Appreciate anyone who feels like contributing. So back to the testing blind spots — another thing I see is that E2E tests often ignore the 'unhappy paths' like error handling. For example, a user enters an expired credit card. The test might check that an error message appears, but does it check that the error message is helpful? Or that the user can still navigate back to fix the card without losing their cart? Luna: Right, the test validates that the system returns a 400 error, but not that the user experience doesn't fall apart. That's a deeper level of quality. Lucas: Exactly. And that's where 'user-journey mapping' really pays off. You map out the entire flow — including recovery flows. If a payment fails, what's the next step? Does the user have to start over? Or do they get a clear path to retry? Those are the things that determine whether they complete the purchase or abandon it. Luna: So you're advocating for a shift from 'does the feature work?' to 'does the user succeed?' That's a subtle but important distinction. Lucas: It's the difference between testing and quality assurance. Testing checks that the code behaves as expected. QA ensures that the user achieves their goal. And sometimes the code is correct but the user still fails. That's the blind spot we keep hitting. Luna: I think one reason teams fall into the happy path trap is that E2E tests are expensive to write and maintain. So they focus on the most common flows and hope for the best. Lucas: That's a real concern. E2E tests are slow and brittle. If you try to cover every edge case, your CI pipeline becomes a bottleneck. So you have to be strategic. Use a small set of E2E tests for critical user journeys, and then complement with unit tests, integration tests, and production monitoring. Don't try to catch everything with E2E. Luna: So what's a good heuristic for choosing which journeys to cover with E2E? I've heard some teams use the '80/20 rule' — cover the 20% of journeys that account for 80% of user actions. Lucas: That's a good start. But I'd also include journeys that are high-risk — like payment, account creation, or data export. Even if they're not the most common, if they break, the impact is severe. And then for everything else, rely on faster test types and monitoring. I worked with a team that had 50 E2E tests — but 30 of them were for payment flows alone. That was deliberate. Luna: That makes sense. And it's a more sustainable approach. So to wrap up — the key takeaway is that E2E tests are necessary but not sufficient. They need to be informed by real user data, combined with production monitoring, and supplemented by other test types. Otherwise, you're just validating your own assumptions. Lucas: Exactly. And that airline example shows it perfectly. They had great test coverage on paper, but their users still suffered. So the next time you see a green CI pipeline, ask yourself: 'Does this actually mean users are happy?' If you don't have the data to answer that, you've got a blind spot. Luna: Good note to end on. Thanks, Lucas. Lucas: Thanks, Luna. Talk to everyone next time.