Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why End-to-End Tests Fail Without Observability
Transcript
- Lucas: So you have a suite of end to end tests that all pass green. Your CI pipeline says everything is fine. But customers are hitting errors in production. That gap—between a passing test and a working system—is where observability comes in. Luna: Wait—end to end tests are supposed to catch that. Are you saying they don't? Lucas: They can, but only if you instrument them right. A traditional E2E test checks that a button click navigates to the next page, or that an API returns a 200. But it doesn't tell you why a slow backend call caused a timeout for 5 percent of users. That's the observability gap. Luna: So the test passes, but the user experience is still broken. That's the worst kind of false positive. Lucas: Exactly. Let me give you a concrete example. A mid-sized e-commerce company had an E2E test for their checkout flow that always passed in staging. But in production, users on a specific mobile carrier were seeing a blank page after entering payment details. The test never caught it because it didn't simulate that exact network condition—and more importantly, it didn't log the internal state of the payment service. Luna: What was happening inside? Lucas: A third-party payment gateway was timing out after 30 seconds. The E2E test waited 10 seconds and then moved on—it treated a slow response as success. But the production timeout was set to 20 seconds, so users got a failure. The test had no visibility into latency or errors from the gateway. Luna: So the fix isn't just adding more tests. It's adding telemetry to the tests themselves. Lucas: Right. You want each E2E test to emit structured logs, trace IDs, and metrics—just like your production services do. When a test passes but a trace shows a 5-second response time on a critical call, that's a signal. You can catch regressions before they hit users. Luna: That sounds like extra work. How much overhead are we talking? Lucas: It's not trivial, but the investment pays off fast. There are open-source tools like OpenTelemetry that let you instrument your test code with a few lines. You add a span around each user action—login, search, add to cart, checkout. Then you export those spans to a tracing backend. Suddenly you can see the exact waterfall of calls and where time is spent. Luna: So instead of just 'test passed green', you get 'test passed, but the payment step took 4.2 seconds and the third-party call failed once.' That's a lot more actionable. Lucas: Exactly. And once you have that data, you can set up alerts on test metrics—not just test status. For example, alert if the 95th percentile response time of any test step exceeds 2 seconds. That catches performance regressions that don't cause a test failure but degrade user experience. Luna: I'm thinking about the flaky test problem. Could observability help there too? Lucas: Absolutely. Flaky tests are often caused by race conditions or timing issues. If you capture thread dumps, logs, and trace IDs on every test run, you can correlate failures with system state. One team I worked with reduced flaky test reruns by 70 percent just by adding resource-usage metrics to their test harness. They found that CPU spikes during parallel test runs caused timeouts. Luna: So the same techniques that help in production—distributed tracing, structured logging, metrics dashboards—apply to the test environment. Lucas: Yes. And the tooling is converging. A lot of modern observability platforms let you query test data alongside production data. You can compare response times from the same code path in staging vs. production, or spot environment-specific anomalies. It changes the conversation from 'did the test pass?' to 'what is the system doing?' Luna: But isn't there a risk of over-instrumentation? Too many traces and logs can get expensive and noisy. Lucas: That's a real concern. You don't need to instrument every assertion. The key is to focus on the critical user journeys—login, checkout, search—and on any integration points with external services. Start with one end to end scenario that covers your highest-revenue flow. Add a trace around each step. Then expand based on what you learn. Luna: So the first step is picking one flow and adding a single trace. That's manageable. Lucas: Exactly. And once you see the data, you'll find gaps you didn't expect. For that e-commerce company, the first trace revealed that their inventory service was making three redundant database calls per checkout. The E2E test passed fine, but the trace showed unnecessary latency. They optimized it and cut checkout time by 40 percent. Luna: That's a huge win from just adding one trace. I bet the team was sold after that. Lucas: They were. And that's the pattern—observability turns your test suite from a binary pass-fail indicator into a continuous debugging tool. Every run becomes a data point about system health. Luna: If this episode gave you something usable, the way we keep the show ad-free is through listener support. You can buy me a coffee dot com slash fexingo—just a small way to say this kind of deep-dive content is valuable. And honestly, it makes a difference. Lucas: Yeah, that's actually a good point. The kind of specifics we talk about—real numbers, real cases—take time to research. If today's conversation helped you think about your test suite differently, consider throwing a few bucks that way. It really does keep us independent. Luna: Alright, back to observability. Let's talk about the ROI. How quickly does this pay off? Lucas: In my experience, within a sprint or two. The first incident you debug using test traces instead of reproducing manually—that saves hours. One SaaS team I consulted for had a critical bug that only appeared when a third-party API returned a specific error code. Their E2E tests passed because the mock always returned success. But with observability, they saw in the trace that the real API was returning a 429—too many requests. They fixed it the same day. Luna: And without the trace, they'd have spent days wondering why users were seeing errors. Lucas: Exactly. The trace told them exactly where the failure happened and what the response looked like. It's like having a security camera inside your test instead of just a green or red light. Luna: I can see that reducing mean time to detection significantly. Do you have a number? Lucas: In that same team, their mean time to detection for production issues caught by tests dropped from 4 hours to 12 minutes after they added observability to their E2E suite. That's a 20x improvement. And they didn't add a single new test—they just instrumented the ones they already had. Luna: So the tests themselves become more valuable without increasing the test count. That's a strong argument for teams that are already overwhelmed by test maintenance. Lucas: Exactly. The number of tests stays the same, but the signal quality goes way up. And you avoid the trap of adding more tests to compensate for poor visibility—which only makes maintenance worse. Luna: What about tools? You mentioned OpenTelemetry. Is that the standard now? Lucas: It's becoming the standard for instrumentation across languages and frameworks. It supports traces, metrics, and logs—and it's vendor-neutral, so you can export to different backends. For test observability, you can use OpenTelemetry's Java or JavaScript SDKs to add spans in your test code. There are also dedicated test observability platforms like Testmo or ReportPortal that integrate with it. Luna: But do teams really need a dedicated platform? Or can you just pipe traces into your existing observability stack like Datadog or Grafana? Lucas: You can use your existing stack, and that's often better because then you can correlate test data with production data in the same dashboard. Just tag your test traces with a 'test run ID' and an environment label. Then you can filter to see only test data or compare it with production. No extra tool needed. Luna: That makes sense. Use what you already have, just extend it to test environments. Lucas: Right. And the key practice is to treat test runs as first-class citizens in your observability system. Same retention, same alerting policies. When a test fails, you should be able to open a dashboard that shows not just the test logs, but the system metrics from that exact time window. Luna: So the real takeaway is: don't just ask 'does the test pass?' Ask 'what was the system doing when the test ran?' Lucas: Exactly. And that shift—from pass-fail to system understanding—is what makes E2E testing actually reliable. Without observability, you're flying blind with a green checkmark. Luna: Good place to leave it. Next time, maybe we can talk about how to set up those first traces step by step. Lucas: I'd like that. For now, the homework is simple: pick your most critical end to end test and add a single span around the main action. See what you learn.