Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Your Test Suite Can Predict Production Failures with Anomaly Detection
Transcript
- Lucas: So you run your test suite, everything passes green, you deploy to production, and then forty-five minutes later, your p99 response time spikes from two hundred milliseconds to four seconds and your error rate jumps to twelve percent. Luna: Right. The classic 'tests pass, production burns' scenario. And the frustrating thing is, the bug was probably already visible in your test data, you just weren't looking at the right signals. Lucas: Exactly. And that's what we're talking about today — using anomaly detection on the metrics your tests generate to predict production failures before they happen. Not just pass or fail, but the actual behavior of the system under test. Luna: I love this angle. Before we dive too deep — and I know this is a bit of a pivot, but it connects — we deliberately keep this show ad-free because we think honest, specific conversations about testing shouldn't be interrupted by sponsorship breaks. If that kind of independent content is valuable to you, and you want to help us keep it that way, you can support the show at buy me a coffee dot com slash fexingo. No pressure, just an option for folks who find these episodes useful. Lucas: Yeah, genuinely, listener support is what makes shows like this possible without a sponsor dictating the topic. So back to anomaly detection — let's talk about a concrete case. A mid-size e-commerce company I spoke with recently had a Node.js service that processed checkout requests. Their unit tests covered every controller, every validation rule, all green. But once a week, around noon on Tuesdays, the service would start throwing timeout errors in production. Luna: And the unit tests couldn't catch it because the bug was a memory leak that only manifested under sustained load. Standard assertions never flagged it. Lucas: Right. But here's what they did: they started recording, for every test run, the heap memory usage after each test, the number of open file descriptors, and the average response time of a health check endpoint they called at the end of the suite. They ran this for two weeks to build a baseline, then set up simple thresholds — anything more than three standard deviations from the rolling mean triggered an alert. Luna: And did it catch the leak? Lucas: On the fifth day, the anomaly detection flagged heap memory growing consistently across consecutive test runs — it had increased about fifteen percent over the previous day's baseline. The team looked at the recent commits, found a change that wasn't properly cleaning up database connections, rolled it back, and the production incident never happened. Luna: That's the dream. But let's be honest — anomaly detection in testing has a reputation for false positives. How do you keep the signal clean? Lucas: The biggest mistake is using a static threshold. Your test environment isn't perfectly stable — maybe a build machine is slower today because another job is running, or your test data set got regenerated. So you need dynamic baselines that adjust for time of day, day of week, and even the specific test suite. The team I mentioned used a seven-day rolling window with seasonal decomposition — they compared Tuesday's metrics to the previous four Tuesdays, not to Monday's. Luna: So you're basically accounting for the fact that your test infrastructure has its own rhythms. That makes a lot of sense. Lucas: Exactly. And you also need to choose the right metrics. Memory and CPU are obvious, but I've seen teams get great results from tracking network I/O, database query latency, and even garbage collection pause times. The key is picking metrics that correlate with the kinds of failures you've seen before. Luna: What about tooling? If I'm a small team, I don't want to build a whole platform. Lucas: You don't have to. There's an open-source library called PyOD — Python Outlier Detection — that implements dozens of anomaly detection algorithms. You can write a simple script that reads your test result JSON, extracts the numeric metrics, and runs isolation forest or a local outlier factor algorithm. I've seen teams get it working in an afternoon. Luna: And if you're already using Datadog or New Relic for production monitoring, you can often feed test metrics into the same dashboards and use their built-in anomaly detection features. Lucas: Right. One team I know sends their test run metrics as custom metrics to Datadog, then sets up monitors with the 'anomaly detection' function based on past behavior. They get alerts like 'Heap memory in test is 3-sigma above baseline' right alongside their production alerts. Luna: That's elegant. So the test suite becomes not just a gate, but an early warning system. Lucas: Exactly. And it catches things that deterministic assertions never will. Like a slow resource leak that takes twenty test runs to accumulate, or a performance regression that's only visible when you look at the distribution of response times, not just the average. Luna: Let me push back on one thing though. If you have a test that always takes three seconds to run, and suddenly it takes five seconds, that might be because the CI runner is under load, not because your code has a bug. How do you distinguish infrastructure noise from actual regressions? Lucas: That's a great point. The trick is to include infrastructure-level metrics in your model. If you also record the CPU utilization of the runner itself, you can filter out anomalies that correlate with runner load. In practice, I've seen teams use two-level anomaly detection: first on the system metrics to detect noisy infrastructure, then on the application metrics after correcting for that noise. Luna: So you build a model that says 'the test took longer, but only because the machine was busy, so ignore it.' That seems sophisticated but doable. Lucas: It is. And it gets better over time. Once you have a few weeks of data, you can train a simple regression model that predicts expected test duration based on runner load, and flag only the residuals that exceed a threshold. The same company I mentioned is now doing exactly that. Luna: I want to zoom in on one thing you said earlier: 'the kinds of failures you've seen before.' Does this approach help with novel failures too, or only known patterns? Lucas: It can catch novel failures if they manifest as metric anomalies. For example, a new type of memory corruption might cause heap growth that looks different from past leaks. An isolation forest algorithm, which builds random trees to isolate anomalies, can flag that as unusual even if it's never seen it before. It's not perfect, but it's a lot better than nothing. Luna: So you're essentially doing unsupervised learning on your test metrics. Lucas: Exactly. And you don't need a data science team. Even a simple moving average with a rolling standard deviation — you can implement that in ten lines of Python. Start with that, see if it catches anything, then iterate. Luna: I think the practical takeaway here is: don't throw away the metrics from your test runs. Store them, plot them, and start watching for patterns. You might discover that your tests have been telling you about production bugs for weeks, and you just weren't listening. Lucas: That's exactly it. And to close with a forward-looking thought: as test suites grow and systems become more complex, the binary pass/fail model is increasingly insufficient. Anomaly detection is one way to add a predictive layer that turns your tests from a checklist into a monitoring system. I think we'll see this become standard practice in the next few years. Luna: I hope so. And if you try it, let us know what you find. We'd love to hear about a case where anomaly detection caught something your unit tests missed. Lucas: Absolutely. That's all for this episode. Thanks for listening.