Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Five Nines Testing Is Overrated for Most Apps
Transcript
- Lucas: So there's this idea in software — five nines — 99.999 percent uptime. It's treated like the gold standard. But I think for most applications, chasing that number with your testing is actually a waste of money. Luna: Okay, I feel like you're about to make a lot of QA engineers uncomfortable. Why is it a waste? Lucas: Because five nines means about five minutes of downtime per year. That's the SLA you see from AWS or Google Cloud. But here's the thing — your average B2B SaaS product? Users tolerate way more than that. If your CRM or project management tool goes down for fifteen minutes, people might grumble, but they don't leave. Luna: Unless it's a payment gateway or a trading platform. Then fifteen minutes is catastrophic. Lucas: Exactly. So the level of testing you need should depend on the actual cost of failure. I came across a case recently — a mid-market CRM provider, about five thousand paying customers. They were running a full regression suite that took eight hours before every deployment. And they were still finding production bugs that the regression suite didn't catch. Luna: Eight hours? That's basically a full workday. How often were they deploying? Lucas: Once a week. So they were spending one-eighth of their development time just waiting for tests to pass. And the regression suite was covering thousands of test cases, many of which tested scenarios that hadn't changed in months. They had this implicit goal of 'zero downtime,' but they hadn't actually defined what that meant for their users. Luna: So they were running a five-nines testing strategy on a three-nines product? Lucas: Right. Their actual uptime was around 99.9 percent anyway — which is about eight hours of downtime a year. Their users barely noticed. So they had this mismatch: the testing effort implied they needed near-perfect reliability, but the product's actual tolerance for failure was much lower. Luna: What did they change? Lucas: They adopted an error budget approach. They set a target of 99.95 percent uptime — that's about four hours of allowed downtime per year. That became their error budget. Then they used chaos engineering to proactively test failure scenarios in production, rather than trying to simulate every possible state in a staging environment. Luna: So they stopped running the full regression suite every week? Lucas: They cut the regression suite by about sixty percent. They kept the critical path tests — login, payment flow, data export — and moved the rest to a weekly run that was allowed to fail without blocking deployment. If a non-critical test failed, it created a ticket, not a deployment halt. Luna: That's a big mindset shift. Most teams treat any failing test as a blocker. Lucas: And that's exactly the problem. If you treat every test as equally important, you end up with a testing strategy that's designed for the worst-case scenario — a financial exchange or a hospital system — when you're actually building a tool that people use to manage their sales pipeline. The cost of a false alarm in testing can be huge: delayed releases, frustrated developers, and a false sense of security. Luna: How did the numbers work out for them? Lucas: Within six months, they cut their QA budget by about forty percent. Their deployment frequency went from weekly to three times a week. And their actual uptime? It stayed at 99.9 percent — exactly where it was before. They just stopped paying to achieve something they didn't need. Luna: That's compelling. But I imagine some teams might push back — saying, 'We can't afford a single outage, our CEO will flip.' Lucas: That's a common fear. But the real question is: what does an outage actually cost you? If your CRM goes down for an hour, do you lose customers? If you're a bank, maybe yes. If you're a project management tool, probably not. You can calculate the cost per minute of downtime — and then compare that to the cost of the testing infrastructure you're running to prevent it. Luna: So you're saying run the numbers before you decide on your testing strategy. Lucas: Exactly. And that's where the concept of service level objectives, or SLOs, comes in. You pick a target based on business impact, not on some industry standard. Then you design your testing to meet that target — not to chase a number you heard at a conference. Luna: It's like the testing equivalent of 'you are what you measure.' Lucas: Right. If you measure uptime, you'll optimize for uptime. But if you measure user satisfaction, you might find that a feature delay annoys people more than a brief outage. That CRM company actually surveyed their users and found that most would prefer faster feature releases over guaranteed 99.99 percent uptime. Luna: That's a fascinating insight. It really reframes the whole testing debate. Lucas: And that's something we hear a lot on this show — the idea that testing should be driven by user value, not by fear. If today's conversation gave you something usable, it's worth mentioning that listener support is what keeps this show ad-free and independent. You can find us at buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to help us keep digging into topics like this. And we really appreciate it. Lucas: Anyway, back to the CRM case. Once they had their error budget in place, they also started using chaos experiments to validate their fallback mechanisms. They'd intentionally kill a database instance during low traffic and see how the system behaved. Luna: And that caught bugs the regression suite had missed for months. Lucas: Exactly. One of their microservices had a retry storm problem — when one service failed, the others kept retrying and made the outage worse. That kind of failure mode is almost impossible to catch in a staging environment because you need real traffic patterns to trigger it. Luna: So by shifting from exhaustive pre-release testing to production chaos experiments, they actually found more impactful bugs. Lucas: Yes. And they did it with less total testing effort. The key insight is that your testing strategy should be shaped by your actual failure modes and your actual tolerance for failure. Five nines is a fine target for a nuclear reactor control system. But for your SaaS app? Probably overkill. Luna: Are there any tools or frameworks that help teams make this decision? Lucas: There are. Google's SRE book popularized the error budget concept. And there are tools like Gremlin for chaos engineering, and various monitoring platforms that help you track SLOs. But the most important step is just having the conversation with your stakeholders: 'What happens if we have a five-minute outage? What about an hour? What's the real cost?' Luna: And then design your testing accordingly. Lucas: Right. Start with the business impact, then figure out the testing. Not the other way around.