Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Most Software Tests Are Just Expensive Noise
Transcript
- Lucas: If today's tech conversation gave you something usable for your morning stand-up, I'd love to hear it. But before we get there, a quick honest thing about how this show works. A small group of listeners chips in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making these episodes without ads or corporate sponsorship. It keeps us independent and lets us dig into the messy details rather than the press releases. Luna: Right, and that independence matters when we're talking about topics like testing, where vendors often have an incentive to sell you more tools instead of fixing your process. Thanks for keeping this ad-free. Lucas: Let's jump into the core issue. We talk a lot about software reliability, but most engineering leaders I speak with are drowning in a different problem: they don't know if their tests are actually telling them anything useful. It's not just about finding bugs anymore; it's about the signal to noise ratio in your continuous integration pipeline. If your build fails three times a week, and two of those failures turn out to be environmental glitches or race conditions, you aren't building resilience. You're building anxiety. Luna: Exactly. And that anxiety leads to people ignoring red lights because they've been burned too many times by false alarms. How do you quantify the cost of that noise? Lucas: The numbers are staggering when you trace them back to engineer hours. At a mid-sized fintech company we looked at recently, they had four thousand automated tests running on every commit. On average, twenty percent of those runs failed, but only eight percent were actual code defects. The rest were timing issues, database state problems, or transient network errors. That means engineers were spending roughly forty hours a month investigating ghosts. Forty hours. That's a full week of one person's salary, gone, looking for bugs that didn't exist. Luna: That is insane. So they were essentially paying developers to debug their own testing infrastructure instead of shipping features. Lucas: Precisely. And here's the counterintuitive part. When they finally audited the suite, they found that sixty percent of those failing tests provided zero incremental value over the remaining forty percent. They were redundant assertions checking the same database row or validating the same API response twice. The team decided to cut the suite size by half. Not to reduce coverage, but to increase precision. Their deployment frequency went up, and their defect rate in production actually dropped because the engineers started trusting the green checkmarks again. Luna: It feels like we've confused volume with quality for the last decade. More tests equals better software, right? Until it doesn't. Lucas: That's the trap. The metric became the goal. Teams celebrated having ten thousand tests, even if those tests ran for forty minutes and broke every Tuesday. The shift we're seeing now is toward what I call 'failure signal clarity.' It's not about whether the system works; it's about how quickly you know when it breaks, and whether that notification is actionable. If a test fails, can a junior engineer fix it in under ten minutes? If not, the test itself is broken. Luna: So the health of the test suite becomes as important as the health of the application. That changes how you hire QA, doesn't it? Lucas: It completely reframes the role. We used to think QA was about writing lots of scenarios. Now, top-tier engineering orgs are treating test maintenance as a first-class engineering discipline. They have dedicated rotations for 'test hygiene,' similar to how you might have a security audit or a dependency update cycle. If a test flakes more than once a month, it gets quarantined immediately. It doesn't block the merge, but it flags a ticket for review. This prevents the rot from spreading to the main branch. Luna: Quarantine sounds smart. It stops the bleeding without stopping the flow. But how do you decide which tests to keep when you're cutting the fat? Lucas: You look at the blast radius. The best teams map their tests directly to business outcomes, not just code paths. Does this test verify that money moves correctly? Or does it just verify that the UI button exists? The former is critical. The latter is often redundant if you have proper component isolation. One logistics platform we tracked replaced their sprawling end to end UI tests with lightweight contract tests between services. They lost some visual verification, sure, but they gained speed and accuracy. The UI could change tomorrow, but the data payload structure remained stable. Testing the contract was cheaper and safer. Luna: I love that distinction. Contract testing feels like the missing link between unit tests and full system tests. It's where the real integration risks live. Lucas: It is. And it forces teams to define interfaces explicitly. Too many organizations let APIs drift because no one is formally checking the handshake. When you implement contract testing, you force a conversation between the service owner and the consumer. It exposes assumptions before they become production fires. We saw a SaaS provider nearly lose a major enterprise client because their internal billing service changed a field name, breaking the client's integration. A simple contract test would have caught that weeks earlier. Luna: That's a nightmare scenario. And it highlights why manual reviews of API specs aren't enough. Code enforces truth. Lucas: Code enforces truth. Specs gather dust. But here's where it gets tricky for managers. Moving to contract testing requires upfront investment. You have to design the contracts, version them, and maintain the tooling. Some leaders resist because they don't see immediate bug reduction. They want faster feature delivery yesterday. But the ROI comes in reduced firefighting. You trade a predictable maintenance cost for unpredictable emergency patches. In September of twenty twenty-six, with economic uncertainty still pressing on tech budgets, predictability is worth more than raw speed. Luna: Absolutely. When margins are tight, you can't afford surprise downtime. Predictable stability is the new growth engine. Lucas: Right. And let's talk about the human element. When you strip away the flaky noise, developer morale improves significantly. There's a psychological toll to constantly chasing phantom bugs. Engineers feel incompetent when the system lies to them. By cleaning up the test suite, you're restoring trust in the feedback loop. It's not just an engineering optimization; it's a culture play. High-performing teams measure their happiness alongside their latency. Luna: That's a powerful point. If your tools make you feel dumb, you'll avoid using them. Or worse, you'll hack around them. Lucas: Exactly. Hacking around the tests is dangerous. It leads to shadow deployments that nobody monitors. We need to normalize the idea that maintaining tests is hard work. It deserves respect. It deserves resources. And it deserves the same rigorous debugging as the product code itself. If you wouldn't ship buggy production code, why ship buggy test code? Luna: Because we assume tests are disposable. We treat them as overhead rather than assets. That mindset needs to flip. Lucas: It does. And the way to flip it is to start small. Pick one service. Clean its tests. Make them fast, deterministic, and meaningful. Prove the concept. Then expand. Don't try to boil the ocean. Most teams fail because they try to rewrite their entire suite overnight. They get overwhelmed, the project stalls, and everyone goes back to clicking 'run all' and hoping for the best. Luna: Practical advice. Start with one domain. Make it reliable. Let that success spread organically to other teams. Lucas: Yes. And track the metrics that matter. Not 'number of tests written,' but 'time to detect failure' and 'rate of false positives.' If those go down, you're winning. If they stay flat, you're spinning wheels. Simple, measurable, actionable. Luna: Clear metrics beat vague goals every time. It gives leadership a reason to invest in the cleanup without feeling like it's a waste. Lucas: It does. And ultimately, this is about respect for the craft. Good software isn't built by accident. It's built by systems that tell the truth, even when it hurts. If your tests lie, you're blind. If they tell the truth, you're invincible. Luna: Invincible is a high bar, but closer than most teams realize. Thanks for the reality check, Lucas. Lucas: Anytime. Let's see if we can make our own pipelines a little less noisy next week.