Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Software Testing Strategy Is Broken
Transcript
- Lucas: We keep talking about speed to market, but there is a quiet tax being levied on every software team right now, and it is called the maintenance burden of their own tests. Luna: You mean when the test suite takes longer to run than the actual deployment? Lucas: Exactly. And it is worse than just time. It is the erosion of trust. When developers stop believing the red lights in their continuous integration pipeline actually mean something, they start ignoring them. That is when production incidents happen. Luna: So the problem isn't that we aren't testing enough, but that we are testing too much of the wrong things? Lucas: It is a structural problem. We built a pyramid where the base was supposed to be lightweight unit tests, but over five years, that base has become a mountain of brittle integration tests that break whenever someone changes a CSS class or updates a library dependency. Luna: I have seen teams spend three days fixing a single flaky test instead of shipping a feature. That is not an anomaly anymore. Lucas: It is an industry-wide inefficiency. Let us look at how major players are trying to fix this. Microsoft, for instance, published data showing that nearly ten percent of their test runs were failing due to non-deterministic issues, not actual code bugs. Luna: Ten percent sounds high, but if you are running thousands of builds a day, that is a massive amount of engineer time wasted on chasing ghosts. Lucas: Right. And the cost is not just hours. It is the psychological toll. When a developer sees a failure, their first instinct should be relief that a bug was caught. But when it is a flake, their instinct becomes anxiety and then avoidance. Luna: They start marking tests as ignored or skipping entire modules just to get their build green again. Lucas: Precisely. Which leads to a false sense of security. The code looks good, the pipeline is green, but the underlying system is rotting because the safety net has holes. Luna: How do they actually quarantine these flaky tests without losing visibility? Lucas: The modern approach is to treat flaky tests as a separate class of technical debt. You don't delete them immediately. You isolate them from the main blocking path. You let the build pass even if those specific tests fail, but you alert the owner. Luna: So you decouple the signal from the noise. The critical path stays strict, but the noisy neighbors get moved to a staging area. Lucas: That is the key insight. Most teams try to force every test to be deterministic. But in distributed systems, some things are inherently non-deterministic. Network latency, race conditions, third-party API timeouts. You cannot unit test your way out of physics. Luna: This feels like a cultural shift as much as a technical one. It requires admitting that your perfect test suite is a lie. Lucas: It does. And it gets harder when leadership asks for coverage percentages. Coverage tells you what code is executed, not whether it works correctly under load or in production. Luna: I remember reading about Netflix’s approach here. They don’t just test for success; they actively break things to see if the system fails gracefully. Lucas: Chaos engineering. Yes. Instead of asking does this test pass, they ask does the system survive when I kill the database connection? It shifts the focus from verification to resilience. Luna: But chaos engineering is expensive to implement. Not every company can afford a dedicated reliability team. Lucas: True, but the principle applies broadly. You don’t need a full chaos grid. You just need to accept that some tests will fail, and design your pipeline to handle that gracefully rather than punishing the developer. Luna: If today's deep dive into the economics of testing gave you a clearer lens on your own CI pipeline, consider supporting Fexingo Business directly. Lucas: Listener support keeps these ad-free episodes coming, allowing us to dig into the real structural costs of technology without selling airtime. You can find us at buy me a coffee dot com slash fexingo. Luna: It helps us keep the research rigorous and the conversations focused on what actually moves the needle for engineering leaders. Lucas: Back to the metric that matters most. If flaky tests are a drag on velocity, what is the leading indicator that a team is falling behind? Luna: Is it the length of the pipeline, or is it the frequency of rollbacks? Lucas: Both, but specifically the ratio of failed tests that turn out to be false positives. If that number climbs above five percent, you are already in trouble. At ten percent, you are bleeding talent. Luna: Because smart engineers leave environments where their work is constantly invalidated by broken tools. Lucas: Exactly. So the solution isn't just better code. It is better hygiene. Regular pruning of the test suite. Treating obsolete tests with the same respect as obsolete documentation. Luna: Most companies archive old code but let old tests linger forever, bloating the suite indefinitely. Lucas: Yes, because no one wants to be the person who deleted the test that caught the critical bug three years ago. It is a fear-based maintenance model. Luna: But fear is a poor substitute for actual risk assessment. If the feature is gone, the test should go too. Lucas: Unless it is a regression guard for a legacy system. But even then, it needs to be stable. Stability is the currency of trust in software delivery. Luna: So we are moving from quantity of tests to quality of signals. Lucas: Ultimately, yes. A smaller suite that gives you instant, reliable feedback is worth more than a massive suite that makes you wait twenty minutes and wonder if the result means anything. Luna: It reminds me of the Pareto principle. Twenty percent of the tests catch eighty percent of the real issues. Lucas: Probably even less. The top five percent of tests cover the core user journey and the critical business logic. Everything else is often redundant or fragile. Luna: And yet, teams add more tests to prove they are doing their job, creating a vicious cycle of bloat. Lucas: It is a classic case of Goodhart’s Law. When a measure becomes a target, it ceases to be a good measure. Coverage percentage became the target, so everyone wrote more tests, regardless of value. Luna: Shifting the target to mean time to recovery might change behavior entirely. Lucas: It would. Because now the goal is not just finding bugs fast, but ensuring that when you ship, you can recover instantly if something goes wrong. That changes how you write tests. Luna: You write tests that verify the rollback mechanism, not just the happy path. Lucas: Exactly. You test the undo button. Which brings us back to the idea of confidence. Confidence is not generated by volume. It is generated by consistency. Luna: Consistency in results, consistency in feedback, and consistency in the team’s ability to trust their own tools. Lucas: If you stripped away all the flaky tests tomorrow, would your team feel safer or more exposed? Luna: More exposed initially, but then much more empowered to move forward without hesitation. Lucas: That is the trade-off we need to make. Short-term vulnerability for long-term velocity. It is a hard sell to management, but it is the only way out of the testing tax. Luna: Does anyone have a framework for deciding which tests to cut first? Lucas: Look at the failure rate. Look at the age of the test. Look at its relevance to current features. If a test hasn't failed in six months and covers deprecated code, it is dead weight. Luna: Simple criteria. Failure history, recency, and relevance. Hard to argue with that data. Lucas: Harder to execute, maybe, because of the political inertia, but the data doesn't lie. The tests are lying to us, and it is time we stopped listening. Luna: What happens when you finally clear the clutter? Does the team breathe easier? Lucas: They do. And they start building better things because they are no longer afraid of breaking what they already have.