Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Code Still Ships with Test Automation
Transcript
- Lucas: You know that sinking feeling when a bug that should have been caught by your test suite makes it all the way to production? That happened to a fintech startup last year — and their test automation was technically 'green' when they shipped. Luna: Wait — green lights across the board, and the bug still hit? How? Lucas: Exactly. Their test suite was automated, but it had accumulated something we call 'test debt.' It's like technical debt, but for your tests. The team had added new features over six months without updating the integration tests — so those tests were either skipped or passing because they weren't actually exercising the new code paths. Luna: So they had a false sense of security because the automation was still running and showing green? Lucas: Right. The CI pipeline reported 100 percent test pass rate. But when I looked at the actual test coverage for the new modules, it was under 20 percent. The test suite hadn't kept pace with the codebase. Luna: That sounds like a classic case of automation theater. Teams automate because they're told to, but then they don't maintain the tests. Lucas: Exactly. And that's the core problem we want to unpack today: why test automation alone doesn't guarantee quality. In fact, if you automate poorly, you can make things worse. Luna: Let's start with that fintech example. What was the actual bug that hit production? Lucas: It was a payment routing algorithm. The team added a new tier of transaction fees but forgot to update the fee calculation logic in one edge case. The old integration test only checked the default fee tier, so it passed. The new tier's logic was never tested. They lost about $50,000 in misrouted payments before they caught it. Luna: Ouch. And they had a full automation suite — unit tests, integration tests, even some end to end tests. But the integration tests were essentially frozen since the last big refactor. Lucas: Right. The team was under pressure to ship features fast, and maintaining tests felt like overhead. So they deferred it. That's test debt. It's invisible until you need it. Luna: So what's the fix? How do you keep test debt from piling up? Lucas: One thing we've seen work is treating test maintenance as a first-class task in your sprint. Same as code refactoring. You don't just add tests for new features — you also review existing tests for relevance and reliability. Luna: I've heard that some teams use a 'test health' metric. What does that look like in practice? Lucas: Good question. A simple one is flakiness rate — the percentage of test runs that fail non-deterministically. Once that creeps above 5 percent, developers start distrusting the suite. They start ignoring failures, or worse, they skip running tests locally. I've seen teams where the flakiness rate hit 15 percent, and nobody bothered to investigate red builds anymore. Luna: So the automation becomes noise. And then the real bugs slip through. Lucas: Exactly. That's the paradox. The more you automate without maintaining, the less reliable the automation becomes. And then you're back to manual testing, but now you think you're covered. Luna: I've also seen teams fall into the trap of automating everything. They write end to end tests for trivial UI flows that change every sprint, and those tests break constantly. Lucas: Right. That's the other side of test debt — over-automation at the wrong level. The industry has been pushing 'shift left' for years: test earlier, test more. But shifting left without discipline just creates a bigger maintenance burden. Luna: So what's the right approach? How does a team decide what to automate? Lucas: The classic test pyramid still holds. Lots of fast, reliable unit tests; fewer integration tests; and only a handful of end to end tests for critical user journeys. But the key is that each level needs to be maintained proportionally. Luna: And how do you enforce that? I mean, it sounds great in theory, but when deadlines hit, tests are the first thing to slide. Lucas: You need to make the cost of not maintaining tests visible. One way is to track 'test coverage over time' for new code. Not just line coverage, but branch coverage and mutation coverage if you can. And then put that number on a dashboard that the whole team sees. Luna: So if coverage drops below a threshold, the build fails? Lucas: Some teams do that. But I've seen it backfire — developers game the system by writing trivial tests that boost coverage without catching bugs. A better approach is to monitor the 'test failure trend' over the last two weeks. If you see a spike in failures or skipped tests, that's a red flag. Luna: That makes sense. So it's more about behavioral signals than a hard number. Lucas: Exactly. And one concrete thing any team can do right now: pull the last 100 test runs from your CI system. Count how many failed due to flakiness versus actual code issues. If flaky failures are above 5 percent, you have a problem. Luna: That's a great takeaway. I'm going to do that for our team this afternoon. Lucas: Do it. And when you find those flaky tests, either fix them or delete them. A passing test that's never run because it's flaky is worse than no test at all — because it gives you false confidence. Luna: But what about the cultural side? I feel like no amount of dashboards fixes a team that doesn't value testing. Lucas: That's the hard part. You need leadership buy-in. One thing that works is to calculate the cost of test debt in terms of production incidents. If you can show that a $50,000 outage was caused by a skipped test, it gets attention. Luna: So you make the invisible visible. Use money as the language. Lucas: Right. And you can frame it positively too: teams that invest in test maintenance actually ship faster in the long run because they spend less time debugging. There's data on that — Microsoft's DevOps research shows that high-performing teams spend 20 percent less time on unplanned work. Luna: That's a compelling argument. So test automation isn't a set-it-and-forget-it thing. It's a living system that needs care. Lucas: Exactly. And we'll dig deeper into specific strategies in future episodes — like how to prioritize which tests to fix first, and tools that can automatically detect flaky tests. Luna: Looking forward to it. For now, I'm going to go audit our test suite health. Lucas: Good luck. And remember: a green build is only as good as the tests behind it.