Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Automation Needs a Feedback Loop
Transcript
- Lucas: Let me start with a number that stopped me this week: a mid-size fintech company I talked to found that seventy percent of their automated test failures were either ignored or took more than three days to resolve. Luna: That's wild. So they're running tests, seeing red, but nobody acts on it? Lucas: Exactly. And the company's head of QA told me the tests themselves were well-written, good coverage, nice framework. The problem wasn't the tests. It was the feedback loop — or rather, the lack of one. Lucas: Right. And that uncertainty means the failure sits there. The developer who might have introduced the bug doesn't know about it. The QA engineer who wrote the test is busy with the next sprint. So the red flag becomes background noise. Luna: What did that fintech company do differently? Lucas: They built a structured feedback mechanism. Every failed test now automatically generates a bug report with the logs, a screenshot, the environment details, and even a git blame — it shows which developer last touched the code that the test exercises. Luna: So the test failure itself creates a ticket with all the context, and it's assigned to the person most likely to be responsible? Lucas: Exactly. And that changed the whole culture. Suddenly failures weren't abstract — they had an owner and a clear action. The result: their bug to fix time dropped by seventy percent. That's from three days to under a day on average. Luna: Seventy percent reduction just from better routing and context. That's huge. Lucas: It really is. And it's not just about speed. It's about the psychology of test failures. When a test fails and nothing happens, engineers start to distrust the test suite. They ignore it. But when a failure immediately triggers a structured, assignable task, the test becomes a teaching tool. Luna: Right. It's not a punishment — it's a signal that the system wants to get better. Lucas: Exactly. And that's the core idea I want to explore today: test automation without a feedback loop is just expensive noise. The loop is what turns test results into better code. Luna: If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. Lucas: Yeah, genuinely. If you got something from this conversation, that's the best way to keep it going. No pressure. Luna: So back to that feedback loop. You mentioned git blame — but isn't that potentially finger-pointing? How do you avoid a blame culture? Lucas: Great question. The company we talked to pairs the assignment with a blameless post-mortem policy. The bug report is seen as a system signal, not a personal failure. The message is: 'The test caught something. Let's understand together.' Luna: So it's about learning, not blaming. That makes sense. Lucas: And it works because the feedback is contextual. The developer gets the logs, the stack trace, the environment — they can often fix the bug within minutes instead of spending hours reproducing it. Luna: That's a huge productivity gain. But what about false positives? If the test is flaky, you're still creating a ticket that wastes someone's time. Lucas: Right. That's why the feedback loop also needs a way to flag flaky tests. In their system, if a test fails, then passes on a re-run within 24 hours without any code change, it gets marked as potentially flaky. That triggers a separate process to investigate and fix the test itself. Luna: So the feedback loop applies to the test suite too. It's meta-feedback. Lucas: Exactly. The health of the test suite becomes visible. You can track metrics like 'mean time to acknowledge a failure' or 'percentage of failures that lead to a code change within a day'. Those are the numbers that matter, not just pass rate. Luna: I love that. Let's talk about how to implement this in a smaller team without a dedicated DevOps person. Lucas: Sure. The simplest version: use your CI tool's webhooks to trigger a script that creates an issue in your tracker — Jira, Linear, GitHub Issues — with a template. Fill in the test name, the error message, a link to the CI run, and the git blame output. Then assign it to the person from the blame. Luna: That's like twenty lines of code. Maybe a GitHub Action or a GitLab CI job. Lucas: Exactly. You don't need expensive tooling. The key is the template — make it useful. Include the environment variables, the database state if relevant, the exact command to reproduce locally. The less friction for the developer, the faster they fix it. Luna: And what about prioritization? Do all failures get the same urgency? Lucas: No. Good point. They categorize by severity based on the test type. A unit test failure affecting a core module gets high priority. A visual regression test on a rarely-used page gets low. The feedback loop should tag the ticket with the test's criticality. Luna: So it's not just automation — it's intelligent routing. Lucas: Exactly. And that's where the loop really pays off. You're not just telling someone 'something failed'. You're saying 'here's exactly what failed, why, who might know, and how urgent it is'. Luna: That changes the conversation from 'the tests are red' to 'here's a precise problem to solve'. Lucas: Yes. And it also reduces the anxiety around deploying. When you know that any real regression will surface as an actionable item within minutes, you can release with more confidence. Luna: Let's talk about the human side. What if the developer assigned feels defensive? Lucas: That's where culture matters. The company we studied made it a point to celebrate fast fixes. They have a Slack channel called 'test-heroes' where they post when someone resolves a test failure quickly. It gamifies the positive behavior. Luna: So it becomes a source of pride, not shame. That's smart. Lucas: And it works. They saw a forty percent reduction in the time between test failure and first commit to fix it within two months of launching the channel. Luna: Forty percent just from social reinforcement. That's remarkable. Lucas: It shows that feedback loops aren't just technical — they're social. The best automation in the world fails if the team doesn't trust it or feel motivated to act on it. Luna: So what's the one takeaway for our listeners today? Lucas: If your test automation isn't consistently leading to code changes within hours, you don't have a test problem — you have a feedback loop problem. Fix the loop, and the tests will start earning their keep.