Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Prioritization Slashes Regression Time
Transcript
- Lucas: So you've got a regression suite that takes 45 minutes to run. You're shipping three times a day. That's over two hours of just waiting on tests — and that's if nothing fails. Luna: Right, and the temptation is to just run a subset. Pick the five most important test files and hope nothing breaks. Lucas: Exactly. But that's gambling. Test prioritization is a smarter middle ground — you run all the tests, but you run them in an order that maximizes the chance of catching failures early. Luna: So you can fail fast and fix fast, without sacrificing coverage. Lucas: Precisely. And I want to talk about a real example. A fintech company I've been following — let's call them PayBridge — had exactly this problem. Their regression suite had about 1,200 tests. Full run took 45 minutes. Developers were merging code without waiting for results because it was too slow. Luna: That's terrifying for a fintech. One bad payment calculation and you've got a regulatory headache. Lucas: Exactly. So they implemented a prioritization system. The core idea is simple: after every commit, you rank your tests by some notion of 'priority' and run the high-priority ones first. If a test fails, you stop the run — or at least flag it immediately — so the developer gets feedback in minutes, not 45. Luna: But how do you define 'priority'? Is it based on how often a test has failed historically? Or is it tied to which lines of code changed? Lucas: Both are valid. PayBridge used a hybrid. They built a risk score for each test using two signals: first, a coverage map — which source files each test touches. Second, historical failure frequency over the last 30 days. Then they scored tests by a weighted sum: if a test covers a file that changed in the latest commit, that test gets a boost. If that test also failed last week, it gets an even higher score. Luna: So it's essentially a greedy algorithm — sort tests by descending score, run them in that order, and hope the failures cluster near the top. Lucas: Exactly. Greedy, and it works surprisingly well. PayBridge reported that after implementing this, their median time to first failure dropped from 23 minutes to under 4 minutes. The full suite still runs eventually, but by minute 4, they've already caught 80 percent of the failures that would have occurred. Luna: Eighty percent? That's huge. But what about the other 20 percent? The failures that only show up in the later, lower-priority tests? Lucas: That's the trade-off. You still catch them — you just catch them later, after the high-priority tests pass. But here's the key: by the time the developer gets the first failure notification, they can start fixing. By the time they're done, the rest of the suite has finished, and they can see if anything else broke. No time wasted. Luna: So it's not about skipping tests. It's about reordering them so the feedback loop is tighter. Lucas: Right. And there's a whole literature on prioritization techniques. There's coverage-based: run tests that cover the most code first. There's risk-based: run tests that historically failed most. There's even machine learning approaches that predict failure probability based on code changes. But for most teams, the simple greedy approach with two signals is enough. Luna: What about tests that are flaky? If a test fails randomly, it might get a high failure score and always run first, wasting time. Lucas: Good point. PayBridge had to handle that. They added a flakiness detector: if a test fails more than once in a row on the same commit, it's flagged. If it fails intermittently without a clear code change, it gets downgraded in priority until the flakiness is fixed. They also set a minimum priority floor so flaky tests don't get skipped entirely — they just run later. Luna: So the prioritization system is dynamic. It adapts as test behavior changes. Lucas: Exactly. It's not a static ranking. Every commit recalculates the scores. And the system learns: if a test that never failed suddenly starts failing, its score jumps. The next commit it'll run early. Luna: I want to zoom out a bit. Test prioritization is one of those techniques that sounds academic, but the implementation is surprisingly accessible. What does the actual code look like? Lucas: It's honestly about 50 lines of Python. You parse your test framework's output, you parse your code coverage data — most CI systems already capture that — and you compute a score per test. Then you feed the sorted list back to your test runner. Luna: But you need to have code coverage enabled, and you need a way to map tests to source files. Not every team has that. Lucas: True. But most modern test frameworks can output coverage data. Pytest with pytest-cov. Jest has --coverage. Even if you don't have granular line-level coverage, you can use file-level: which test file imports which source module. It's a start. Luna: And for teams that don't have coverage at all, can they still do prioritization? Lucas: They can fall back on pure historical failure frequency. That alone gives you a decent improvement. A study at Google showed that just sorting by recent failure history cut time to first failure by 40 percent compared to random order. Not as good as coverage-based, but it's free. Luna: I'm thinking about the cost-benefit for a smaller team. Is it worth the engineering time to set this up? Lucas: If your regression suite takes more than 15 minutes and you ship daily, yes. The setup is maybe two days of work. The payoff is faster feedback and less context switching for developers. PayBridge estimated they saved about 10 developer-hours per week just from reduced waiting time. Luna: That's concrete. I'd love to know if anyone has tried prioritization in a monorepo with hundreds of microservices. The mapping gets complex. Lucas: There's research on that. One approach is to treat each microservice's test suite as a separate entity and prioritize within each. Another is to build a global dependency graph across services. But that's heavy. Most teams keep it simple: prioritize per service, and use a timeout to cut off long-running suites. Luna: Speaking of keeping it simple, that fintech example we mentioned — PayBridge — did they publish any metrics? Lucas: They presented at a QA conference last year. Their key numbers: median time to first failure went from 23 minutes to 3.8 minutes. Total regression run time stayed at 45 minutes, but developers got feedback in under 4 minutes. Also, the number of bugs that escaped to production dropped by 12 percent quarter-over-quarter. Luna: That's a strong result. And it's not about running tests faster — it's about running them smarter. Lucas: Exactly. And the best part is that prioritization works alongside other techniques like test impact analysis and flaky test detection. They complement each other. Luna: I want to ask about the edge case where a high-priority test fails, but the failure is actually a false alarm — maybe the test itself is wrong. Does the prioritization amplify that? Lucas: It can. If a test is poorly written and fails on every commit, it'll always be top priority and waste time. That's why you pair prioritization with test health monitoring. If a test fails more than, say, 10 percent of the time without a corresponding code change, it gets flagged for review. PayBridge had a weekly report of 'top failing tests' that the QA team triaged. Luna: So it creates a feedback loop: poor tests get exposed and fixed. Lucas: Right. Prioritization doesn't just speed up regression — it also incentivizes clean tests. Because if a test is flaky or brittle, it'll be at the top of every run and annoy everyone until it's fixed. Luna: That's a nice side effect. I'm also thinking about the developer experience. If I'm a dev and my commit triggers a regression run, and it fails in 4 minutes, I'm still context-switched. But at least I'm not waiting 45 minutes to know. Lucas: Exactly. And you can set up notifications: the CI sends a Slack message with the first failure and a link to the log. You can start investigating while the rest of the suite runs. By the time you've looked at the error, the full report is ready. Luna: I think this episode is a good example of something that sounds more complex than it is. If a team is struggling with slow regression, prioritization is probably the easiest win. Lucas: Absolutely. And it's free. No new infrastructure. Just a sorting algorithm and some historical data. Luna: Right. And honestly, if this episode gave you a concrete idea you can try, that's exactly the kind of thing that makes the show worth supporting. Lucas: Yeah. If today's tech conversation gave you something usable, you can find us at buy me a coffee dot com slash fexingo. A coffee's worth five bucks, and it keeps the show ad-free and focused on practical stuff like this. Luna: No pressure at all. Just if it's useful to you. Lucas: Anyway, back to prioritization strategies. I want to touch on one more advanced technique: using machine learning to predict failure probability based on the actual diff. Luna: Is that overkill for most teams? Lucas: Mostly, yes. But there are cases where it shines. For example, if you have a massive test suite and your codebase has complex dependencies that coverage tools miss. ML can learn patterns like 'changing this config file often breaks these three tests.' But it requires a lot of historical data and ongoing model retraining. Luna: So greedy with two signals is the pragmatic recommendation for 90 percent of teams. Lucas: Exactly. Start simple. Measure your time to first failure. Then add prioritization and measure again. If you see improvement, iterate. If not, add a third signal. Luna: And don't forget to handle flaky tests separately. Otherwise you're just amplifying noise. Lucas: Right. One more thing: prioritization works best when your tests are fast individually. If you have a test that takes 10 minutes to run, it'll be a bottleneck no matter where it is in the order. So combine prioritization with test splitting — parallelize across machines. Luna: Good point. So it's part of a larger strategy: fast tests, parallel execution, and smart ordering. Lucas: Exactly. And the payoff is real. PayBridge's developers reported higher satisfaction because they got feedback faster. They could stay in flow state longer. Luna: I think that's the ultimate metric: not just time saved, but developer happiness. Lucas: Yeah. And fewer bugs in production. That's the real win.