Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Prioritization Saves Hours of CI Time
Transcript
- Lucas: You know, I was reading Google's 2025 test automation report, and one number jumped out at me — teams that implement test prioritization reduce their CI pipeline time by an average of 40 percent. That's not some theoretical optimization, that's a real-world saving. Luna: Forty percent is huge. But when people hear 'test prioritization,' I think they often imagine just running the most important tests first. Is there more to it than that? Lucas: Yeah, there's a lot more. The core idea is you have a suite of tests — maybe thousands, maybe tens of thousands — and instead of running them all every time you push code, you order them so the ones most likely to fail run first. If a test fails, you stop the pipeline early, get feedback faster. Luna: So it's not about skipping tests, it's about reordering them to fail fast. Lucas: Exactly. But the real sophistication comes in how you define 'most likely to fail.' You can use historical data — which tests have failed most often in the past week, which tests are linked to the files you just changed. Or you can use risk-based analysis: tests covering critical user workflows get higher priority. Some teams even use machine learning models that learn failure patterns over time. Luna: I've seen Spotify's engineering blog talk about this. They built a system called 'Test Selection at Scale' that uses a random forest model to predict which tests are relevant to a given code change. Lucas: Right, that's a great example. Spotify reported that after implementing that model, they cut their average CI time from 45 minutes to under 10 minutes for most changes. And their production incident rate didn't go up. That's the key metric — you're not sacrificing quality for speed. Luna: But there's a concern I hear from QA engineers: if you prioritize, you might miss a bug that only shows up in a test that didn't run. How do you guard against that? Lucas: That's the right question. No prioritization scheme is perfect. The trick is to have a safety net. Most teams run a full regression suite overnight or on a less frequent schedule — the prioritization is for the fast feedback loop during the day. Also, you can use a technique called 'time-bucketed prioritization': run the top-priority tests within a time budget, say 10 minutes, then if time permits, run lower-priority tests. Luna: So the full suite still runs, just not on every commit. That makes sense. Lucas: Yeah, exactly. Another approach is to maintain a separate smoke test suite — a small set of critical tests that always run first. Then you layer in prioritized tests after that. Google's research shows that with a well-tuned prioritization, you can catch 90 percent of failures within the first 20 percent of test execution time. Luna: That's a powerful stat. It means developers get feedback in minutes instead of hours. Lucas: And that changes how teams work. When CI is fast, developers are more likely to run tests before pushing, less likely to context-switch while waiting. It improves the whole development cycle. Luna: If today's conversation gave you something usable, the way these shows stay ad-free is listener support. You can find us at buy me a coffee dot com slash fexingo. It's a simple way to keep the episodes coming. Lucas: Yeah, every little bit helps. And we mean it — this show exists because of listeners who find value in it and want to keep it independent. Luna: Alright, back to prioritization. One technique I've seen work well is using code coverage data. If you know which tests cover the lines of code you changed, you can prioritize those. Lucas: Yeah, coverage-based prioritization is a common starting point. But it has a limitation: coverage doesn't tell you about the test's ability to detect faults. A test that covers a lot of code might be a weak test. That's why many teams combine coverage with historical failure data. Luna: So you're weighting tests not just by how much code they touch, but by how often they've actually caught bugs. Lucas: Exactly. And you can get even more granular. Some systems track the 'last failure time' of each test and prioritize tests that haven't run recently. Or they prioritize tests that cover code that's been frequently modified. The more signals you combine, the better the prioritization. Luna: What about tests that are slow but important? If a slow test has a high failure probability, you still want it early. Lucas: Yeah, that's a trade-off. Some prioritization models incorporate execution time as a factor — they rank tests by a ratio of 'failure probability' divided by 'execution time.' That way, fast tests that are likely to fail get bumped up. But if a slow test is truly critical, you might run it in a separate parallel job. Luna: Parallelization is another lever. You can combine prioritization with running tests in parallel across multiple machines. Lucas: Right, they're complementary. Prioritization reduces the number of tests you need to run to get feedback, and parallelization reduces the wall-clock time of running those tests. Together, they can get CI down to under five minutes for many teams. Luna: I want to circle back to the machine learning approach. How does a team even start building a model for test prioritization? Lucas: You don't have to start from scratch. There are open-source tools like 'TestRank' and 'CoverageBasedTestPrioritization' that provide frameworks. The basic pipeline is: collect data on test outcomes over time, features like code change size, files modified, test coverage, then train a model to predict failure probability. It's a classification problem. Luna: But you need a decent amount of historical data. For a new project or a small team, that might not be feasible. Lucas: True. For smaller teams, simple heuristics work fine. For example, prioritize tests based on the last time they failed, or prioritize tests that cover recently changed files. Even a basic rule-based prioritization can give you big wins. Google's report shows that even random prioritization — just shuffling tests — can catch failures faster than running in a fixed order. Luna: That's surprising. Why would random work better than a fixed order? Lucas: Because fixed orders tend to accumulate biases. If your test suite runs alphabetically, tests starting with 'A' always run first. But failures might cluster in tests later in the alphabet. Randomizing spreads the risk. Of course, smart prioritization is still better, but it shows how much room for improvement there is in most CI pipelines. Luna: So the low-hanging fruit is just not running tests in a static order. Lucas: Exactly. If your team is still running tests in the order they were added to the suite, you're probably wasting time. Even adding a simple risk-based sort — like running tests that cover user login first, because login changes are high-risk — can make a big difference. Luna: Let's talk about measurement. How do you know your prioritization is working? Lucas: The standard metric is 'average percentage of faults detected' or APFD. It measures how quickly your test suite catches failures. You plot the cumulative percentage of faults detected against the percentage of test suite executed. A higher APFD means you're catching bugs earlier. Teams should track that over time as they tune their prioritization. Luna: Are there any pitfalls? I've heard of teams where prioritization actually slowed them down because they spent too much time ranking tests. Lucas: Yeah, that's a real risk. If your prioritization algorithm takes longer to compute than the time it saves, you've lost the benefit. That's why most teams don't re-rank every single commit. They do it periodically — say, once a day — or they use incremental algorithms that only update the ranking for tests affected by recent changes. Luna: So it's about picking the right granularity. You don't want to over-optimize. Lucas: Right. And you need to monitor that your prioritization isn't causing tests to be consistently skipped. A safety net is to run a full suite at least once before merging a pull request, even if prioritization runs on every commit. Luna: That sounds like a good compromise: fast feedback on each commit, but a thorough check before merge. Lucas: Exactly. And that's where many mature teams land. They use prioritization for the inner loop — the push feedback — and a full regression for the outer loop — the pre-merge gate. It balances speed and safety. Luna: I think this is one of those areas where even a small investment pays off quickly. If you have a 30-minute CI pipeline, cutting it to 18 minutes with basic prioritization saves 12 minutes per push. Over a day with 20 pushes, that's four hours of collective developer time. Lucas: Yeah, and that's not just time — it's flow. Developers aren't context-switching as much. They get results faster, they iterate faster. The compounding effect on productivity is real. Luna: So if someone listening wants to start tomorrow, what's the first step? Lucas: Look at your current test suite. Check if you have a fixed order. If you do, add a simple shuffle — randomize the test order. Measure your CI time before and after. Then, if you want to get smarter, start collecting test failure data. Which tests fail most often? Sort by that. You'll see immediate improvement. Luna: And from there, you can incrementally add more signals. Lucas: Exactly. Start simple, measure, iterate. That's the fastest path to a faster CI pipeline.