Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Impact Analysis Saves Hours of CI Time
Transcript
- Lucas: You push a one-line fix — maybe a null check on a checkout button — and then you wait forty minutes for the full test suite to run. Every single time. There's a smarter way, and it's called test impact analysis. Luna: Impact analysis. So instead of running everything, you figure out exactly which tests could possibly be affected by that change. Lucas: Exactly. The core idea is simple: map the dependency graph between your source code and your tests. When a file changes, you only run the tests that depend — directly or transitively — on that file. Everything else stays green from the last run. Luna: Right. And that's not a hypothetical. Facebook's CI pipeline has been doing this for years. They reported cutting test execution time by over 70 percent for typical diffs. Lucas: Seventy percent. That's the difference between grabbing a coffee and actually getting feedback while your context is still hot. And it's not just the big tech companies — the technique is increasingly accessible. Luna: Yep. And speaking of context, you know what else keeps this show ad-free and focused on practical tech conversations? Listener support. If today's chat gave you something useful, you can throw a few bucks our way at buy me a coffee dot com slash fexingo. Lucas: Totally. It helps us keep digging into topics like this without chasing sponsors. So, back to impact analysis — let's talk about how the dependency graph actually works. Luna: Yeah, walk me through a concrete example. Lucas: Say you have an e-commerce app with a checkout module. That module depends on a payment service, a cart service, and a shipping calculator. You change one line in the shipping calculator. Luna: Under traditional CI, that change would trigger all tests — maybe ten thousand of them. Lucas: Right. But with impact analysis, the build system looks at the dependency graph and says: 'The shipping calculator is used by the checkout module, and tests for checkout exist. Those are the only tests that can be affected.' So you run maybe two hundred tests instead of ten thousand. Luna: And the rest stay green. That's a massive time save. But how do you build that dependency map in the first place? Lucas: There are a few approaches. The most robust is static analysis — you parse the source code and trace import statements, function calls, and type references. Google's Kythe is an open-source tool that does exactly that for languages like Java, C++, and Go. Luna: Kythe. I've heard of it. It builds a graph of code entities and their relationships. Lucas: Exactly. Then you map each test to the production code it touches. Some build systems, like Bazel, do this natively — they track dependencies at the rule level. When you run 'bazel test //checkout/...', it knows exactly which targets depend on which sources. Luna: So Bazel users get impact analysis almost for free. Lucas: Pretty much. But even if you're on Maven or Gradle, you can use tools like Intellij's dependency analyzer or custom scripts that parse your module graph. The key is having a clear, automated way to compute the transitive closure of a change. Luna: Transitive closure — that's the set of everything that could be reached from the changed file through dependency chains. So if your shipping calculator is imported by a utility module, which is imported by the checkout tests, those tests are included. Lucas: Yes. And that's where it gets tricky. If your dependency graph isn't accurate — say you have reflection, dynamic imports, or runtime class loading — you might miss tests that actually break. Luna: Right. That's the safety concern. If you skip a test that would have caught a regression, you ship a bug. Lucas: So teams usually run a full suite on the main branch once a day, or on release candidates, to catch anything the impact analysis might have missed. The selective runs are for developer feedback loops. Luna: That's a good compromise. So what about language ecosystems? I imagine Python's dynamic nature makes static analysis harder. Lucas: It does. For Python, you often rely on runtime tracing — like using coverage.py to record which lines are executed during a test run. Then you build a dynamic dependency map. But that requires running the test at least once to collect the data, so it's more of a profiling approach. Luna: So you can't do it purely from code for dynamic languages. Lucas: Not purely, no. But you can combine static import analysis with runtime traces. Pytest has plugins like pytest-testmon that do exactly that — they monitor which files were accessed during the last test run and only re-run tests for changed files. Luna: I've actually used testmon. It's pretty solid for small to medium projects. But I've heard it can get confused with concurrency or multiprocessing. Lucas: That's a known limitation. The more your code uses subprocesses, threads, or external services, the harder it is to trace dependencies accurately. Luna: So there's no one-size-fits-all tool. You have to pick based on your stack and risk tolerance. Lucas: Exactly. And the payoff can be huge. I've seen teams reduce CI feedback from fifty minutes to under five for typical pull requests. That changes how developers work — they can iterate faster, catch issues sooner, and stay in the flow. Luna: It also reduces infrastructure costs. Fewer test runs mean less CPU time on your CI servers. Lucas: Definitely. Especially if you're paying per minute for cloud runners. The savings can easily justify the setup effort. Luna: What about monorepos? Impact analysis seems especially valuable there, because full test suites can take hours. Lucas: Monorepos are actually where impact analysis shines brightest. With hundreds of services, you can't run everything on every commit. Bazel and similar tools were built for that scale. Luna: And the graph becomes huge, but the logic is the same. Lucas: Right. You compute the affected targets, run those tests, and merge quickly. The full suite runs nightly or on release tags. Luna: But there's a subtle issue: what about tests that are flaky? If you only run affected tests, flaky failures become more visible and annoying. Lucas: That's true. When you run fewer tests, each failure gets more attention. Teams often combine impact analysis with flaky test detection — quarantining flaky tests so they don't block the pipeline. Luna: Makes sense. So the overall strategy is: use impact analysis for speed, but keep a safety net of full runs and flaky test management. Lucas: Exactly. It's not an all-or-nothing switch. You phase it in. Start with a small service, measure the time savings, then expand. Luna: And the tooling is getting better. I've seen more CI platforms — like GitHub Actions and GitLab CI — offer native support for test impact analysis. Lucas: Yeah, it's becoming a standard feature. GitHub's 'test impact analysis' action is still in beta, but it shows where the industry is heading. Luna: So what's the one thing our listeners should take away? Lucas: Don't run tests you don't need to. Invest in understanding your dependency graph. Even a simple script that maps module dependencies can save your team hours every week. Luna: And if you're on a dynamic language, start with a tool like pytest-testmon and see how much faster your feedback loop gets. Lucas: Right. It's one of those optimizations that pays for itself almost immediately. Less waiting, more shipping.