Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Impact Analysis Saves Hours of Regression Time
Transcript
- Lucas: Let me ask you something. Your team has a test suite that takes forty-five minutes to run. You push a one-line change — fixing a typo in a button label. Do you really need to run every single test in that suite? Luna: Ugh, I've been there. And the honest answer is no — but most CI pipelines run them all anyway, because nobody wants to be the one who accidentally skipped a critical test. Lucas: Right. And that's where test impact analysis — TIA — comes in. The core idea is simple: figure out exactly which parts of your codebase changed, map those changes to the tests that actually exercise that code, and only run those tests. Luna: So instead of running four thousand tests, you run maybe two hundred. That sounds like a huge time saver. But how do you actually build that map? Lucas: You need a dependency graph. Every source file, every function, every test — and the relationships between them. When you change a file, you trace which tests depend on it, directly or indirectly. Then you run only those. Google's internal CI system does this at massive scale. Luna: I've heard Google claims they cut test execution time by something like forty percent. But that's Google — they have custom infrastructure. What about a normal team? Lucas: The forty percent figure comes from a 2017 paper where Google engineers described their approach. They used a combination of static analysis and build system metadata. But you don't need Google-scale to benefit. Tools like Bazel, Nx, and even some modern CI platforms have built-in TIA capabilities. Luna: Let's break down the two main approaches. You mentioned static versus dynamic analysis. What's the difference? Lucas: Static analysis looks at the source code and the build graph — it says, 'this file is imported by these files, which are tested by these tests.' It's fast, conservative, and might include tests that don't actually exercise the changed code. Dynamic analysis actually instruments the code to record which tests hit which lines during execution. That's more accurate, but it requires a profiling run and maintaining that data. Luna: So static is easier to set up but might over-select tests, and dynamic is more precise but has overhead. Which one do most teams start with? Lucas: Most start with static analysis because it integrates naturally with build tools. Nx, for example, uses a project graph to determine affected projects. When you change a library, it runs only the tests that depend on that library. It's not perfect — if you change a utility function used by fifty components, it'll run all fifty component tests. But that's still better than running the entire suite. Luna: I talked to a fintech startup last month. Their CI was taking thirty-four minutes — painful for a team of twelve. They implemented Nx's affected:test command and got it down to twelve minutes. That's a twenty-two-minute saving per push. Lucas: That's a perfect example. And the savings compound. If you push ten times a day, that's over three and a half hours saved per developer per day. That's not just faster feedback — it's less context switching, less frustration. Luna: But there's a catch. TIA only works well if your codebase is modular and your dependencies are explicit. If you have a monolithic file that's imported everywhere — like a shared utils.js — changing it will still trigger most of your tests. Lucas: Exactly. The technique exposes architectural debt. If a change to one file cascades to hundreds of tests, that's a signal to refactor. TIA gives you both efficiency and architectural insight. Luna: What about dynamic analysis in practice? I've only seen it used at companies with dedicated performance teams. Lucas: It's more common than you think. Tools like TestImpact for.NET or Wallaby for JavaScript do dynamic analysis. They run your tests with code coverage enabled, persist the mapping, and then use it for future selections. The overhead is a one-time cost. For a suite that runs frequently, that cost pays for itself quickly. Luna: But do you trust it? If dynamic analysis says 'skip test X' because it didn't cover the changed code, but test X actually would have caught a regression — that's scary. Lucas: That's the biggest concern, and it's valid. Dynamic analysis can miss tests if the profiling run didn't exercise all code paths — for example, if your coverage data comes from a single test run that hit only the happy path. If you then change error-handling code, dynamic analysis might not select the tests that cover that branch. Luna: So the solution is to combine both approaches, or to use static analysis as a safety net? Lucas: Right. Many teams use a hybrid: static analysis as the primary selector for speed, and then run a subset of high-risk tests — like integration or end to end — on every change regardless. The key is to treat TIA as a optimization, not a guarantee. You never skip tests that are tagged as 'critical' or 'regression-sensitive.' Luna: That makes sense. So the goal isn't to run zero tests — it's to run the right tests. And TIA helps you get closer to that ideal. Lucas: Exactly. And the industry is moving in this direction. GitHub Actions now has an experimental 'test impact analysis' feature. GitLab has similar capabilities. Even if your CI doesn't support it natively, you can implement it with a simple script that compares the file list from 'git diff' against a mapping file. Luna: I think one reason teams don't adopt it is the upfront investment. You have to build the dependency graph, and that takes time. Lucas: It does. But the return on that investment is measured in developer hours saved. If your CI takes thirty minutes and you run it twenty times a day, that's ten hours of waiting per day. If TIA cuts that to ten minutes, you save almost seven hours a day. Across a team of ten, that's seventy hours — almost two developer weeks per day. Luna: When you put it that way, it's hard to ignore. I'm actually going to look into Nx for our internal tools. Lucas: Let me know how it goes. One thing to watch out for: if your project uses dynamic imports or reflection, static analysis might miss some dependencies. In those cases, supplement with a manual test configuration. Luna: Good tip. So let's say someone wants to start. What's the simplest first step? Lucas: Run a coverage report on your current test suite. Identify the files that are touched by the most tests. Then, the next time you change one of those files, manually compare the list of tests that coverage says are affected versus the test suite that actually runs. That gives you a baseline for how much time you're wasting. Luna: That's a low-risk experiment. You're not changing anything, just gathering data. Lucas: Exactly. And once you see that you're running, say, two thousand tests when only three hundred could possibly be affected, it becomes easier to justify the investment in TIA. Luna: Alright, I'm convinced. One last thing: does TIA help with flaky tests? Lucas: Indirectly, yes. Because you're running fewer tests, the noise from flaky tests decreases. Plus, if a test is flaky, TIA might not select it unless it's truly affected — so you get fewer false failures. Luna: That's a nice side benefit. So TIA is about efficiency, but also about making the test results more trustworthy. Lucas: Exactly. And on that note, I want to say something about how we keep this show going. We deliberately don't run ads — no sponsors, no commercial breaks. That's a choice we made because we think the conversation should be about the content, not about selling you something. If that approach resonates with you, and you want to support it, the link is buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to keep the show exactly as it is. We really appreciate those who do. Lucas: So back to TIA. I think the big takeaway is this: the goal of testing isn't to run every test every time. It's to give developers fast, accurate feedback. Test impact analysis is one of the most practical ways to achieve that, and it forces you to think about your code's structure along the way. Luna: And it scales. Whether you're a startup or a large enterprise, the same principles apply. Lucas: Right. Next time you're waiting for a CI pipeline, ask yourself: how many of these tests are really necessary? The answer might change how you build.