Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Needs a Static Analysis Pass
Transcript
- Lucas: We talk a lot on this show about what tests catch — unit tests, integration tests, property-based tests — but there's a whole category of bugs that tests are structurally bad at finding, and static analysis is designed to catch them. Luna: You mean the kind of bug where the code compiles, passes all your tests, and still does the wrong thing in production? Lucas: Exactly. I want to start with a concrete example. A few years ago a payment-processing company I know — midsize, handling about fifty million dollars in monthly transactions — had a bug in their refund logic. Unit tests passed. Integration tests passed. end to end tests passed. But under a specific race condition, the system would double-refund a customer. Luna: Oof. That's the kind of bug that costs both money and trust. Lucas: Right. The root cause was a non-atomic check and act pattern. The code checked whether a refund had already been issued, and then applied the refund — but between those two lines, another thread could slip in. A unit test would never catch that because unit tests don't run concurrent scenarios by default. Integration tests might, but only if you modeled the exact timing. Luna: And that's where static analysis comes in — because it doesn't execute the code, it just reads it. Lucas: Right. A static analyzer — like the ones built into modern IDEs or run in CI — can detect that pattern: a read followed by a write without a lock or a transaction boundary. In this case, the team had a static analysis step in their pipeline, but it was configured to only run on the main branch after merge. The race condition had been introduced in a pull request, merged, and only flagged after the fact. Luna: So the tool was there, but the timing was wrong. That feels like a common story. Lucas: It is. Most teams treat static analysis as a CI gate — run it before deployment — but by then you've already committed the code, and the cost of fixing it is higher. A study from Google a few years ago found that developers who received static analysis warnings during code review — before merging — fixed issues three times faster than those who only saw them in CI. Luna: Three times faster. That's not marginal. Lucas: No. And Microsoft published similar findings. They looked at their internal use of static analysis across thousands of developers and found that the most effective deployments were the ones that surfaced warnings in the pull request workflow itself — as comments on the diff — not in a separate report you have to go look at. Luna: That makes sense because context-switching is expensive. If I'm already in the PR reviewing the diff, I can fix a warning in thirty seconds. If I get an email later, I have to re-read the code, remember what I was doing, and then fix it. Lucas: Exactly. But there's a catch — and it's the reason a lot of teams eventually disable static analysis or ignore its output. False positives. Luna: Ah, the boy who cried wolf problem. Lucas: Right. A typical static analysis tool has hundreds of rules. If you turn them all on, you'll get warnings about variable naming conventions, unused parameters, or stylistic preferences — and those drown out the actually dangerous patterns. The payment company I mentioned had that exact problem. Their initial configuration generated so many low-severity warnings that developers started routinely dismissing them. The race condition warning was in there, but it was buried. Luna: So what's the fix? Do you just turn off the noisy rules? Lucas: That's the first step. But you don't want to just disable everything — you want to categorize. There's a well-known taxonomy from the software engineering literature: bugs that cause crashes or data corruption, bugs that cause incorrect behavior but don't crash, and then stylistic or maintainability issues. Most teams should start by enabling only the crash and corruption category, which is usually a small subset of rules. Luna: And then gradually expand as the team gets comfortable. Lucas: Yes. The key is to treat static analysis as a configurable design tool, not a fixed scanner. I've seen teams reduce their false-positive rate from sixty percent to under ten percent just by tuning rule sets to their codebase and suppressing warnings that don't apply to their architecture. Luna: Let's talk about what static analysis catches that tests don't. You mentioned race conditions — what else? Lucas: Null pointer dereferences in languages where null is a valid reference. Buffer overflows in systems code. SQL injection and cross-site scripting in web applications. Resource leaks — like opening a file handle or a database connection and never closing it. There's a class of bug called 'initialization order' where you use a variable before it's guaranteed to be initialized. All of these are very hard to test exhaustively because the state space is enormous. Luna: And static analysis can find them in milliseconds. Lucas: Milliseconds, yes, because it's not running the code — it's building an abstract model of the code's control flow and data flow and checking it against a set of patterns. Now, it's not perfect. Static analysis can't tell you whether your business logic is correct. It can't tell you whether the refund amount is computed correctly. But it can tell you that you're writing to a variable that another thread might be reading without synchronization. Luna: So it's complementary to testing, not a replacement. Lucas: Exactly. And there's a particularly powerful combination: coupling static analysis with mutation testing. Mutation testing introduces small changes — mutants — into your code and checks whether your tests catch them. If they don't, you have a gap. But mutation testing is expensive; it can take hours on a large codebase. Static analysis is cheap. So you can use static analysis to catch the low-hanging fruit — the bugs that are structurally obvious — and then use mutation testing to probe the deeper semantic gaps. Luna: I want to come back to the practical side. If I'm a QA engineer or a developer listening, and we don't have static analysis in our pipeline at all, what's the first step? Lucas: Pick one tool. There are excellent open-source options — for JavaScript, ESLint. For Python, Pylint or mypy. For Java, SpotBugs or Error Prone. For Go, staticcheck. Run it locally first on your own codebase and look at the output. You'll probably get hundreds of warnings. Don't try to fix them all. Just look for the ones that say 'null pointer dereference' or 'unused variable that could indicate a logic error' or 'potential SQL injection'. Luna: So start with the dangerous patterns, not the style issues. Lucas: Yes. Then add it to your CI pipeline, but configure it so that it only fails the build on severe warnings. Let the moderate warnings be advisory. And importantly — this is the lesson from Google — make it part of the pull request review. If your CI system can post comments on the PR with the warnings, that's ideal. Otherwise, at least make the report easily accessible. Luna: And what about tuning? How do you avoid the false-positive spiral? Lucas: You need a feedback loop. When a developer gets a warning they believe is a false positive, they should be able to suppress it with an inline comment — like 'suppress warning X because of reason Y'. Then periodically, the team reviews all suppressed warnings and decides whether to permanently disable that rule or adjust its parameters. Over time, the set of active rules converges to what's actually useful for your codebase. Luna: I've also seen teams use static analysis to enforce architectural rules — like 'no direct database access from the presentation layer' or 'all public methods must have a unit test'. Lucas: That's a great point. Some tools let you write custom rules. For example, you can say 'any class that extends Controller must not import java.sql.Connection'. That's not something a test would catch unless you specifically test for it, but static analysis can enforce it automatically. Luna: And that kind of rule is especially valuable in large codebases with many contributors, where drift is inevitable. Lucas: Absolutely. I want to give one more concrete number. In a study of over a hundred open-source Java projects, researchers found that static analysis detected an average of one real bug per two thousand lines of code. That doesn't sound like a lot, but think about it: if your codebase is two hundred thousand lines, that's a hundred bugs that your tests didn't catch. Luna: And those bugs are often the kind that cause production outages. Lucas: Right. And they're also the kind that are cheapest to fix if you catch them early. The cost of fixing a bug found in production is something like ten to a hundred times higher than fixing it during development. Static analysis is one of the cheapest ways to shift detection left. Luna: You know, this conversation reminds me of something I've been thinking about lately. This show — and the whole Fexingo network — is ad-free, and we keep it that way because of listeners who contribute small amounts. Lucas: Yeah, it's true. If today's conversation gave you something usable, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo. It makes a real difference, and it's how we stay independent. Luna: Absolutely. And it keeps us from having to chase sponsors or compromise on the content. So — back to static analysis. I want to push on something you said earlier about combining it with mutation testing. Lucas: Sure. The idea is that static analysis gives you a broad, shallow scan — it covers the whole codebase for certain patterns very quickly. Mutation testing gives you a deep, narrow probe — it checks whether your tests actually exercise specific behaviors. When you combine them, you get coverage of both the structural and the semantic dimensions. Luna: Is there any research on the combined effect? Lucas: Yes. A 2021 paper from researchers in Sweden looked at a medium-sized Java project and found that the combination caught about thirty percent more defects than either technique alone. And the false-positive rate for static analysis didn't increase because mutation testing doesn't have false positives — it just tells you whether your tests are adequate or not. Luna: So they complement each other almost perfectly. Lucas: They do. And the order matters: run static analysis first because it's fast, fix those issues, then run mutation testing on the cleaned codebase to find the gaps in your test logic. Luna: I want to ask about the tooling landscape. Are there any emerging trends worth paying attention to? Lucas: The biggest trend right now is integrating static analysis with large language models. Some new tools use machine learning to reduce false positives by learning which warnings are actionable for your specific codebase. There's also a movement toward 'continuous static analysis' — not just running it on every commit, but running it in the background as you type, like an IDE plugin that highlights potential issues in real time. Luna: So it becomes part of the development flow, not an afterthought. Lucas: Exactly. And that's where the biggest productivity gains come from. One study from a large tech company showed that developers who saw static analysis warnings in their editor fixed issues in an average of four minutes. Developers who saw them in CI took over an hour. Luna: That's a massive difference. So the lesson is: put the tool where the developer already is. Lucas: That's the takeaway. And start with the dangerous rules, tune aggressively, and measure your false-positive rate. If you do that, static analysis becomes one of the most cost-effective investments you can make in software quality. Luna: Alright, I think I'm going to go check our CI pipeline after this. Lucas: Same. Let's see how many of those two-thousand-line bugs we've been missing.