Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Testing Your Test Pyramid When It Collapses
Transcript
- Lucas: So the test pyramid. We all know it. Unit tests at the base, integration tests in the middle, end to end tests at the top. It's the first diagram they show you in any testing course. But what happens when your pyramid starts to look like a sandcastle at low tide? Luna: A sandcastle at low tide. That's a vivid image. What does that mean in practice? Lucas: It means your base is so wide and so slow that it stops being a foundation and just becomes a pile of sand. I was talking to a lead engineer at a mid-sized fintech company last week, and she told me their unit test suite takes forty minutes to run. Forty minutes for what are supposed to be fast, isolated tests. Luna: Forty minutes for unit tests. That's not a unit test suite, that's an integration test suite in a trench coat. Lucas: Exactly. And the worst part is, they still had bugs slipping through. Their coverage was at eighty percent, which on paper sounds decent. But the tests were so slow and so brittle that developers were skipping them locally and just pushing to CI, hoping for the best. Luna: So the pyramid's base is both too heavy and too weak. What did they do about it? Lucas: They took a two-pronged approach. First, they did what they called a 'test suite fitness' audit. They ran each test with a fault injection tool that deliberately broke the code under test, and then checked which tests actually caught the fault. Turns out, a third of their unit tests never failed when the code broke. Luna: A third of tests never fail even when the code is broken. That's basically test theater. Were those tests just asserting trivial things? Lucas: Mostly, yeah. They were asserting that a mock was called, or that a particular getter returns the value it was just set to. Tests that are so tightly coupled to implementation that they can't detect real regressions. They were passing for the wrong reasons. Luna: So they get rid of those? That must have cut the suite down a lot. Lucas: It did. They deleted about twenty percent of their tests. But the bigger change was structural. They realized a lot of their 'unit' tests were actually spinning up a database, hitting a cache, or doing real HTTP calls. Which is why they were slow. So they moved those into a separate integration suite and made them run nightly, not on every commit. Luna: That's a classic move. But doesn't that risk losing the fast feedback on integration issues? Lucas: That's the trade-off, but they built a safety net. They introduced what they call a 'slow test budget' on their main branch. Any test that takes more than two seconds gets flagged, and if a new commit pushes the total suite time over a hard threshold, the build is red. So the suite can't silently bloat again. Luna: A slow test budget. That's a concrete rule I could see teams actually adopting. Lucas: And it's not just about time. They also started measuring what they call 'test suite fitness' — the ratio of tests that catch a real fault when injected. That's a much better metric than coverage alone. Coverage tells you what code was executed, but not whether the test would actually catch a regression. Luna: Right, because you can have a test that executes every line but asserts nothing meaningful. So fitness is about the test's ability to detect a change in behavior. Lucas: Exactly. And they set a goal: every unit test must catch at least one injected fault. If a test doesn't, it's either rewritten or deleted. That's a high bar, but it forces tests to be behavioral rather than implementation-obsessed. Luna: That sounds like it would take a lot of effort. Did they actually do it for every test? Lucas: They did it for the unit suite. They killed another ten percent of tests that were redundant or just couldn't be made useful. And they replaced some with a handful of well-designed property-based tests that catch edge cases you'd never think to write by hand. Luna: So after all that, what's the suite look like now? How fast is it? Lucas: The unit suite went from forty minutes down to about six. And the integration suite runs nightly in about twenty minutes. They still have end to end tests, but those are down to a few critical journeys, and they run on a schedule and on-demand, not on every commit. Luna: Six minutes for the fast feedback loop. That's a huge improvement. Do you think the pyramid model is still useful, or is it maybe time to retire it? Lucas: I think the pyramid's shape is still a useful mental model, but it oversimplifies. In this team's case, they ended up with a diamond shape — a big middle of integration tests that run nightly, a smaller base of fast unit tests, and a sliver of end to end. And that's okay, as long as the fast feedback loop is genuinely fast. Luna: A diamond shape. I like that. But what about teams that don't have the resources to do fault injection audits and property-based testing? Is there a simpler first step? Lucas: Definitely. Start by measuring your suite time on every commit. If it's over ten minutes, you already have a problem. Then use a simple script to find the top ten slowest tests and see why they're slow. Often they're hitting a real service or doing file IO. Move those to a slower tier. And then, maybe most importantly, start a 'test debt' backlog item and carve out time each sprint to delete or rewrite tests that don't add value. Luna: It feels like the key is treating your test suite as a product itself, not just a byproduct of development. It needs maintenance, pruning, and a clear owner. Lucas: That's exactly right. And that's a nice segue because we put the same care into this show. We don't run ads, and we like to keep it that way. If you find these conversations useful, you can support the effort at buy me a coffee dot com slash fexingo. It's a simple, optional way to keep us independent and ad-free. Luna: Yeah, that's the only reason we can talk for ten minutes about test fitness and not have to interrupt for a sponsor read. So if that's your thing, the link is there, no pressure. Lucas: And it genuinely helps us decide what to cover next. We read every message. Now, back to that diamond shape — one thing that struck me is that even after they cut thirty percent of their tests and moved a bunch to nightly, their bug rate didn't go up. It actually went down. Luna: That's counterintuitive to a lot of people who think more tests equals more safety. But it makes sense if the remaining tests are higher quality and actually catch real faults. Lucas: Right. And they also reduced their CI time so much that developers started running the full unit suite locally before pushing. That alone caught a ton of issues earlier, before they even hit the shared pipeline. Luna: So the real win wasn't just deleting tests — it was restoring trust in the fast feedback loop. When tests are fast, people actually run them. Lucas: Exactly. And that's the thing. The pyramid is a tool, not a law. The real principle is: fast, deterministic tests that catch real regressions, and a balance that matches your risk profile. For a fintech company, you might want more integration coverage than a content site. That's fine. Luna: So what's the one thing a listener should do on Monday morning if they're staring at a forty-minute test suite? Lucas: Pick the ten slowest tests and write down why they're slow. Just that. You'll probably find that ninety percent of the time is spent in a handful of tests that are doing way too much. Move them out of the unit tier, and you'll already see a dramatic improvement in your feedback loop. Luna: And then set a budget for the rest. That's a good Monday plan. Lucas: And if you want to take it further, try the fault injection audit. It's more work, but it's the only way to know if your tests are actually earning their keep. I'd love to hear from anyone who tries it — we're on the usual channels. Luna: So the pyramid might be crumbling, but that's okay. We can build something better. Lucas: We can. And we should. Next time we'll talk about how to make your test data itself more reliable — property-based testing is just the start.