Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Teams Stop Testing From Becoming The Bottleneck
Transcript
- Lucas: The most expensive bug in software engineering isn't the one that crashes production on Black Friday. It's the one that makes your team wait three days for a release because the QA team hasn't finished their regression suite yet. We talk a lot about shadow testing and chaos engineering, but there's a structural problem that happens long before code hits production. It's the bottleneck of human judgment slowing down automated systems. Luna: Right, and I see this constantly where the 'gate' mentality turns QA into a police force instead of a partner. If the goal is speed, why do we still structure our workflows like we're trying to catch criminals? Lucas: Exactly. And since listener support keeps these conversations running ad-free, if today's tech conversation gave you something usable, consider supporting the show at buy me a coffee dot com slash fexingo. It helps us dig deeper into these structural issues without chasing clicks. Luna: Appreciate that, and it definitely aligns with the idea of removing friction from the process we're discussing. Let's get back to the bottleneck then. Lucas: So here's the concrete reality. Most mid-sized engineering teams have ten times more automated tests than they have developers. But those tests don't run in parallel efficiently. They run sequentially because they depend on shared databases or mocked services that aren't isolated. The result? A developer pushes code at nine in the morning. The CI pipeline breaks at noon because a flaky integration test failed. By the time someone manually investigates whether it's a real bug or just bad data, the developer has already moved to another task. Luna: That flakiness is the silent killer of velocity. You lose trust in the pipeline when red means nothing half the time. Lucas: And that loss of trust leads to the worst habit in modern software: ignoring the failures. Engineers start marking tests as ignore. They start bypassing checks to meet sprint deadlines. We covered data contracts earlier, which help prevent integration breakdowns, but we haven't really talked about the human workflow around those contracts. The issue isn't just that tests fail. It's that the feedback loop is too slow to be useful. Luna: So you're saying the problem is temporal? That by the time the test result comes back, the context is gone? Lucas: Precisely. In high-performing teams, the feedback loop needs to be under five minutes for unit tests and under fifteen minutes for integration tests. If it takes longer, the cognitive load spikes. Developers stop writing tests because the penalty for writing them is lost focus. So what do they do? They write fewer tests. Or they write brittle tests that only check surface-level UI changes. Luna: Which brings us right back to why so many teams end up with expensive noise instead of actual protection. Lucas: Right. And this is where the shift left philosophy gets misinterpreted. People think shift left means testing earlier in the timeline. But it actually means testing closer to the source of change. If a developer changes a single function, they shouldn't need to spin up a full staging environment to verify that change. They should need a local sandbox that mirrors the production dependency graph. Luna: I've seen companies try this with Docker containers, but the overhead of spinning up five different microservice containers locally can take twenty minutes. That defeats the purpose. Lucas: That's the trap. Containerization solved infrastructure, but it didn't solve orchestration latency. The solution isn't faster containers. It's mocking at the protocol level. When you mock the API contract instead of the entire service, you cut that setup time from minutes to seconds. But mocking requires discipline. You have to agree on the shape of the data before you write the code. Luna: Which sounds like a communication problem disguised as a technical one. Who defines that contract first? Lucas: In the best teams, it's the consumer. The team consuming the API writes the test for the contract they expect. Then the provider implements the service to pass that test. This is often called consumer-driven contract testing. It flips the traditional model where the provider says, 'Here's my API,' and the consumer complains later when it doesn't work. Luna: It puts the burden of accuracy on the person who actually uses the output. Makes sense. Lucas: And it eliminates the bottleneck. Because the test lives next to the code that uses it, it runs every time that code changes. No waiting for a central QA team to pick up the ticket. No waiting for a scheduled nightly build. The test becomes part of the pull request. If the contract changes, the build fails immediately. The developer sees it while they're still thinking about the logic. Luna: But doesn't that require devs to understand the whole system's interface, not just their own module? Lucas: It does. And that's the cultural shift. We used to silo QA as the guardians of quality. Now, quality is a property of the commit, not the department. This doesn't mean QA roles disappear. It means they evolve. Senior QA engineers become platform builders. They build the tools that make it easy for developers to write those contract tests. They remove the friction. Luna: So the QA engineer stops being the tester and starts being the architect of the testing experience. Lucas: Exactly. And this is where we see the biggest ROI in recent years. Companies that restructured their QA teams into Enablement Platforms saw deployment frequency increase by forty percent within six months. Not because they wrote better code, but because they stopped interrupting the flow of code. Luna: Forty percent is huge. What was the trade-off? Did bugs leak through? Lucas: Actually, escape defects dropped. Here's why. When you rely on a centralized QA team, you create a queue. Things sit in the queue. While they sit, requirements change. Context decays. The tester is testing yesterday's understanding against today's code. In the distributed model, the person who wrote the code verifies it against the immediate expectation. The signal to noise ratio is much higher. Luna: So the centralized model creates lag, which creates errors. Distributed testing reduces lag, which preserves accuracy. Lucas: Yes. But there's a risk. If everyone tests their own slice, you might miss the integration points. Two services might work perfectly in isolation but fail when combined. That's why you still need a small layer of end to end tests. But those should be rare, stable, and run only on main branch merges, not on every pull request. Luna: So you keep the fast lane for individual components and the slow lane for the whole vehicle. Lucas: That's the metaphor. Fast lane, slow lane. Fast lane catches ninety-five percent of errors instantly. Slow lane catches the two percent that only appear in complex interactions. And because the slow lane runs infrequently, it can afford to be comprehensive. It can spin up the real database. It can simulate the network latency. It doesn't need to be fast because nobody is waiting for it during development. Luna: This feels like a fundamental redesign of the developer lifecycle. Are most teams ready for this complexity? Lucas: Most aren't. Most are still stuck in the middle. They've automated their tests but kept the manual gate. They have faster horses instead of cars. The transition requires leadership to accept that testing is no longer a phase. It's a continuous activity. You can't schedule quality. You have to build it in. Luna: And that means measuring test health differently. Not just coverage percentages, but how much time developers spend waiting for results. Lucas: Exactly. Lead time for changes is the metric that matters. If your test suite adds four hours to the lead time, it's costing you money. Every minute of pipeline wait time is unpaid labor. The goal isn't zero bugs. The goal is zero delay. You want to fail fast, but you also want to recover instantly. Luna: Fail fast, recover instantly. That's a much better north star than perfect code. Lucas: Because perfect code doesn't exist. Broken processes do. When you fix the process, the code follows. The best testing strategy is invisible. You don't notice it until it's gone, and then you realize you were driving blind for years. Luna: So the question for listeners isn't how to test more, but how to test less effectively. Lucas: Or rather, how to test in a way that doesn't interrupt the work. If your tests feel like homework, they're probably wrong. They should feel like guardrails. You only notice them when you almost go off the road. Luna: Guardrails, not gates. I like that distinction. Lucas: Right. And as your team grows, those guardrails need to be standardized. That's where the platform team comes in again. Providing standardized testing frameworks so every squad isn't reinventing the wheel. Consistency reduces cognitive load. Luna: Standardization enabling speed. It's counterintuitive but it holds up. Lucas: It always does. Complexity demands standardization. Simplicity allows flexibility. If your testing framework is simple enough for a junior dev to extend, you win. If it's complex enough to require a specialist, you lose velocity. Luna: So the bar for test simplicity should match the bar for code simplicity. Lucas: Absolutely. Tests are code. They deserve the same refactoring attention. If a test file is getting too long, break it up. If it's hard to read, rewrite it. Don't let the testing layer become the legacy codebase nobody touches. Luna: That's a practical takeaway. Treat your tests with the same respect as your product code. Lucas: They are the documentation of your intent. When you read a test, you understand what the system is supposed to do. When tests rot, that documentation dies. And then you're guessing. Luna: Guessing is expensive. Especially when it involves production deployments. Lucas: It is. So the challenge for this quarter is to look at your CI pipeline. Count the minutes spent waiting. Identify the longest bottleneck. Ask if that bottleneck adds value or just adds delay. Usually, it's just delay. Luna: And replace that delay with confidence. That's the real shift. Lucas: Confidence built into the flow, not bolted on at the end. Thanks for listening. How are you handling test bottlenecks in your current projects?