Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Performance Testing Catches What Unit Tests Miss
Transcript
- Lucas: Earlier this year, a fintech startup I've been loosely following had a near-disaster. Their checkout service—the thing that actually charges customers—survived two full rounds of unit tests with 94% coverage. Every function returned the right value. Every mock behaved. And then Black Friday hit, and the service collapsed after about 200 concurrent users. Luna: 200 concurrent users is not a lot. What happened? Lucas: A database connection-pool leak. The unit tests were all written against an in-memory SQLite mock that never actually exhausted connections. In production, each request opened a connection, did its work, and under load, a code path in the error handler skipped the close call. After about 250 connections, the pool was gone. New requests just hung. Luna: So the code was correct in isolation, but wrong in a real system. That's exactly the kind of bug that performance testing is supposed to catch. Lucas: Exactly. And it's a category of bug that unit tests are fundamentally blind to—because unit tests test one thing at a time, usually with mocked dependencies. They can't reproduce the timing, the resource contention, the state accumulation that happens when hundreds of requests overlap. Luna: I think a lot of teams treat performance testing as optional or 'phase two.' They get the unit tests passing, ship to production, and only find out about bottlenecks when something breaks. Lucas: Right. And the irony is that performance testing doesn't have to be that expensive. Take that fintech case: a simple load test with a tool like k6 or Locust, running just 50 virtual users for five minutes per pull request, would have shown the connection count climbing without dropping. The test would have failed. Luna: But don't those tools require a full staging environment? That's the pushback I hear: 'We can't run load tests in CI because we don't have the infrastructure.' Lucas: It's a fair concern, but I think people overestimate what's needed. You can run a smoke-level load test—say 10 to 20 virtual users—against a containerised version of the service in CI. It won't show you absolute scale limits, but it will catch resource leaks, thread deadlocks, and connection-pool exhaustion. That's where the highest-impact bugs live. Luna: So you're saying a small, fast load test is better than no load test. And you save the full-scale 10,000-user test for a weekly staging run. Lucas: Exactly. The fintech company now has three tiers: a 20-user smoke test on every merge to main, a 500-user test that runs every night in staging, and a full-scale holiday load test they schedule a month before peak events. The unit tests still run, obviously, but they also check response time percentiles and error rates under load. Luna: What about the metrics themselves? Latency and throughput are the obvious ones, but are there other signals that performance tests should track? Lucas: Great question. I'd add three things. First, database connection count and pool wait time—that's what caught the bug I mentioned. Second, garbage collection metrics for languages like Java or Go—a high GC pause can make latency spike even if the code is correct. And third, thread or goroutine count. If that number keeps climbing and never stabilises, you have a leak. Luna: Those are all resources. It's like monitoring the system's vital signs. Lucas: If today's tech conversation gave you something usable, I'll mention something we don't usually talk about on the show. We keep Fexingo completely ad-free—no sponsors, no mid-rolls. The only reason we can do that is a small percentage of listeners who choose to support the show directly. Luna: Yeah, it's a model we believe in. If you find value in episodes like this, the simplest way to back that choice is buy me a coffee dot com slash fexingo. Totally optional, but it helps keep the production independent. Lucas: And back to performance testing. Another misconception I run into is that performance testing is only about speed. But it's also about correctness under concurrency. Shared mutable state—like a static cache that isn't thread-safe—can cause data corruption that only shows up with two simultaneous requests. Luna: Right, the classic race condition. And unit tests with mocks often run sequentially, so they never trigger the race. Lucas: Exactly. I've seen a payment service where a simple increment of a retry counter used 'counter += 1' instead of an atomic operation. Under low traffic, it worked perfectly. Under 100 concurrent requests, the counter would occasionally skip numbers, leading to incorrect retry limits and failed payments. Luna: A load test with two concurrent users would have caught that. That's the bar—two concurrent users. Lucas: Yeah. And I think that's the key insight: performance testing isn't just about scale. It's about timing. When two things happen at nearly the same time, does the system still behave correctly? Unit tests, by design, almost never answer that question. Luna: So if a team has limited time, where should they start? Which performance test gives the highest return for the least effort? Lucas: I'd say a load test that simulates the 95th percentile of your normal traffic, run against a local or containerised version of the service. Focus on three metrics: error rate, p99 latency, and a resource that's likely to be constrained—like database connections or memory. If you run that on every pull request, you'll catch most of the 'works on my machine' bugs that turn into production outages. Luna: And you don't need a dedicated performance engineer for that. It's a script that takes maybe 50 lines of code with a tool like k6. Lucas: Exactly. And once you have that script, you start building a performance regression baseline. You can see: 'This deploy added 50 milliseconds to p99 latency.' That's actionable data that unit tests can't give you. Luna: What about the argument that performance tests are flaky? That they fail because of noisy environments, not real bugs? Lucas: It's a real issue. The solution is to run performance tests in a dedicated environment or at least with resource limits. Also, use statistical thresholds rather than strict pass/fail—like 'p99 latency must be under 500 milliseconds for 9 out of 10 runs.' That filters out noise without hiding regressions. Luna: So you're trading a bit of precision for stability. Makes sense. Lucas: And combine that with APM tools like Datadog or New Relic in production. If your load test passes but users still see slowness, the APM traces will show you exactly which database query or external API call is the bottleneck. That's the feedback loop—performance testing in CI catches the obvious stuff; APM catches the subtle production-only issues. Luna: I think the deeper point is that performance testing forces you to think about your system as a system—not just a collection of functions. The same reason that integration tests catch bugs unit tests miss. Lucas: Exactly. And the cost of not doing it is visible. That fintech startup I mentioned lost an estimated 300,000 dollars in revenue during the four hours their checkout was down on Black Friday. Their unit test suite, with its 94% coverage, reported all green. Luna: A single performance test would have cost maybe a developer-hour to write and pennies to run. That's a pretty stark return on investment. Lucas: It's one of those cases where a little upfront effort saves a lot of pain later. And the tools are mature enough now that there's really no excuse not to run at least a basic load test in your pipeline. Luna: Alright, I'm convinced. Next time I'm setting up a CI pipeline, I'm adding a k6 script before the unit tests even finish. Lucas: That's the spirit. And if you hit a tricky performance bug that your load test missed, let us know—maybe we can cover it in a future episode.