Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Load Testing Exposes Hidden Performance Bugs
Transcript
- Lucas: You know, I was talking to a friend last week who works on a fintech API — think payment processing, transaction history, that kind of thing. They had all these unit tests passing, integration tests green, even some contract tests in place. Luna: Sounds like a solid setup. So what was the problem? Lucas: They launched a new feature and within hours, response times went from 200 milliseconds to over 8 seconds during peak traffic. Users started seeing timeouts. And the thing is, their test suite hadn't caught a thing. Luna: Because none of those tests simulated real load, right? Lucas: Exactly. They had never run a proper load test. So today I want to talk about load testing — not as a checkbox activity, but as a way to expose performance bugs that live quietly under the surface until production hits them. Luna: I think a lot of teams treat load testing as something you do right before launch, maybe once a quarter. But the real value is catching regressions early, like any other test. Lucas: Right. And the fintech story I mentioned — after the incident, they ran a load test with 10,000 concurrent users, and what they found was a memory leak in a new service that processed transaction logs. Under low load, it was invisible. The garbage collector kept up. But under sustained load, memory usage climbed until the JVM started thrashing. Luna: So load testing is really about finding those non-linear behaviors. A system can handle 100 requests per second fine, but at 500, something breaks. Lucas: That's exactly the kind of bug I'm talking about. And it's not just memory leaks. Connection pool exhaustion is a classic — your database has a pool of, say, 50 connections. Under normal load, fine. But when a load test spikes, threads start queuing up, waiting for a connection, and suddenly your response time graph looks like an exponential curve. Luna: And that's often because developers set the pool size based on intuition, not on actual throughput requirements. Lucas: Right. So how do you do load testing right? First, you need to define what 'load' means for your system. Is it concurrent users? Requests per second? Throughput in bytes? For a web API, concurrent users is usually the right metric. Luna: And you need a realistic traffic pattern. Most tools let you define a ramp-up period — starting with a few users, then increasing over time. That's crucial because it lets you observe how the system behaves as load increases, rather than just hitting it with a constant flood. Lucas: Exactly. A flat load of 1,000 users from the start might trip a circuit breaker instantly, but a gradual ramp-up might reveal that at 800 users, response times start degrading. That's actionable data. Luna: And then there's think time. If your test scripts send requests back to back with no pause, you're not simulating real users. Real users read the screen, type, click. So you need to insert realistic delays between actions. Lucas: Yeah, that's a common mistake. Without think time, you're effectively doing a stress test, not a load test. Stress testing is valuable too — finding the breaking point — but load testing should reflect expected peak conditions. Luna: So what tools are people using these days? I know k6 has become pretty popular because it's scriptable in JavaScript and integrates well with CI/CD. Lucas: k6 is great. Locust is another — it's Python-based, very flexible. And then there's the old guard like JMeter, which still has a huge ecosystem but can be clunky to maintain as code. The important thing is to run these tests automatically in your pipeline, not as a manual step before release. Luna: That's where a lot of teams fall short. They do a load test once, get a green light, and never run it again until something breaks. Lucas: Right. And performance regressions are subtle. A new feature might add a database query inside a loop that's fine for 100 users but kills performance at 1,000. If your load test runs nightly, you catch that the next morning. Luna: So what metrics should you track during a load test? Obviously response time and error rate. But what else? Lucas: Apdex score is a great one. It measures user satisfaction based on a target response time — say, 500 milliseconds. Requests that complete under that are 'satisfied', between 500ms and 2 seconds are 'tolerating', and over 2 seconds are 'frustrated'. The score is a number between 0 and 1, and anything below 0.94 is usually a problem. Luna: I like that because it translates technical metrics into business impact. A 0.8 apdex means 20 percent of users are having a bad experience. Lucas: Another key metric is the 95th or 99th percentile response time. Averages lie — you might have a 200ms average but 5 percent of requests take 5 seconds. That's what users remember. Luna: And then there's the database. I've seen load tests where the app servers are fine but the database CPU goes to 100 percent because an index is missing. Lucas: Yes, you need to monitor infrastructure metrics too — CPU, memory, disk I/O, connection counts. Load testing is a full-system exercise. It's not just about the application code. Luna: So if someone is new to load testing, where should they start? Pick one endpoint, one realistic scenario? Lucas: Start with the most critical user journey — the one that makes the company money. For an e-commerce site, that's the checkout flow. Write a script that simulates a user browsing, adding to cart, and checking out. Run it with 50 concurrent users, then 100, then 200. See where it breaks. Luna: And don't forget to test with realistic data volumes. If your database has 1,000 users in test but 1 million in production, a query that's fast in test can be slow under load with real data. Lucas: That's a huge point. I've seen teams load test against a staging database with a fraction of the production data, and everything looks fine. Then in production, a join across a 10-million-row table kills performance. Luna: So you need to either copy a sanitized subset of production data or generate synthetic data that matches the volume and distribution. Lucas: If today's conversation gave you something usable — maybe a new metric to track or a tool to try — and you'd like to support the show, you can do that at buy me a coffee dot com slash fexingo. Luna: Yeah, even a couple of dollars a month genuinely helps keep these episodes ad-free and focused on the content. No pressure, but if you've gotten value from it, it makes a real difference. Lucas: We don't run ads, so listener support is what makes this sustainable. And we're grateful for every contribution. Luna: Alright, so back to load testing — we talked about starting with one critical journey. But once you have that baseline, how do you decide what thresholds to set for pass/fail in your CI pipeline? Lucas: That's the next step. You need to define Service Level Objectives, or SLOs, for your performance tests. For example, '99 percent of requests must complete in under 500 milliseconds' or 'error rate must be below 0.1 percent'. Then your CI job fails if those are breached. Luna: And those thresholds should be based on actual business requirements, not just arbitrary numbers. If your app needs to handle Black Friday traffic, you should test at that level. Lucas: Right. And you need to test at different load levels — not just peak. Test at 50 percent of peak, 80 percent, 100 percent, and maybe 120 percent to see the safety margin. The 80 percent test is often the most telling because it's close to real conditions but leaves room for spikes. Luna: One thing I've seen teams struggle with is interpreting results. They run a load test, get a bunch of graphs, and don't know what to look for. Lucas: The first thing to look for is the inflection point — where response time starts to increase non-linearly. That's usually where a bottleneck kicks in. Then look at error rates and resource utilization. If CPU is at 100 percent but response time is still good, you might be okay, but you're close to the edge. Luna: And compare with previous runs. A regression might not be a hard failure but a gradual degradation. If your p99 response time was 300ms last week and 350ms this week, that's a trend to investigate. Lucas: Exactly. Load testing is a practice, not a one-time event. The teams that do it well integrate it into their development cycle and treat performance as a first-class quality attribute, not an afterthought. Luna: And they avoid the kind of production incident my friend's fintech team had. That's a good goal to aim for.