Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Performance Regressions Before Users Notice
Transcript
- Lucas: So Luna, here's a number that should make any engineer wince. A payments API we can't name saw its average response time creep up by 30 percent over six months. No crashes, no errors, no deploy that broke anything. Just a slow, steady drift that nobody noticed until a customer complained. Luna: And that's the classic performance regression. It's not like a functional bug that fails a test. It's a quiet degradation that users feel before your dashboards do. Lucas: Exactly. And that's why performance regression testing is so hard, and so essential. It's not a one-time load test. It's a continuous discipline. Today I want to dig into how you actually catch these drifts before they hit production. Luna: So where do you start? You can't just throw a load test into your CI pipeline and call it a day. Lucas: Right. First, you need a clear definition of what a performance regression is. It's not just 'the app feels slower.' It's a measurable change in a key metric, like response time, throughput, or resource usage, that makes the system violate a budget you've set. Luna: And that budget has to be realistic. If you set your p99 latency goal at 200 milliseconds but your users are on 4G in rural areas, you might be chasing an impossible target. Lucas: Exactly. So step one is to define your performance budget based on real user data. Tools like synthetic monitoring can give you a baseline, but you also need real user monitoring, or RUM, to see what actual devices and networks are doing. Luna: So once you have that budget, how do you enforce it? You can't run a full load test on every commit. Lucas: You're right. That's where a tiered approach helps. On every commit, you might run a quick smoke test with a handful of virtual users. That catches gross regressions, like an accidental infinite loop. Then, nightly, you run a more realistic load test with a bigger user mix. And before a major release, you do a full-scale test in a staging environment. Luna: But even nightly tests can be flaky. What if the network or the test environment itself is slow? You end up chasing false alarms. Lucas: That's a real problem. The key is to make the tests deterministic. You need to isolate the environment, use dedicated hardware or containers, and keep the test data consistent. And you should compare against a known good baseline, not an absolute threshold. So instead of 'response time must be under 500 milliseconds,' you say 'response time must not increase by more than 10 percent from the previous run.' Luna: That's clever. It reduces the impact of environment noise. But what if the regression is subtle, like a 5 percent increase in CPU usage that doesn't affect response time yet? Lucas: That's where profiling comes in. You don't just measure response time. You measure CPU, memory, I/O, and even database query time. A regression can show up as a slow memory leak or a query that's suddenly scanning more rows. Tools like continuous profiling can give you a flame graph for every release, so you can see what changed. Luna: So it's about catching the root cause, not just the symptom. But how do you decide what metrics to track? Lucas: Start with what your users care about. For a web app, that's usually page load time. For an API, it's latency and error rate. But you also need to track the underlying resources, like CPU and memory, because those are leading indicators. If memory usage is climbing after every release, you're headed for a crash. Luna: And what about percentiles? Everyone talks about p99, but is that enough? Lucas: P99 is a good start, but it's not the whole picture. If you only look at p99, you miss the tail. Some systems have a few requests that take 10 seconds, and those might be the ones that cause users to churn. So you should track p50, p95, and p99, and also look at the maximum. Luna: And then you have to decide what to do when a regression is detected. Do you block the release? Lucas: That depends on the severity. If it's a small regression that's within your noise margin, you can log it and move on. But if it breaks your budget, you should fail the build. And that's where the culture comes in. You need to have clear ownership. Someone has to be responsible for performance, or it'll slip through the cracks. Luna: And that's often the hard part. It's not a technical problem, it's a team problem. Lucas: Right. You can have the best tools in the world, but if the team doesn't value performance, you'll get ignored. So one thing I've seen work is to make performance part of the definition of done. Every feature has to include a performance test, or at least a checklist item. Luna: So let's talk about testing in production. Some people say that's the only way to get real performance data. What do you think? Lucas: Testing in production is valuable, but it's risky. You need to have the right guardrails. For example, you can use traffic mirroring or canary releases. You send a small percentage of live traffic to the new version and compare its performance to the old one. If you see a regression, you roll back. Luna: That's smart. But you still need a baseline in your staging environment, because production is noisy. Lucas: Absolutely. Staging tests give you a clean comparison. Production gives you the truth. You need both. Luna: And what about the cost? Running performance tests, especially load tests, can burn through cloud credits pretty fast. Lucas: That's a real consideration. One approach is to use a shared, reserved environment for nightly tests, rather than spinning up new infrastructure every time. And you can use open-source tools like k6 or Locust to keep costs down. But you also need to weigh the cost of a performance regression. A single incident can cost more than a year of testing. Luna: That's a good point. And speaking of cost, I think there's a connection here to the show itself. We keep this podcast ad-free, and that's only possible because of listener support. If you find these episodes useful, you can support us at buy me a coffee dot com slash fexingo. It helps us keep doing deep dives like this. Lucas: Yeah, that's a nice parallel. Just like performance testing, listener support is a small investment that prevents a bigger problem down the road. But back to the technical side. I want to give you a concrete example of how to set up a performance regression test. Luna: Please. I love concrete examples. Lucas: So let's say you have a REST API. You write a simple script that hits the health endpoint, the login endpoint, and a data-heavy endpoint. You run it against a known good build and record the response times. Then you add it to your CI pipeline as a post-deploy check. On every deploy, it runs the same script and compares the results. Luna: And what happens if it fails? Do you have to roll back automatically? Lucas: Not necessarily. It depends on the severity. If it's a major regression, you might want to alert the on-call engineer. But if it's a minor blip, you can just log it and track the trend. The key is to have a threshold that's meaningful. A 5 percent regression might be acceptable, but a 30 percent regression is not. Luna: And how do you determine that threshold? It's not always obvious. Lucas: You have to look at your historical data and see what the natural variation is. If your response times swing by 10 percent on a normal day, then a 10 percent regression is not meaningful. But if it's normally stable within 2 percent, then a 5 percent change is a red flag. Luna: So it's about understanding your own system's volatility. And that takes time and monitoring. Lucas: Exactly. And that's why you need both automated tests and human oversight. The machine can flag the anomaly, but a human has to interpret it in context. Luna: I think we've covered a lot here. Do you have any final advice for someone starting out with performance regression testing? Lucas: Start small. Pick one critical endpoint, set a realistic budget, and add a simple test to your CI. Don't try to boil the ocean. Once you have that working, expand to other endpoints and add profiling. The key is to make it a habit, not a one-off project. Luna: And remember, the goal isn't to catch every regression. It's to catch the ones that matter. Because users don't always tell you when things are slow. They just leave. Lucas: Right. Performance is a feature. And like any feature, it needs to be tested. Thanks for listening, everyone. We'll see you next time.