Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Performance Testing Reveals Hidden Bottlenecks in Microservices
Transcript
- Lucas: So there's this story from a couple years back — a major e-commerce platform, the kind that does billions in revenue, was gearing up for Black Friday. Their QA team ran the usual load tests on the checkout service. Everything looked fine in the lab. But when the real traffic hit, the checkout page took over three seconds to load. Users started abandoning carts. The team was scrambling. Luna: Three seconds? That's basically an eternity for checkout. What did they find? Lucas: They eventually discovered a 300-millisecond latency spike in a single microservice — the one that applies discount codes. But here's the kicker: their load tests never caught it because they were sending requests at a steady rate, not the bursty pattern of real Black Friday traffic. The discount service had a hidden bottleneck when hit with sudden spikes. Luna: So the tests were too clean. They weren't simulating the chaos of actual users clicking 'apply coupon' all at once. Lucas: Exactly. And that's the theme of today's episode — why performance testing in microservices often misses the real bottlenecks unless you deliberately design for bursty, uneven traffic. And how distributed tracing is the tool that finally reveals where those milliseconds are going. Luna: Before we dive deeper — if today's tech conversation gave you something usable, know that these episodes stay ad-free because of listener support. You can find us at buy me a coffee dot com slash fexingo. It's a small way to keep the show independent and focused on what matters. Lucas: Yeah, it really does help. And it means we can keep digging into stories like this one without any sponsor influence. So back to that discount service — the team finally traced the slowdown to a database query that was doing a full table scan when it received too many concurrent requests. In isolation, the query took 50 milliseconds. Under burst load, it ballooned to 350. Luna: Right — the classic 'works fine in dev, falls over in prod' problem. But how did they even find that specific service? In a microservices architecture, you've got dozens of services involved in a single checkout. Lucas: They used distributed tracing — every request carried a unique ID that propagated across service boundaries. They could see the entire waterfall of calls: API gateway, user service, inventory service, discount service, payment service. The discount service's span was the one growing under load. That pinpointed the culprit. Luna: So without tracing, they would've been guessing. Maybe blamed the payment gateway or the front end. Lucas: Exactly. And this is why I think every team running microservices should treat distributed tracing as a first-class part of their performance testing setup. Not just something you turn on after an outage. The e-commerce team learned that lesson the hard way — they added tracing to all services and now run load tests with traffic patterns recorded from previous Black Fridays. Luna: Recorded traffic patterns — that's a great point. You're replaying actual user behaviour, not synthetic scripts that hit endpoints in a predictable order. Lucas: Precisely. They used a tool that captured production traffic and replayed it against a staging environment with the same data volume. That's how they found the discount service's bottleneck before the next Black Friday. They added an index on the discount code table and implemented connection pooling. The 300-millisecond spike dropped to 30. Luna: Thirty milliseconds? That's a tenfold improvement from a database index and connection pooling. Simple fix, but invisible without the right test. Lucas: And that's the takeaway — performance testing in microservices isn't just about throwing more requests at a service. You have to think about concurrency patterns, data volume, and observability. A lot of teams run what I call 'happy path load tests' — they hit a few endpoints with steady ramp-ups and call it done. But real traffic is spiky, bursty, and full of edge cases like expired coupons or invalid discount codes. Luna: So what's the alternative? If steady ramp-up is insufficient, what's a better approach? Lucas: I'd say two things. First, model your load test after production traffic patterns — use tools like Gor or Tcpreplay to capture real requests and replay them. Second, instrument every service with tracing from day one, so when a slowdown happens, you can follow the breadcrumbs. And then run the load test at different times of day, because traffic patterns change. Luna: That second point — tracing from day one — is something a lot of teams overlook. They add tracing after an incident, but by then they've lost historical context. Lucas: Exactly. The e-commerce team now has tracing data going back years. They can compare Black Friday 2025's performance to 2026's. They can spot regressions before they become outages. And they've built dashboards that show the 99th percentile latency for every service in the checkout flow. If a service exceeds 200 milliseconds, they get an alert. Luna: That's proactive. You're not waiting for users to complain — your system tells you something's off. Lucas: Right. And I think this is where performance testing is heading — not a separate phase before release, but a continuous feedback loop integrated with monitoring. You run load tests nightly, compare them to production baselines, and any anomaly triggers a review. That's how you catch a 300-millisecond spike before it becomes a three-second checkout. Luna: Let's talk about the database side for a second. You mentioned the full table scan. What other database-level bottlenecks do you see in microservices performance tests? Lucas: A big one is connection pool exhaustion. Each microservice often has its own pool, and under burst load, one service can hog all the connections, starving others. I've seen cases where a single misconfigured pool caused cascading timeouts across five services. Performance tests that only hit one service at a time won't catch that. You need end to end tests that simulate the full flow. Luna: So you're saying you need both — service-level tests for isolation and end to end tests for interaction effects. Lucas: Exactly. And the end to end tests should use realistic data volumes. A lot of teams test with a thousand users in the database, but production has ten million. Queries that are fast with a thousand rows can become slow with ten million. The e-commerce team actually found that the discount code table had grown to 50 million rows — the index they added needed to be a composite index to cover the query pattern. Luna: Fifty million rows — that's not something you can simulate easily in a local dev environment. So staging environments need to be production-scale or at least have representative data. Lucas: Right. And that's expensive. Not every team can afford a full-scale staging environment. So they use techniques like data subsetting — take a sample of production data that preserves the distribution of values, including edge cases like expired coupons. Or they use synthetic data generation tools that mimic production patterns. The key is to avoid the 'happy path' data set. Luna: I've also seen teams use something called 'traffic mirroring' — they copy a portion of production traffic to a staging environment and observe how it performs. That's like a real-time load test. Lucas: That's a great technique. It's called shadow traffic or dark launch. You route a copy of live requests to a staging version of your new service, but the response is discarded. You can measure latency and errors without affecting users. It's the ultimate realistic load test because it's actual traffic. But it requires careful infrastructure to ensure the mirrored traffic doesn't disrupt staging. Luna: And you need to handle sensitive data — you might have to strip or anonymize personally identifiable information before mirroring. Lucas: Absolutely. Data privacy is a concern. But when done right, shadow traffic gives you the highest fidelity feedback. The e-commerce team used it to validate their fix before the next Black Friday. They mirrored a small percentage of traffic to the new discount service, confirmed the latency was stable, then rolled it out to all users. Luna: That's a smooth rollout strategy. So to sum up — use realistic traffic patterns, distributed tracing, production-scale data, and continuous performance testing. And don't forget shadow traffic for high-risk changes. Lucas: Exactly. And one last thing — don't overlook the human side. The team that found this bottleneck had a culture of blameless post-mortems. When the checkout slowed down, they didn't point fingers at the discount service team. They asked, 'What in our process allowed this to go unnoticed?' That led them to invest in better load testing and tracing. Culture matters as much as tools. Luna: That's a great note to end on. Performance testing isn't just about technology — it's about how teams learn from failures. Lucas: Right. And if you want to share your own performance testing war stories, we'd love to hear them. Until next time.