Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How a CTO Used Load Testing to Prevent a Black Friday Meltdown
Transcript
- Lucas: So a CTO I know — let's call her Priya — runs the engineering team at a mid-size e-commerce company. Think clothing, home goods, maybe fifty million in annual revenue. Last October, six weeks before Black Friday, she does something pretty smart. Luna: Load testing, I'm guessing? Lucas: Exactly. She runs a full-scale load test against the checkout flow. Simulates three times the expected Black Friday traffic. And within the first minute, the entire system locks up. Checkout fails for every simulated user. Luna: That's terrifying. But also kind of the point — better to find it in October than November. Lucas: Right. So the bottleneck turns out to be a single PostgreSQL query. A query that had been running fine in staging with maybe fifty concurrent users. But under three thousand concurrent users, it created a row-level lock that cascaded into a full database stall. Luna: What was the query doing? Lucas: It was a product inventory lookup — checking stock levels and applying a discount rule. The discount logic required a sequential scan on a table with about two million rows. In staging, the table had maybe ten thousand rows. So the query plan was completely different. Luna: Classic staging vs production data mismatch. How did she even decide to load test? A lot of teams skip it until something breaks. Lucas: She told me she had a bad feeling about the new discount engine they'd shipped in September. The feature had been rushed — three sprints instead of the planned five. So she made load testing a condition for the Black Friday readiness review. She used k6, which is open-source, scripted ten user journeys: browse, add to cart, checkout, payment failure, password reset, the whole thing. Luna: And the checkout journey was the one that broke. Lucas: Yeah. And here's the interesting part — the fix wasn't a massive rewrite. They added a read replica to offload the inventory lookup, and they cached the product catalog endpoint with a five-second TTL. Total infrastructure cost increase: about four hundred dollars a month. Luna: Four hundred bucks to prevent a Black Friday outage that could have cost them millions. That's a no-brainer. Lucas: Exactly. But the bigger lesson is about how she sold it to the CEO. She didn't lead with 'we need to spend more money on infrastructure.' She led with 'we found a risk that could take down checkout, here's the fix, it costs four hundred a month, and here's the expected revenue impact if we don't do it.' Luna: That's the right framing. CEOs understand revenue risk. They don't always understand database locks. Lucas: Right. And that's part of what I want to talk about today — not just the technical side of load testing, but the organizational side. How do you run a test that actually surfaces production-like issues, and how do you communicate the findings so that the business acts on them? Luna: Before we dive deeper — and this is totally on brand — I have to say, if this conversation is giving you a concrete takeaway you'll actually use, that's exactly what listener support helps us keep doing. No ads, no sponsors, just us digging into real engineering stories. Lucas: Yeah, it's a small thing that makes a big difference. If today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Seriously, it helps us stay independent and keep episodes like this coming. Luna: Alright, back to load testing. Lucas, you mentioned the query plan changed because of data size. That's one classic pitfall. What are other common blind spots that load testing reveals? Lucas: One big one is external dependencies. Priya's test also revealed that their payment gateway had a rate limit they didn't know about. Under high concurrency, the gateway started returning 429 errors, and the checkout code didn't handle that gracefully — it just retried until timeout. Luna: So the system looked fine in isolation but broke when all the pieces talked to each other under load. That's the whole point of end to end testing. Lucas: Exactly. Another blind spot is session management. If you're using sticky sessions or a distributed cache, load testing can expose hotspots where certain nodes get hammered. In Priya's case, they found that their Redis cluster had an uneven key distribution because of a bad hash function. Luna: That sounds like something that would only show up at scale. How do you even design a load test to catch that? Lucas: You have to model realistic user behavior. Not just a constant stream of requests, but spikes, think time, abandoned carts, mobile vs desktop. Priya used real session logs from the previous year to build the test profile. That's the gold standard. Luna: So the test itself is a software project. You need to invest in it like one. Lucas: Absolutely. And the payoff is that you get to learn about your system's failure modes before your customers do. Priya's team also discovered that their auto-scaling policy was too slow — it took three minutes to spin up new instances, but the traffic spike hit in under thirty seconds. Luna: Three minutes is an eternity in load time. So they had to pre-warm instances or change the scaling metric. Lucas: They changed the metric. Switched from CPU utilization to request queue depth, which reacts much faster. That alone cut the scaling lag to about forty-five seconds. Luna: I want to push back a little though. Some engineers argue that load testing is overrated — that chaos engineering gives you more realistic failure modes because it tests actual production behavior, not synthetic scenarios. What do you think? Lucas: I think they serve different purposes. Chaos engineering tests how your system handles partial failures — a server goes down, a network partition happens. Load testing tests how it handles scale. You need both. Priya's team actually does chaos engineering too, but they run it in production, not staging, because they want real traffic patterns. Luna: So load testing is about capacity, chaos engineering is about resilience. They're complementary. Lucas: Exactly. And the best teams run both in a regular cadence. Priya now runs a load test every quarter, and a chaos experiment every sprint. It's part of their definition of done for any major feature. Luna: That sounds like a culture shift. How did she get the team on board? Lucas: She started small. First load test was just the checkout flow. After it caught the database lock, the team saw the value. Then she expanded it. The key was making it a blameless process — the test is not about finding who wrote bad code, it's about finding weaknesses in the system. Luna: That's crucial. If people feel like they'll be blamed for a bottleneck, they'll resist testing. Lucas: Right. And the other thing is that Priya made the results visible to the whole company. After every load test, she sends a one-page summary to the entire org: what we tested, what we found, what we fixed, and what the business impact would have been if we hadn't tested. That builds trust and shows engineering's value. Luna: That's a great practice. It also educates non-technical stakeholders about why engineering needs time for testing. Lucas: Exactly. And it creates a feedback loop — the next time a product manager wants to rush a feature, the team can point to the load test results and say, 'we need time to test this properly.' Luna: So what's the one thing you'd recommend to a CTO who's never done load testing but wants to start? Lucas: Pick the most critical user journey — the one that makes or breaks your business — and write a simple script that simulates twice your peak traffic. Run it against a staging environment that mirrors production as closely as possible. Don't worry about making it perfect. Just run it and see what breaks. That first run will teach you more than any architecture review. Luna: And if nothing breaks? Lucas: Then you probably didn't push hard enough. Increase the concurrency until something fails. The goal is to find the breaking point, not to prove the system is perfect. Luna: Good advice. Priya's story is a great reminder that some of the most valuable engineering work happens before there's a fire. Lucas: Exactly. And it's often the simplest fixes — a read replica, a cache, a faster scaling metric — that prevent the biggest disasters.