Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Testing Your API Rate Limits Prevents Production Meltdowns
Transcript
- Lucas: You've probably seen a 429 error at some point — 'Too Many Requests'. Most developers just shrug and retry. But when that 429 hits your API at scale during a peak event, it can cascade into a full production meltdown. Luna: I've definitely been on the receiving end of a rate limit, but I never thought about testing them. Isn't that just part of normal load testing? Lucas: That's exactly the common misconception. Load testing checks how your system handles high traffic. Rate-limit testing checks whether your system correctly enforces limits — and what happens when clients hit those limits. They're related but fundamentally different. Luna: Okay, so give me a concrete example where not testing rate limits blew up. Lucas: There's a fintech startup I looked into — they process payments for small businesses. Last Black Friday, they got hit with a massive spike in transaction requests. Their rate limiter was supposed to cap each merchant at 100 requests per second. But because they'd never tested the actual behavior of that limiter under load, they didn't realize the sliding window counter had a subtle bug. Luna: What kind of bug? Lucas: The limiter was using a Redis-backed sliding window. But when multiple requests came in at the exact same millisecond, the Redis transaction ordering caused the window to skip counting some requests. So a merchant could burst to 500 requests per second before the limiter kicked in. The API stayed up, but the downstream payment processor started timing out, orders got duplicated, and some customers were charged twice. Total cost: about $2 million in lost transactions and three days of firefighting. Luna: Ouch. So testing rate limits isn't just about verifying the limit — it's about testing the implementation of the limiter itself. Lucas: Exactly. Most teams test the happy path: send 100 requests, get 100 successes, then send one more, get a 429. But that doesn't test the reset behavior — does the Retry-After header match the actual window? What about burst allowance? Token bucket and leaky bucket algorithms behave very differently under sudden spikes. Luna: I want to get practical. If I'm a QA engineer or a developer on a team that hasn't tested rate limits, what's the first thing I should do? Lucas: Start by reading your API's rate limit documentation carefully. Identify the algorithm — is it token bucket, leaky bucket, fixed window, sliding window? Each one has different failure modes. For a token bucket, test the refill rate: after you hit the limit, does a new token appear after exactly the expected interval? We've seen bugs where tokens refill faster than documented, leading to soft rate-limit evasion. Luna: So you need to test the boundaries around the limit, not just the limit itself. Lucas: Right. Boundary testing is key. For a sliding window log, the edge case is the exact moment the window rolls over. You want to send bursts right at the boundary — 99 requests in one second, then 2 requests in the next. If the implementation uses an ordered set in Redis, the timestamp precision matters. We've seen bugs where timestamps are truncated to seconds, causing double-counting across windows. Luna: And what about distributed rate limiting? If you have multiple API servers, each with its own local counter, the limit per server might be 100, but globally you could get 300 if you have three servers. Lucas: That's a classic. Centralized rate limiting with a shared Redis cluster is the standard approach, but it introduces latency and consistency issues. You should test what happens when Redis is under load — does the limiter degrade gracefully? We've seen cases where Redis command timeouts cause the limiter to fail open, allowing unlimited requests. Luna: So you need chaos engineering for your rate limiter. That's a great angle. Lucas: Exactly. Inject latency into the Redis connection, simulate a node failure, and see if the API still enforces some reasonable limit or if it just gives up. And don't forget to test the 429 response format itself. Some clients parse the Retry-After header. If your header format changes between environments, clients might ignore it and retry immediately, making the problem worse. Luna: How do you convince your team to prioritize rate-limit testing when nothing has broken yet? I imagine the response is, 'The API is fine, we have load tests.' Lucas: You frame it as insurance. A single rate-limit bug during a peak event can cost more than the entire QA team's annual salary. You can also point to public postmortems — like the one from that fintech startup I mentioned, or even the 2020 AWS outage where a rate limiter misconfiguration took down a large chunk of the internet. Luna: That's a good point. Another angle is to start small: pick one critical API endpoint that has a documented limit, write a quick test for the exact boundary, and show the results. Even if the test passes, you've established a baseline. Lucas: And if it fails, you just saved your team a future incident. I'd also recommend testing the rate limiter in isolation — not through the full application stack. Use a dedicated test harness that sends requests at precise intervals and measures response times and headers. That way you can iterate quickly. Luna: Right. And then integrate that into your CI pipeline. Make sure every deployment validates that the rate limiter behaves as expected. Lucas: One more gotcha: testing rate limits in production can be tricky because you don't want to actually throttle real users. That's where dedicated test API keys with higher limits or a sandbox environment come in. But make sure the sandbox uses the exact same rate-limiting code and Redis instance as production — otherwise you're testing a different system. Luna: That's a trap I've seen before. Teams test in staging with a standalone Redis, then deploy to production with a clustered Redis, and the behavior changes. Lucas: This is one of those areas where a couple of dollars a month is genuinely what keeps these conversations going. If you've gotten something useful out of this, you can support the show at buy me a coffee dot com slash fexingo. It helps us stay ad-free and dig into these niche topics. Luna: Yeah, it really does make a difference. Every little bit adds up. Lucas: Now, let's get back to the practical side. Once you've got your rate-limiter tests in place, the next step is to test how your clients handle 429s. If your mobile app or web frontend doesn't respect Retry-After, you could end up with a thundering herd problem where clients retry simultaneously and overwhelm the API. Luna: That's a great point. You need end to end tests that simulate a 429 response and verify the client waits the appropriate time. Otherwise, your server-side fix doesn't matter. Lucas: Exactly. And don't forget the exponential backoff. If your API returns 429 with a Retry-After of 1 second, but your client implements exponential backoff starting at 1 second and doubling, the actual retry rate is very different from the server's expectation. That can lead to underutilization or accidental bursts. Luna: So you need to test the contract between client and server. That's where contract testing comes in — you know, we did a whole episode on that. Lucas: We did, episode 73. Rate-limit headers are a perfect candidate for contract testing. If your API promises a Retry-After header, your contract test should verify that header is present and parseable. And your client tests should mock a 429 with that header and ensure the client respects it. Luna: I want to bring this back to a concrete takeaway. If you're listening and you want to start testing rate limits tomorrow, what's the minimal first step? Lucas: Pick your most critical public API endpoint. Read its rate-limit documentation. Write a script that sends requests just below the limit, then exactly at the limit, then one over. Verify the 429 response includes the expected headers — especially Retry-After. Then repeat that test under load from multiple concurrent clients to see if the limit holds. That will catch 80 percent of common rate-limit bugs. Luna: And if it passes, you have a baseline. If it fails, you just prevented a potential outage. Lucas: Exactly. The cost of writing that test is a few hours. The cost of a rate-limit failure in production can be millions. This is one of those high-leverage testing activities that doesn't get enough attention. Luna: Well said. Thanks, Lucas. I think we've given listeners plenty to work with. Lucas: Yeah, I hope so. Until next time, keep your rate limits tested.