Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test API Rate Limiting Without Angering Users
Transcript
- Lucas: So you've just built a shiny new API rate limiter, you've tested it with a script that fires off a hundred requests a second, and everything looks perfect. Then you roll it out, and suddenly your most loyal users are getting locked out for no apparent reason. Luna: That's the classic rate limiter false positive. And it's one of the most frustrating bugs to chase because the limiter is technically doing exactly what it's supposed to do. Lucas: Right. And that's why I wanted to dedicate an episode to testing rate limiting more thoughtfully. Because the usual approach — hammering the endpoint and checking that you get a 429 — misses all the interesting failures. The ones that happen when your rate limiter interacts with everything else in the system. Luna: Before we dive into those failures — and I know we've got some good war stories coming — a quick note. You're listening to Fexingo, and if you've gotten something useful out of these episodes, a couple of dollars a month is genuinely what keeps these going. You can find us at buy me a coffee dot com slash fexingo. No pressure, just a little help if you can. Lucas: Yeah, and honestly, every little bit — it's what keeps our servers running and our coffee mugs full while we research these deep dives. Luna: Exactly. Okay, back to rate limiters. So, Lucas, what's the first thing we're getting wrong in our test approach? Lucas: The first thing is that we test the limiter in isolation, with a clean slate, and we never test it against the real traffic patterns that your users actually produce. One of the most eye-opening exercises is to pull a day of real production traffic — anonymized, of course — and replay it against your rate limiter in a staging environment. You'll see patterns you never imagined: users who retry exactly at the retry-after interval, mobile clients whose clocks are skewed by five minutes, and bursts that are way spikier than any load test I've ever written. Luna: So you're saying our test scripts are too polite. They send requests like a well-behaved bot, but real users are messy. Lucas: Messy is the word. And messy is where the bugs live. Let me give you a concrete example. There was a ride-sharing app a few years back — I won't name them — that introduced a token-bucket rate limiter on their driver location update endpoint. On a busy Friday evening, during a surge, the limiter started rejecting a small percentage of requests that were actually within the limit. The problem? They were using Redis to store the bucket state, and when Redis was under heavy load, it would occasionally return a stale value — a value that hadn't been updated with the latest refill. So the limiter thought the bucket was empty when it actually had tokens left. Luna: Oh, that classic distributed systems problem — the read your write consistency issue. So the limiter was making decisions based on stale data. Lucas: Exactly. And their tests never caught it because they always tested against a single instance of Redis with no latency and no concurrent writes. When they replayed production traffic, they saw the rejection rate spike, and that forced them to dig into the Redis logs. The fix wasn't just about the limiter logic — it was about how they read and wrote the bucket state. Luna: That's a great example of why you need to test the whole system, not just the unit. So what's another layer we're missing? Lucas: The retry behavior of your own SDK. A lot of companies have their own API client libraries, and those libraries have retry logic built in. You can test the rate limiter all day long, but if your SDK doesn't honor the retry-after header correctly, you're going to cause a retry storm. I've seen cases where the SDK retries immediately, ten times, without waiting, which is even worse than the original request. Luna: And that's not just a bug in the SDK — it amplifies the load on your servers, and then your rate limiter starts rejecting even legitimate requests because the influx is so high. Lucas: Right. So the test here is to simulate a 429 response with a retry-after header, and then verify that your SDK actually waits that long before trying again. And you also need to test the edge case where the retry-after header is malformed — what does your SDK do then? Does it panic? Does it default to a fixed wait? Luna: That's a great point. I've also seen applications that ignore the header entirely and just retry immediately, and you can catch that with a simple integration test. Lucas: And here's another one that I think is underappreciated: client clock skew. Let's say you're using a sliding window log or a fixed window based on the client's timestamp. If the client's clock is off by even a few minutes, the server thinks they're in a different window, and you get false positives or false negatives. I once saw a mobile app that had a bug where the device clock was set to 1970 — the classic epoch problem — and every request got rate-limited because the timestamps looked like they were from the past. Luna: Oh, that's a nightmare. So how do you test for that? You'd need to simulate a client with a skewed clock. Lucas: Exactly. You can write a test that sets the client's system clock to a value that's off by, say, five minutes, and then make a request that should be within the limit. And you should also test the opposite: what if the client's clock is in the future? A lot of rate limiters will just reject those requests because they think the window hasn't started yet. That's a real user-facing problem, especially after daylight saving time changes. Luna: I never would have thought of that. So we're testing for the actual user experience, not just the happy path. Lucas: Right. And the last piece I want to hit on is the failover scenario. Most rate limiters are backed by Redis, and if that Redis instance goes down, what happens? Does your entire API become unavailable? Or does the limiter fail open and let everyone through? Both are bad in different ways. Failing open could let a malicious user overwhelm your system; failing closed could take down your service for everyone. You need to test both scenarios. Luna: So how do you test a Redis failover in a controlled way? That's where chaos engineering comes in, right? Lucas: Exactly. You can use something like Chaos Monkey or just manually kill the Redis node in your test environment. But the key is to verify that your fallback limiter — maybe a simple in-memory limiter — kicks in and starts counting, and that it's not so restrictive that it blocks all your users. We tested one system where the fallback was a basic fixed window limiter with a very low threshold, and it ended up blocking legitimate requests because it wasn't shared across instances — each instance had its own counter, so the total allowed was way higher than intended. Luna: So you think you're being safe with a fallback, but you're actually creating a whole new problem. That's a subtle bug. Lucas: Yeah, and these are the kinds of things that only show up when you're testing the system under real-world conditions. So the takeaway here is: when you're testing rate limiting, don't just test the limiter in isolation. Test the whole system — the Redis interaction, the client SDK, the clock, the failover — and especially replay real traffic patterns. Your users will be a lot happier when that Friday night surge doesn't turn into a five-hour outage. Luna: And that's exactly what we want to avoid. Well, thanks for walking us through all that, Lucas. I'm definitely going to look at our rate limiting tests with fresh eyes.