Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Timeouts and Retries Without Flaky Tests
Transcript
- Lucas: So, Luna, you remember that payment service we've talked about before, the one that kept timing out during peak traffic? Luna: Oh yeah, the one where users would hit 'pay' and then just stare at a spinner for thirty seconds? Lucas: Exactly. And the fix seemed simple: add a timeout and a retry. But the real nightmare started when we tried to test it. Every test either passed when it shouldn't have, or failed randomly. We spent more time debugging the tests than the actual service. Luna: So today we're talking about how to test timeouts and retries without making your test suite flaky. Lucas: Right. And this is a topic that doesn't get enough attention. Most teams just write a test that waits for the timeout to trigger, and they end up with tests that take forever or fail on a slow CI runner. Luna: What's the fundamental challenge? I mean, time is inherently non-deterministic in tests. Lucas: That's exactly it. When you write a test that says 'wait for 5 seconds and assert that the request fails,' you're betting that the CI machine isn't slow that day. But if the machine spikes, your test fails for the wrong reason. Luna: So how do you make time deterministic? Lucas: The standard answer is a virtual clock. Instead of using the real system clock, you inject a fake clock that you can advance manually in your test. That way, you can simulate a timeout in milliseconds instead of waiting the full duration. Luna: Okay, so you're mocking the time. But what about the actual network calls? You still need to simulate a slow or unresponsive server. Lucas: That's where fault injection comes in. You can use a mock server that you control, and you program it to delay its response or return an error. Combine that with your virtual clock, and you have full control over both the timing and the failure. Luna: So you can test the 'timeout' path and the 'retry then success' path without any real sleeping. Lucas: Precisely. And that's the key to avoiding flakiness. But there's a trap: you have to be careful not to test implementation details. You want to test the behavior, not 'did we call this mock 3 times?' Luna: What do you mean by that? Give me an example. Lucas: Suppose you have a retry policy that says 'try 3 times with exponential backoff.' A naive test might assert that the mock was called 3 times and that the wait between calls was exactly 200 milliseconds. But that's testing the implementation, not the outcome. Luna: Right, because you could change the backoff algorithm and your test would break even though the system still works correctly. Lucas: Exactly. What you want to assert is: 'given a transient failure, the client eventually succeeds' or 'given a persistent failure, the client gives up and returns an error.' Those are behavioral assertions. Luna: So how do you handle the backoff timing in a behavioral test? Lucas: You still use a virtual clock, but you don't assert the exact wait time. Instead, you advance the clock in steps and verify that the retry happens after some delay, not immediately. The exact duration is an implementation detail. Luna: Makes sense. But what about testing the timeout value itself? Like, how do you know 30 seconds is the right threshold? Lucas: That's a great question, and it's more about performance testing than unit testing. You need to measure the actual latency distribution of your dependencies in production, then set the timeout to something like the 99th percentile plus a buffer. Luna: So you're saying the timeout shouldn't be a guess. Lucas: Right. And that's where a lot of teams mess up. They set timeouts based on intuition, and then they either fail fast on slow days or wait forever on dead services. You want the timeout to be long enough to allow for normal jitter, but short enough that failures are detected quickly. Luna: And retries? How many should you do, and what backoff strategy is best? Lucas: There's no one-size-fits-all, but a common pattern is 3 attempts with exponential backoff and jitter. The jitter is important because if all your clients retry at the same time, you create a thundering herd. Luna: So tests should also verify that the system doesn't retry too aggressively in a distributed scenario. Lucas: Exactly. That's where integration tests with a mock server can help. You can simulate a temporary outage and verify that the system's retries are spread out, not synchronized. Luna: But wait, doesn't adding jitter make tests more non-deterministic? Lucas: It can, if you're relying on real time. But again, with a virtual clock, you can control the jitter's randomness or even make it deterministic in tests by injecting a fixed seed. Luna: So the mantra is: virtual clocks for time, fault injection for failures, and behavioral assertions for outcomes. Lucas: You nailed it. And that's the core of what we want listeners to take away. If you're testing timeouts and retries, you have to take control of time and failure, not leave them to chance. Luna: Before we go deeper into anti-patterns, I want to mention something. We're listener-funded, and that's what keeps this show going. A number of you chip in a few dollars a month at buy me a coffee dot com slash fexingo, and that genuinely covers our costs. Lucas: Yeah, it's a small but mighty group. And it means we don't have to run ads or chase sponsors. So if you're getting value from these conversations, that's the way to support us. Luna: No pressure, just something to keep in mind. Now, let's talk about some common anti-patterns in timeout testing. Lucas: One of the worst is using real sleeps in tests. Some people write 'Thread.sleep' to wait for a timeout, and that instantly makes your suite slow and flaky. Luna: And if the timeout is 30 seconds, that's a 30-second test. No one wants that in their CI pipeline. Lucas: Another anti-pattern is 'testing the error path by expecting a specific exception type that's actually thrown by the mock, not by your code.' That's testing the mock, not your system. Luna: How do you avoid that? Lucas: You focus on the observable behavior. For example, if the system is supposed to return a 503 after three failed attempts, you assert that the response is a 503. You don't care whether it threw a TimeoutException or a NetworkException internally. Luna: So behavioral testing again. What about testing the retry logic when the first attempt succeeds? Should you have a test for that? Lucas: Absolutely. That's the happy path, and it's often overlooked. You want to verify that if the first call succeeds, there's no unnecessary retry. That's part of the contract. Luna: And what about idempotency? If a retry happens, the duplicate request shouldn't cause double charges. That's a business requirement. Lucas: Exactly. And that's a perfect segue to our payment service example. We had to ensure that retries were safe, meaning the same payment wouldn't be processed twice. We tested that by simulating a network failure after the server committed the payment but before the client got the response. Luna: So the retry sends the same request with the same idempotency key, and the server dedupes it. Lucas: Right. And that test is critical. If you don't test that, you're one retry away from charging a customer twice. Luna: So, to recap: virtual clocks, fault injection, behavioral assertions, and testing idempotency. Anything else? Lucas: One more thing: don't forget to test the 'no retry' scenarios. For example, if the server returns a 4xx error like 400, you should not retry. Your retry logic should only kick in for 5xx or network errors. Luna: That's a common mistake, retrying on all errors. Good point. Lucas: So the next time you're adding timeouts and retries to a service, think about how you're going to test them. If you're not using a virtual clock and fault injection, you're probably in for a world of flaky tests. Luna: And if you're a listener facing a stubborn flaky test, what's the first thing you should do? Lucas: Ask yourself: is this test controlling time and failure, or is it at the mercy of the environment? If it's the latter, start refactoring toward deterministic testing.