Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Idempotency Testing Prevents Double Payments
Transcript
- Lucas: Luna, have you ever hit the 'submit payment' button twice because the page was loading, and then you ended up with two charges? Luna: Oh, absolutely. It's that moment of panic when you see the spinner hang and you can't resist clicking again. Then the confirmation email comes twice. Lucas: Right. That's the user experience side of a problem that keeps distributed systems engineers up at night: idempotency. An operation is idempotent if performing it multiple times produces the same result as doing it once. Every payment gateway, every order placement system, every database write that can be retried needs to be idempotent. Lucas: And I'll say this — if today's conversation is giving you a practical takeaway on how to prevent those double charges, the way these shows stay ad-free is through listener support. If that's been useful, you can keep it going at buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to make sure we keep digging into real-world testing problems. No ads, no sponsors, just solid techniques. So, Lucas, how do you actually test that an endpoint is idempotent? Lucas: Great question. The core approach is to send the same request twice — with the same idempotency key — and verify that the system's state changed only on the first call. For example, Stripe popularised the idea of clients generating a unique key for each request. The server checks if it's already processed a request with that key, and if so, returns the stored result instead of executing again. Luna: So the key is the linchpin. What if the client doesn't generate one properly? Or if the key is weak — like a timestamp? Lucas: Exactly. That's a common failure. A weak key like a timestamp can collide. Stripe's keys are version-four UUIDs. In testing, you need to validate that the server rejects duplicate keys from the same client, and also that different keys create separate transactions. And you have to test retries: what if the server crashes after processing but before returning a response? Luna: That's the nightmare scenario. The client times out, retries with the same key, but the first request actually went through. The server needs to recognize the key and return the previous result. Lucas: Right. The system must guarantee that exactly one write occurs for each unique key. To test that, you can simulate network failures. For instance, use a test harness that sends a request, drops the connection before getting a response, then sends the exact same request again. Then verify the final state — only one order, one charge. Luna: That's a great technique. But it's not just payments. Idempotency matters for any resource creation — think of a ticket booking system or a database insert. Have you seen real-world bugs from missing idempotency? Lucas: Absolutely. There's a famous one from a major airline's booking system. Due to a network blip, a customer's seat reservation was created twice — same name, same flight, different confirmation numbers. The seating system then showed two bookings for one person, and later when they cancelled one, they cancelled the wrong one. Chaos. Luna: Ouch. And how could they have caught that in testing? By specifically testing retry scenarios. Lucas: Exactly. A comprehensive idempotency test suite would include: first, a clean call with a unique key — expect success. Second, a duplicate call with the same key — expect the same response, no state change. Third, a call with an invalid key — should be rejected. Fourth, a call after a server restart — the idempotency store must survive reboots. Luna: What about caching? If the server caches the response and then a new request comes in with a different key but identical payload, it might incorrectly return the cached result. Lucas: That's a subtle bug. The idempotency check must happen before any caching layer. In testing, you can craft two requests with different keys but identical payloads and ensure the server processes both independently. Also, you need to test for concurrent duplicate requests — two requests arriving at nearly the same time with the same key. Luna: Race condition. One of the hardest to test. How do you handle that? Lucas: You can use a distributed lock or a database unique constraint on the key column. For testing, you'd fire two parallel requests with the same key and verify that only one succeeds — the other gets a 409 or 200 with the stored result. The tricky part is ensuring your test runner can actually send requests concurrently. Tools like Locust or a simple Python thread pool can do it. Luna: Let's talk about the idempotency store. Where do you store these keys? Lucas: Typically a high-performance, available key-value store. Redis is common, but you need to make sure it's persistent — or at least that the key expires after a reasonable window, like 24 hours. In tests, you should verify key expiration behavior: after the window, the same key should be treated as new and allow a fresh operation. Luna: So the tests need to cover both the happy path and these edge cases. Any tooling recommendations? Lucas: Many teams write custom integration tests using their existing test framework — JUnit, pytest, or similar. But there's also a tool called Idempotency Tester that generates duplicate requests and verifies state. It's open-source. Also, some API testing tools like Postman or Insomnia allow you to send requests with custom headers and manually re-execute them, but that's not scalable. Luna: What about fuzzing? Could you use a fuzzer to send random mutations of idempotency keys? Lucas: Interesting idea. You could fuzz the key field to ensure the server handles malformed keys gracefully — like null, empty string, or excessively long keys. But the deterministic duplicate tests are more critical. The key insight is this: idempotency is a contract between client and server. Your tests must prove the server upholds that contract under retries, failures, and concurrency. Luna: And if you don't test it, your customers will discover it for you — usually through a double charge. Lucas: Exactly. The cost of one production incident can dwarf the cost of writing these tests. So start with a simple test: send the same request twice. If you never built idempotency, that test will fail spectacularly — and then you know what to fix. Luna: Great advice. Thanks, Lucas. I'm going to add an idempotency test suite to our next sprint. Lucas: You'll sleep better. And your customers' wallets will thank you.