Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Uses Idempotency Keys to Prevent Double Charges
Transcript
- Lucas: A few years back, I was building a small payment flow for a side project. First time integrating with Stripe. I implemented the standard charge call, tested it, and it worked. Then I introduced a retry on network timeout. Suddenly I had two charges for the same customer. The customer was not happy. Luna: Classic distributed systems problem. at least once delivery without idempotency — you get duplicate side effects. Lucas: Exactly. And Stripe has a dead-simple solution for this: the Idempotency-Key header. You generate a unique key per request — usually a UUID — and pass it in. If Stripe receives the same key within a certain window, it won't process the charge again. It just returns the original response. Luna: So it's basically a deduplication layer on the server side. How long does Stripe keep those keys around? Lucas: They keep them for at least 24 hours. That's the window where the same key guarantees the same response. After that, the key expires and Stripe may process a new request with the same key. That's a critical detail — idempotency keys aren't permanent. They have a time to live. Luna: Makes sense. You don't want to store keys forever. But what about the first request that actually succeeded? You need to cache the response, not just the fact that the key was seen. Lucas: Right, and that's exactly what Stripe does. On the server side, when a request comes in with an idempotency key, Stripe looks up whether that key has been used before. If it has, and the previous request succeeded, it replays the response — including the exact same charge ID, status, everything. If the previous request failed, it retries the operation. So you get exactly-once semantics from the client's perspective. Luna: That's a powerful pattern. And it's not just for payments. I've seen idempotency keys used in order creation, inventory deduction, even email sending. Anywhere a duplicate could cause real damage. Lucas: Absolutely. Shopify, for example, uses idempotency keys in their REST API for creating orders. If you're building a checkout integration and your server crashes after sending the request but before receiving the response, you can retry with the same idempotency key and not worry about creating two orders for the same customer. Luna: But there's a subtlety: the idempotency key has to be tied to the request payload. If you send the same key with a different payload, what happens? Lucas: Stripe returns an error. The key is bound to the exact parameters of the first request. If you change the amount or the currency, Stripe rejects it with a 409 Conflict. That's a safety measure. The assumption is that if you're retrying, you want the exact same operation, not a different one. Luna: Good. That prevents subtle bugs where a developer accidentally modifies the payload and the server silently ignores it. Lucas: Exactly. Now, implementing idempotency keys on your own service isn't trivial. You need a data store that can enforce uniqueness under high concurrency. Most people use Redis with a SET NX command — set if not exists — and a TTL. The key becomes the idempotency key, the value is the serialized response. If SET NX returns success, you process the request and store the result. If it returns failure, you fetch the stored response and return it. Luna: What about race conditions? Say two requests with the same key arrive at the same time, both SET NX succeed because the Redis node hasn't replicated yet? Lucas: That's a real risk if you're using a single Redis instance or a cluster with asynchronous replication. Stripe uses a strongly consistent storage layer for idempotency — likely something like Cassandra with quorum reads and writes, or even a database with linearizable transactions. The key insight is that the idempotency check must be atomic and consistent. Otherwise you can still get duplicates. Luna: So the pattern is simple in concept but demands careful infrastructure underneath. Lucas: Right. And there's another trade-off: key expiration. If you set your TTL too short, the client might exhaust its retries after the key expires, leading to a duplicate. Too long, and you accumulate storage. Stripe's 24-hour window is a good heuristic for most payment flows, but for something like a subscription creation that might be retried over days, you'd need a longer window. Luna: Or you could let the client specify the key expiration, like some APIs do with a separate header. Lucas: That's an option. But it adds complexity. Simpler is to pick a window that covers your maximum expected retry duration. For most web APIs, 24 hours is more than enough. If a client hasn't got a response in 24 hours, something else is probably broken. Luna: Let's talk about the client side. The client has to generate a unique key per request. How do they ensure uniqueness across retries? Lucas: A UUID version 4 is the standard. It's random, collision-resistant, and easy to generate. The client generates one UUID per logical operation — say, one per checkout attempt — and sends it with every retry of that same request. If the user refreshes the page and starts a new checkout, the client generates a new UUID. Luna: But what if the client crashes and loses the UUID before getting a response? Then on recovery, it doesn't know the key, so it can't safely retry. Lucas: That's a real problem. The solution is to persist the idempotency key in local storage or a database before sending the request. Some APIs allow the client to generate a deterministic key based on request parameters, like a hash of the payload plus a client-generated nonce. That way, even if the key is lost, the client can reconstruct it. But that's more complex. Luna: Deterministic keys have their own issues. If you hash the payload, and the payload includes a timestamp or a sequence number, you have to ensure those are stable across retries. Lucas: Right. That's why most developers start with random UUIDs and accept the crash-recovery gap. For many applications, the risk of a duplicate after a client crash is low, and the cost of implementing a robust recovery mechanism outweighs the benefit. Luna: Stripe's documentation actually recommends using idempotency keys for all POST requests. They even have a pattern where you can pass an idempotency key in the header without a value, and Stripe generates one for you and returns it in the response. That's useful for clients that don't want to manage keys themselves. Lucas: I didn't know that. That's clever. It offloads the responsibility to Stripe, but the client still gets a key back that they can use for retries. Luna: Exactly. And it works because Stripe can generate a key server-side and associate it with the request. The client just has to remember that key for retries. Lucas: That's a nice developer experience. It makes idempotency almost frictionless. Luna: Alright, so we've covered the basics. But I want to go deeper into the distributed systems theory. Idempotency keys are an implementation of the 'idempotent receiver' pattern. They ensure that an operation can be applied multiple times without changing the result beyond the first application. But they're not a silver bullet. Lucas: No, they're not. For example, idempotency keys don't help with concurrent updates to the same resource. If two different clients send two different idempotent requests to update the same record, you still need concurrency control — like optimistic locking or a compare and swap pattern. Luna: Right. Idempotency is about retries of the same operation, not about ordering of different operations. Lucas: And there's a philosophical point: idempotency keys push the complexity to the client. The client has to generate and manage keys. Some argue that the server should be responsible for deduplication based on request content alone. Luna: But that's much harder. How would the server know if two identical requests are duplicates or intentional? Without a client-generated key, you'd have to use a combination of fields that uniquely identify the operation, like an external order ID. But then you're coupling your idempotency logic to your business logic. Lucas: Exactly. That's why the idempotency key header is such a clean abstraction. It separates the concern of exactly-once execution from the business semantics. The server just needs a key-value store with atomic operations. The client decides what constitutes a unique operation. Luna: Let's talk about a real-world failure I saw. A team at a previous company implemented idempotency keys on their order creation API. They used a database table with a unique constraint on the key column. But they forgot to set a TTL. Over time, the table grew to millions of rows, and inserts became slow because of the index. Eventually, the API latency increased, and they had to clean up old keys manually. Lucas: That's a classic operational pitfall. Idempotency keys need a cleanup strategy. If you're using a database, you should partition the table by creation time or use Redis with a TTL. Alternatively, you can store keys in a time-series database and query only recent ones. Luna: Or you can use a bloom filter for cheap existence checks, but then you lose the ability to replay responses. That might be acceptable if you only care about preventing duplicates, not about returning the same response. Lucas: Right. That's a trade-off: storage cost versus functionality. Stripe needs to replay responses, so they store the full response. But for a simple deduplication layer, a bloom filter could work. Luna: One more thing: idempotency keys are not just for HTTP APIs. They're used in message queues too. For example, when a consumer processes a message from a queue with at least once delivery, it can use a message ID as an idempotency key to ensure it processes the message only once. Lucas: Good point. That's the same pattern applied at a different layer. In fact, many message brokers like AWS SQS provide a message deduplication ID that works exactly like an idempotency key. Luna: So whether you're building a REST API, a gRPC service, or a message-driven system, the idempotency key pattern is a fundamental tool in your distributed systems toolbox. Lucas: Absolutely. And the beauty is that it's simple to understand, but implementing it correctly requires attention to detail — consistency, expiration, client-side management. It's one of those patterns that separates novice from experienced distributed systems engineers. Luna: Speaking of patterns that make a real difference in day-to-day building, a lot of the conversations we have on this show are exactly the kind of thing that helps engineers and leaders think more clearly about their systems. And if you've gotten value from these episodes, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo. It's how we stay ad-free and focused on the stuff that matters to builders. Lucas: Yeah, it's a small thing that adds up. We appreciate everyone who chips in. Now, back to idempotency — I want to talk about a pattern that goes beyond simple key-value storage: idempotency with conditional logic. Luna: What do you mean by conditional? Lucas: Imagine an API endpoint that creates a subscription. The first request creates it and sends a confirmation email. A retry with the same idempotency key comes in. The server sees the key exists and replays the response. But should it also prevent the email from being sent again? That's a different concern — the email sending is a side effect that should also be idempotent. But the idempotency key alone doesn't guarantee that side effects are handled. Luna: Right, because the side effect happens outside the transaction. The server might have committed the subscription but the email sending failed. The next request with the same key replays the response, but the email might still be missing. Lucas: Exactly. So you need to make the side effect part of the idempotent operation. Either you make the email sending transactional with the database update — using something like an outbox pattern — or you store the fact that the email was sent alongside the response, and replay that too. Luna: That's where the idempotency key becomes part of a larger saga pattern. The key is just one piece. Lucas: Right. And that's why I think idempotency keys are a gateway concept. Once you understand them, you start thinking about exactly-once semantics across your entire system, not just at the API boundary. Luna: It's a good mental model to have. I've seen teams reduce entire classes of bugs just by adding idempotency keys to their critical write paths. Lucas: And the best part is, it's a pattern you can introduce incrementally. You don't have to redesign your whole system. Start with the most sensitive operations — payments, orders, account changes — and add keys. Then extend to the rest. Luna: That's a good practical takeaway. If you're listening and you haven't done this yet, pick one endpoint that you're afraid of duplicating and add an idempotency key header. It'll save you a weekend of firefighting. Lucas: Couldn't agree more. And with that, I think we've given everyone enough to go and build something more robust. Thanks for listening.