Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Uses Idempotency Keys to Prevent Double Charges
Transcript
- Lucas: If these conversations are useful for what you're building or running, that's basically the whole point of this show. And today I want to talk about something that sounds like a tiny technical detail but actually sits at the heart of how Stripe keeps its payment infrastructure trustworthy. Luna: Idempotency keys. I've seen that term thrown around, but honestly, I don't think most engineers outside payments really understand why Stripe made them mandatory. Lucas: Exactly. And the reason is simple: payments are irreversible. If Stripe double-charges a customer, that's not just a bug — it's a trust-breaking event. So they designed an API where every write request can include an idempotency key — a unique string, usually a UUID, that the client generates before sending the request. Luna: And Stripe stores that key on its side, right? So if the client retries for any reason — network timeout, ambiguous response — Stripe sees the same key and says 'I already processed this, here's the same result'. Lucas: Exactly right. Let me walk through a concrete example. Imagine you're a customer on an e-commerce site that uses Stripe for payments. You hit 'Place Order' and your card is declined. You hit it again — this time the card goes through. But what if the first request actually succeeded on Stripe's end, and the decline message was just a network glitch on the way back? Luna: Without idempotency, you'd be charged twice. And the merchant would have to issue a refund, which costs them processing fees and customer goodwill. Lucas: Right. With idempotency, the merchant's backend generates an idempotency key — say, a hash of the order ID plus a timestamp — before even sending the first request. Stripe's API receives it, checks its idempotency store — which is backed by Redis — and if that key doesn't exist, it processes the payment and stores the response. The second time, even if the amount is identical, Stripe sees the same key and returns the cached response without processing anything. Luna: How long does Stripe keep those keys around? Because you can't store them forever — that's infinite storage. Lucas: Stripe uses a 24-hour time to live. After that, the key expires and you'd need to generate a new one for a new attempt. That's a reasonable window for most retry scenarios — network blips, browser refreshes, even overnight retries. Luna: So what happens if Redis goes down? No idempotency store means you lose the safety net. Lucas: That's the million-dollar question. Stripe runs Redis with replication — a primary and at least one replica. Writes go to the primary, reads can go to the replica. If the primary fails, they fail over to the replica. But there's still a tiny window where a write might not have replicated before the primary dies. In that case, they accept the risk of a duplicate charge, which is rare but not zero. They've chosen availability over perfect consistency. Luna: And they're okay with that trade-off because duplicate charges are something they can detect and reverse with automated refunds within seconds. Lucas: Exactly. They have a separate system that scans for duplicate payments using a combination of amount, merchant, and timestamp within a short window. So idempotency is their first line of defense, but not their only one. Luna: Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Lucas: Yeah, listener support is literally why this show stays ad-free and focused on the deep engineering stuff. Appreciate it. Luna: Alright, back to idempotency. So Stripe's approach is a common pattern — but I've heard some engineers say idempotency keys are overkill for simpler systems. Where's the line? Lucas: Good question. The rule of thumb I use: if a request can result in a real-world side effect that matters — a charge, a booking, a shipment — you need idempotency. If it's just a read or a stateless operation, skip it. But the tricky part is that even something like 'update user email' can be non-idempotent if the system logs every change. So think about the business impact. Luna: What about race conditions? Two clients sending the same idempotency key at the exact same moment. Lucas: Stripe uses a conditional write to Redis — something like 'SET if Not eXists' — which is atomic. So only one of those requests wins; the other gets back 'key already exists' and then reads the cached response. Race condition handled. Luna: That's elegant. So the key insight is not just the pattern itself, but the fact that Stripe built their entire API contract around it — every POST endpoint requires an idempotency key header. Lucas: Right. And that's the part that really matters for engineering teams. Idempotency isn't just a Redis hack — it's a design philosophy. It forces you to think about every request as potentially retried, which in turn influences how you write your database transactions, how you structure your error responses, and how you communicate with clients. Luna: I've seen teams that add idempotency as an afterthought — they end up with a messy dedup table that doesn't cover all retry scenarios. What's the right way to design it from the start? Lucas: Start with the client-side responsibility. The idempotency key should be generated by the client, not the server. Because the server can't distinguish between a duplicate request and a completely new one without some external identifier. So the merchant's backend generates a key — usually a version 4 UUID or a hash of the request payload plus a timestamp. Then the server stores it and uses it for exactly 24 hours. Luna: And the server needs to return the same idempotency key in the response so the client can correlate. That's part of the contract. Lucas: Exactly. And there's a subtlety: the client should never reuse an idempotency key for a different request. If the first request was for a $20 charge, and the client retries with the same key but a $30 amount, Stripe will reject it. The key is tied to the request payload. Luna: That prevents accidental or malicious mutation. But what about the case where the client doesn't get a response at all — say a TCP timeout. The client doesn't know if the request was processed. So it retries with the same key. That's safe. Lucas: Perfectly safe. And that's the beauty: idempotency turns a 'maybe' into a 'definitely'. The client can retry as many times as it wants, and the worst case is a duplicate charge that Stripe's secondary system catches. But in practice, it's extremely reliable. Luna: Let's talk about scale. Stripe handles billions of API requests per day. How does Redis keep up? Lucas: They shard the idempotency store across multiple Redis instances by key hash. Each key lives on exactly one node, so lookups are fast. They also use a separate Redis cluster for idempotency versus other caches, so a spike in one doesn't affect the other. And they monitor hit rates and TTLs aggressively. Luna: So if you're building a payments system today, would you recommend copying Stripe's approach exactly? Lucas: I'd say: take the pattern, but adjust the TTL and storage based on your retry behavior. If your clients retry within minutes, a one-hour TTL might be enough. If you have long-running workflows that retry over days, you might need longer. But the core idea — client-generated key, atomic check-and-set, cached response — is universal. Luna: And one more thing: make sure your database transactions are idempotent too. If the idempotency key check says 'not processed', but then the transaction fails halfway through, you need to either roll back or mark the key as failed. Lucas: That's a great point. Stripe's approach is to write the idempotency key and the payment status in the same database transaction. If the payment fails, the key is stored with a 'failed' status, so a retry will see that and not attempt again with a different outcome. Otherwise, you could end up with a key that's marked 'processing' but no actual payment. Luna: So it's really about designing the entire flow — from client to API to database to cache — as one idempotent unit. Lucas: Exactly. And that's why idempotency is such a rich topic for a leadership show. It's not just a technical detail — it's a systems design decision that affects reliability, customer trust, and operational complexity. The best teams bake it in from day one. Luna: I think the takeaway for any CTO listening is: if your system touches money, bookings, or any irreversible action, go read Stripe's API docs on idempotency. It's one of the best-documented patterns out there. Lucas: And then ask yourself: what's the worst thing that could happen if a request retries? If the answer is 'a duplicate charge', you need idempotency. If it's 'nothing', you're probably fine. But it's worth thinking about.