Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Rebuilt Its API for Idempotency at Scale
Transcript
- Lucas: So Stripe processes somewhere north of four hundred billion dollars in payment volume per year. Every single one of those transactions starts with an API call — and every one of those calls has to be safe to retry. Luna: Safe to retry — meaning if my internet drops and I hit the button twice, I don't get charged twice. Lucas: Exactly. That property is called idempotency, and it's one of those things engineers take for granted until it breaks. Stripe rebuilt their entire core API around it a few years ago, and the design choices they made are worth unpacking. Luna: And if you're a builder listening, this kind of deep-dive is exactly why we keep this show ad-free — listener support keeps it that way. If these conversations are useful for what you're building or running, you can throw a few dollars in the tip jar at buy me a coffee dot com slash fexingo. Lucas: Yeah, it genuinely helps. So let's get into the architecture. Stripe's old API had idempotency too, but it was bolted on — each endpoint had to implement its own logic. Luna: Which sounds like a recipe for inconsistency. Lucas: Right. One team might use a database unique constraint. Another might check a cache key. In a distributed system with thousands of services, that's fragile. So they centralized it: every write request now carries an idempotency key — a unique string the client generates. Luna: And the server uses that key to deduplicate. Lucas: Yes. But the devil is in the latency. If you have to check a central store for every request, your p99 goes through the roof. Stripe's solution was a custom transaction log built on top of FoundationDB — a distributed database that gives them strong consistency with low tail latency. Luna: Why FoundationDB specifically? Lucas: Because it provides exactly the guarantees they needed: ACID transactions across a distributed cluster with a predictable performance profile. They could model the idempotency check as a simple read then write within a transaction. If the key exists, return the cached response. If not, process and store. Luna: So the transaction log becomes the source of truth for 'has this request been handled?' Lucas: Exactly. And they tuned it to handle about fifty thousand writes per second at peak, with a p99 latency under ten milliseconds. That's not easy when you're also dealing with network partitions and node failures. Luna: What about the case where the client retries but the original request is still in-flight? Lucas: Great question. That's the classic distributed race condition. Stripe uses optimistic locking: the first request acquires a lock on that idempotency key, and any concurrent request with the same key either waits briefly or gets a 'conflict' response. They chose to have the client retry on conflict, because it's simpler than building a distributed lock manager. Luna: So the client needs to be smart about retries. Lucas: Yes, but the contract is straightforward: send the same key, get the same result. The server guarantees it, as long as the key is unique per operation. Stripe recommends UUIDs, but any high-entropy string works. Luna: One thing I've always wondered: do they store the full response in the idempotency cache, or just a status? Lucas: They store the full response. That way, a retry doesn't just avoid double-charging — it also returns the same data. If you're creating a customer object, the second call returns the same customer ID. That's crucial for clients that cache responses or need consistency across retries. Luna: But that must balloon the storage footprint. Lucas: It does, but they keep the cache relatively short-lived — typically twenty-four hours. After that, the key can be reused. They also compress the response bodies. And because FoundationDB handles the storage efficiently, the cost is manageable for the reliability win. Luna: Were there any edge cases that surprised them? Lucas: One that Stripe engineers have talked about publicly: what happens when the server crashes after processing the request but before storing the response? The next retry sees a missing key and tries again, resulting in a duplicate charge. To prevent that, they made the write to the transaction log part of the same transaction as the business logic — if the server dies mid-flight, both roll back. Luna: So the idempotency guarantee is only as strong as the transaction boundary. Lucas: Exactly. And getting that right required rethinking their entire request pipeline. They moved from a traditional request-response model to what they call a 'unified log' architecture — every write operation goes through that central log, which ensures ordering and idempotency before any business logic runs. Luna: That sounds like a massive refactor. Lucas: It was. It took them over two years. But the payoff is that today, Stripe can guarantee exactly-once semantics across their entire platform. That's table stakes for a payment processor, but very few companies actually achieve it at this scale. Luna: So what's the takeaway for a CTO listening who's not Stripe? Lucas: Idempotency is not something you bolt on after launch. It needs to be baked into your API design from day one — even if your traffic is a tiny fraction of Stripe's. Because once you have customers depending on your retry logic, changing the contract is painful. Luna: And the simplest version is just an idempotency key header, right? Lucas: Yes. Accept a key, store the response, return cached on repeat. You don't need FoundationDB — a simple key-value store with strong consistency works for most volumes. The principle is the same.