Latest / The API Podcast with Fexingo: REST, GraphQL, and Modern Web APIs / How API Idempotency Keys Prevent Double Charges
Transcript
- Lucas: Let's talk about the most expensive button on the internet: the pay now button. It seems simple enough, you click it, money moves from your account to the merchant's. But if the network lags, or your finger slips, or the UI just fails to update visually, that same click can register twice. And when it does, you have a problem that isn't just annoying for the customer but potentially disastrous for the business. Luna: Right, because the first transaction went through, but the second one triggers a new charge. Most people think this is a bug in their code, but it's usually just human nature meeting poor interface feedback. It happens all the time. Lucas: Exactly. And for years, developers tried to solve this with client-side fixes like disabling the button after the first click. That works until the user has a second device, or uses a script, or simply refreshes the page. The server still sees two identical requests. The real solution lives in what we call idempotency keys, a concept that sounds dry but is actually the bedrock of reliable digital commerce today. Luna: So instead of relying on the user not clicking twice, you give the system a way to recognize that these two clicks are actually the same attempt. It is like giving each request a unique fingerprint so the server knows not to process the second one as a new event. Lucas: Spot on. An idempotency key is essentially a unique string generated by the client, attached to every request. Think of it like a serial number stamped on a package before it leaves the warehouse. If the delivery driver tries to drop off the same package again, the recipient checks the serial number, sees it already arrived, and sends the second one back. In API terms, the server stores that key along with the result of the first successful transaction. Any subsequent request with that same key gets the original response, not a new action. Luna: It is clever because it shifts the burden of proof from the client to the server. You do not need to trust the user to only click once, you just need to trust the uniqueness of that key. Does this require massive database changes? Lucas: Not really, provided you structure your storage correctly. The key piece of data is the combination of the idempotency key and the HTTP method. A GET request with a specific key is different from a POST request with the same key. So you index your storage by that pair. When a request comes in, the server checks for that pair. If it exists, it returns the cached response immediately. If it does not, it processes the request, saves the result against that key, and then responds. Luna: And that cached response needs to be robust. You cannot just store the status code, you have to store the entire outcome. Otherwise, if the first request was a fifty internal server error, you might accidentally suppress a retry that should have succeeded. Lucas: That is a critical distinction. You have to store the full response body and status code. But there is a timing nuance here. What happens if the request is still processing when the second click comes in? The key has not been saved yet because the work is ongoing. If you return an error or block the second request during processing, you create a deadlock where the user is stuck waiting for a response that never comes. Luna: Yeah, so you need a middle ground. Maybe a pending state in the database? Or perhaps you let the second request wait briefly for the first one to finish. It feels like a concurrency headache. Lucas: It is a classic concurrency challenge. The standard approach is to lock the key while the request is processing. If a second request arrives with the same key, the server puts it in a queue or waits for the lock to release. Once the first request finishes, the lock is released, the second request wakes up, sees the key is resolved, and gets the cached result. It adds latency, but it ensures consistency. Without it, you risk race conditions where two threads both decide the key doesn't exist and both process the charge. Luna: That makes sense. It is basically a distributed version of checking if you already paid for that coffee. I am surprised more small businesses don't implement this manually. It seems like a basic hygiene factor for any API handling sensitive data. Lucas: It should be, but many teams skip it because they assume their infrastructure handles retries automatically. They rely on load balancers or reverse proxies to deduplicate traffic. Those tools are good for health checks, but they are terrible for business logic. A proxy might see two identical packets and drop one, but it doesn't know if those packets represent a legitimate retry or a new intent. Only the application layer understands the context of the transaction. Luna: So the application has to own this responsibility. It cannot outsource it to the network layer. That means every endpoint that changes state needs this logic built into its controller or service layer. It is tedious, but necessary. Lucas: It is tedious, yes. But the cost of getting it wrong is far higher. Let's look at a concrete example. In September twenty twenty six, we saw several fintech startups facing regulatory scrutiny because their systems processed duplicate refunds. Not because they were malicious, but because their webhook handlers lacked proper idempotency. When the payment provider sent the same success notification twice, the app issued two refunds. That is a direct loss of capital. Luna: Wow, double refunds. That is an easy way to bleed cash overnight. Did those companies recover, or did the damage go beyond the financial hit? Lucas: The financial hit was immediate, but the reputational damage was worse. Customers lost trust because their accounts showed erratic balances. Regulators stepped in because the audit trails were messy. You cannot explain away duplicate transactions as a feature. Once you have an idempotency key strategy, though, these problems vanish. The audit trail becomes clean because every transaction is tied to a unique key. If something goes wrong, you can trace exactly which key triggered the action and why. Luna: It turns a debugging nightmare into a straightforward lookup. Instead of guessing which request caused the issue, you just query the key. It brings order to chaos. How long does the server keep these keys? You cannot store them forever. Lucas: Good point. Storing every idempotency key indefinitely creates bloat. The standard practice is to set a TTL, or time to live, on these records. Usually around twenty four hours. Why twenty four hours? Because that covers the typical window where a client might retry a request due to timeout. If a request hasn't responded within twenty four hours, it is likely lost forever, and storing the key provides no value. After that period, the record can be safely deleted. Luna: Twenty four hours is a reasonable sweet spot. Long enough to catch retries, short enough to avoid filling up your database with stale data. It feels like a very practical constraint. Do you recommend generating the key on the client or the server? Lucas: Client side generation is the norm. The client creates a UUID, usually v4, which guarantees randomness and uniqueness. This prevents the server from having to generate IDs for every incoming request, saving CPU cycles. The client attaches it in a custom header, often named X-Idempotency-Key. It keeps the logic localized. The server just reads it, checks the cache, and acts. It is a clean separation of concerns. Luna: Clean headers, clear logic. It is elegant in its simplicity. I wonder if this pattern will expand beyond payments into other areas like inventory management or social media posting. Anything that must not happen twice. Lucas: Absolutely. Inventory updates are a prime candidate. If two users try to buy the last item in stock simultaneously, you want the second request to fail gracefully rather than oversell. Social media posts too. Nobody wants their tweet duplicated because the network hiccuped. As APIs become more complex and our reliance on asynchronous communication grows, idempotency is becoming a non-negotiable requirement for any serious system. Luna: It is interesting how a simple string solves such a complex problem. It reminds me that sometimes the best solutions are the ones that force you to think about failure modes upfront. You build the safety net before you jump. Lucas: Precisely. It forces discipline. You stop assuming the network is perfect and start designing for imperfection. That shift in mindset is what separates robust systems from fragile ones. It is not just about code; it is about expectation management. Luna: I love that perspective. Expectation management. It applies to everything from API design to customer support. If you tell your users their actions are safe, you better make sure they are. This conversation has been incredibly useful for anyone building anything that touches money or state. Lucas: If today's tech conversation gave you something usable, and you want to keep these episodes ad-free, that is the link — buy me a coffee dot com slash fexingo. Even a small contribution helps us dig deeper into these technical details. Luna: Yeah, and honestly, knowing that support exists lets us spend more time testing these edge cases rather than worrying about the next sponsor slot. It is genuinely appreciated. Lucas: Back to the mechanics, though. One common mistake teams make is using the request body as part of the idempotency key. They hash the payload and use that as the key. This is dangerous because if the client sends slightly different whitespace or order of fields, the hash changes, and the server treats it as a new request. Luna: Oh, right. So the key must be independent of the content. It is purely an identifier for the intent, not a checksum of the data. You validate the content separately. You do not mix identity with validation. Lucas: Exactly. The key says who and when. The body says what. Keep them separate. Another pitfall is forgetting to include the key in your logs. If an incident occurs, and you cannot find the idempotency key in the log stream, you lose the ability to trace the duplication. Make sure your logging framework captures custom headers by default. Luna: Logging is the unsung hero of debugging. If you do not record the key, you are flying blind. It is a small detail that saves hours of investigation later. Are there any libraries that handle this automatically? Lucas: Some frameworks offer middleware plugins that can inject this logic. For example, certain Node.js or Python web frameworks have community packages that wrap your endpoints with idempotency checks. However, I always recommend writing your own wrapper. It gives you control over the storage backend, the TTL settings, and the error handling. Generic libraries often force you into their specific architectural choices, which might not fit your scale. Luna: Control is valuable. Especially when things go wrong at three in the morning. You want to know exactly how your safety net is woven. It feels like mastering idempotency is a rite of passage for senior engineers. Lucas: I would agree. It separates the juniors who write code that works from the seniors who write code that survives. It is a quiet skill, not flashy, but absolutely essential. As we move further into twenty twenty six, with more distributed systems and edge computing, this kind of resilience will only become more important. Luna: It is reassuring to know that there are proven patterns for these chaotic moments. We do not have to reinvent the wheel every time we launch a new feature. Sometimes the best innovation is just applying old wisdom to new problems. Lucas: Well said. And that is the takeaway here. Idempotency is not a new technology; it is a disciplined habit. Build the habit early, and your systems will thank you for years to come. Thanks for listening.