Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Shopify Rebuilt Its Checkout for One-Click Conversions
Transcript
- Lucas: If these conversations are useful for what you're building or running, you already know we keep this show ad-free by listener choice. There's no other business model here — just people who value the content enough to throw in a few bucks. Luna: It's a pretty simple bargain. You get the full deep dive without a sponsor break, and if it's worth the time, you can support it at buy me a coffee dot com slash fexingo. Lucas: Right. So today — Shopify's checkout rewrite. The most expensive form on the internet, and they rebuilt it from the ground up. Luna: Expensive as in 'every millisecond costs them millions' expensive. Lucas: Exactly. Shopify processes over ten percent of US e-commerce. Their checkout handles tens of millions of requests per second during peak — Black Friday 2025 they hit ten million requests per second at the apex. And for years, that checkout lived inside a monolithic Rails app. The same codebase that rendered product pages, handled cart logic, processed payments — all in one giant process. Luna: And that worked for a long time. But at some point, the scaling limits become real. Lucas: Right. The monolith couldn't independently scale the checkout path. If the product page got a traffic spike, it could steal CPU from checkout. Worse, the checkout code had to pass through layers of middleware designed for the entire storefront. A single request touched dozens of Ruby objects before it even validated a credit card. Luna: And that's where the latency becomes a conversion problem. Lucas: Huge problem. Shopify measured that every extra hundred milliseconds on checkout load time cost them roughly one percent in conversion. Their baseline checkout load time was around two point four seconds. On mobile, even higher. So the business case for a rewrite was straightforward — reduce checkout time by half, get a meaningful lift in GMV. Luna: But they didn't just optimise the existing Rails code. They went for a full architectural split — checkout as a standalone service. Lucas: Yes. They spun checkout out into its own service, written in Rust, running on edge nodes. The key insight was that checkout doesn't need the full storefront context. It only needs four things: cart data, customer identity, payment instrument token, and order metadata. Everything else — product descriptions, reviews, recommendations — is irrelevant at checkout. Luna: So they built a thin API that accepts exactly those four inputs and returns a confirmation. No HTML rendering, no session middleware, no asset pipeline. Lucas: Exactly. The checkout service runs as a set of lightweight HTTP handlers on Cloudflare Workers. They used Workers because they wanted compute as close to the user as possible. The checkout page itself is rendered as static HTML with JavaScript that calls the checkout API directly. No server-side rendering. Luna: That's a big shift from the Rails approach where everything was server-rendered with ERB templates. Lucas: Massive shift. But the real cleverness is in how they handle idempotency. If you think about it, checkout is the one place where duplicated requests can cause real financial damage. A double charge on a customer's card is a nightmare. Luna: Yeah, and with edge compute, requests can arrive out of order or be retried automatically if a worker crashes. Lucas: Right. So Shopify built an idempotency key system into the checkout API. Every checkout request includes a unique idempotency key — typically a UUID generated by the client. The service deduplicates based on that key. If the same key arrives twice, the second request returns the same response as the first, without processing payment again. They use a distributed key-value store — FoundationDB — to track these keys with a TTL of 24 hours. Luna: FoundationDB is interesting. Why that over Redis or DynamoDB? Lucas: They needed strict consistency — not eventual. If a worker crashes after writing the idempotency record but before returning the response, the retry needs to see that record. Redis with its async replication could lose a write. DynamoDB is eventually consistent by default. FoundationDB gives them linearizable transactions across a distributed cluster. It's overkill for most use cases, but for payment idempotency, overkill is the right bar. Luna: And it's also the same database they use for their inventory system. Lucas: Same cluster, different keyspaces. That's another architectural choice — consolidate core state into a single strongly consistent store rather than spreading it across multiple databases. It reduces operational complexity at the cost of a single point of failure, but FoundationDB's fault tolerance is well-proven at Shopify scale. Luna: Let's talk about the payment form itself. Shopify handles payment details directly — they don't outsource the entire form to Stripe or Braintree. Lucas: They built their own encryption layer. The credit card form is rendered with JavaScript that encrypts each field client-side using a public key provisioned by Shopify's key management service. The encrypted blobs are then sent to the checkout API, which decrypts them inside a hardware security module before tokenizing with the payment processor. This means the raw card number never touches Shopify's application servers in plaintext. Luna: That's a direct parallel to what Stripe does with Elements, but self-built. Lucas: Which gives them full control over the UX and the ability to optimise for conversion. For example, they can pre-fill the cardholder name from the customer profile without exposing the full card number. They can also run real-time validation on the encrypted fields — checking Luhn algorithm, expiration date format — without decrypting. That reduces round-trips to the server. Luna: And all of this runs on the edge. So the entire checkout flow — from form load to confirmation — happens in under 200 milliseconds for a typical user. Lucas: They published benchmarks showing p95 checkout duration dropped from 2.4 seconds to 187 milliseconds on desktop, and from 4.1 seconds to 340 milliseconds on mobile. That's an order of magnitude improvement. Luna: And the conversion lift? They measured a 12 percent increase on Black Friday 2025 compared to the previous year, controlling for traffic mix. Lucas: Twelve percent is huge for a change that's entirely backend — no new features, no marketing. It's purely a latency and reliability improvement. But the rewrite wasn't without its scars. They had a major incident during the 2024 holiday rollout where a bug in the idempotency key collision handling caused about three thousand duplicate charges over a two-hour window. Luna: I remember that. They had to reverse all those charges manually. What was the root cause? Lucas: The idempotency key was being generated from a combination of customer ID and timestamp. But on slow mobile connections, two requests from the same customer could have the same timestamp if the clock wasn't granular enough. They switched to a server-assigned key — the client requests a unique key from the checkout API before sending the payment, which guarantees uniqueness. Luna: So that adds one extra round-trip, but it's worth it for correctness. Lucas: Exactly. And they optimised that key request to be cached locally on the edge — the first request in a session fetches a batch of 10 keys, so subsequent checkouts don't need an extra round-trip. It's those sorts of detail that make the architecture robust. Luna: One thing I find striking is the cultural shift. Shopify used to be very much 'move fast and break things'. But you can't break checkout. Lucas: You absolutely cannot. So they adopted what they call 'blast radius reduction' — every new feature to checkout is behind a feature flag, rolled out to one percent of traffic for at least 48 hours. If any error rate metric crosses a threshold, the flag auto-disables. They also run canary deployments where a single edge node serves the new code for 10 minutes before full rollout. Luna: And they have a dedicated checkout reliability team on call 24/7. Not the feature teams — a separate SRE group that does nothing but monitor checkout latency and error rates. Lucas: Right. That team built a dashboard that shows real-time checkout conversion rate by region, by device type, by payment method. If conversion dips by more than half a percent, they page the on-call engineer. That's a remarkably tight feedback loop. Luna: So the net result: faster checkout, higher conversion, fewer incidents. But it cost them about two years of engineering effort and a complete rewrite of one of the most critical pieces of infrastructure. Lucas: Worth it, if you look at the GMV lift. Shopify's take rate is around two point nine percent on GMV. A twelve percent conversion lift on a platform that processes over two hundred billion dollars in GMV annually — the numbers speak for themselves. The checkout rewrite likely paid for itself within the first few months. Luna: That's the kind of ROI that makes a CTO's case easy to sell to the board. Lucas: Absolutely. And it's a template for any company where a single transaction path is the highest-stakes part of the application. You don't have to be Shopify scale to benefit from isolating that path, making it low-latency, and building strong idempotency guarantees. Luna: We deliberately don't run ads on these. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. Lucas: And we're back to the architecture. One last detail I want to cover: how they handle payment method updates. If a customer's card expires, they need to prompt for a new one without forcing a full re-checkout. Luna: That's the beauty of the decoupled design. The checkout API can store a reference to the payment token, not the card details. When the token expires, the client-side code fetches a new encryption key and re-renders just the card form — no page reload, no loss of cart state. Lucas: Exactly. And that's the kind of experience that keeps customers coming back. It's not flashy, but it's reliable. And in e-commerce, reliability is the feature people notice most when it's missing. Luna: On that note — if you're building something where every millisecond and every error matters, the Shopify checkout architecture is a masterclass in tradeoffs. Lucas: It really is. And it's live in production right now, processing millions of transactions a day. That's the CTO Podcast for this week. We'll be back with another deep dive.