Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Shopify Rebuilt Checkout for One-Click Conversions
Transcript
- Lucas: If these conversations are useful for what you're building or running — and it's Friday, you might be wrapping up a week of tough architectural decisions — let's talk about a company that had to make one of the hardest ones: how do you rebuild checkout when it's the single most revenue-critical path in your entire platform? Luna: Shopify, I'm guessing? They process something like tens of billions in GMV annually, so any change to checkout is high-stakes. Lucas: Exactly. Shopify's checkout was historically a monolithic Rails app — very standard for ecommerce in the 2010s. But as they grew to 1.7 million merchants, that monolith created real problems. The team found that even small deployments could cause cascading failures during peak events like Black Friday, where they were handling over 10,000 checkouts per minute at the peak. Luna: Right, and the monolith meant every payment gateway integration — they support over 100 — was tangled together. One gateway's bug could bring down the entire checkout for every merchant. Lucas: Precisely. So around late 2023, Shopify's checkout engineering team began a multi-year effort to decouple checkout into an event-driven, headless service. They called it 'Project Checkout Unbundled' internally. The core idea was to isolate checkout as its own service, with its own data store, its own deployment pipeline, and its own SLAs. Luna: That's a big departure from the typical incremental refactor. Why go straight to a full service isolation? Lucas: They had tried incremental steps — extracting parts like shipping logic or tax calculation into separate services — but the checkout core was still too coupled. Every time they tried to optimise the conversion flow, they'd hit a wall because the payment state machine was embedded across a dozen models. So they decided to wrap the entire checkout flow — from cart review to order confirmation — into a single service that owns the complete state machine. Luna: And they went headless, meaning the front-end is completely decoupled from that service. So merchants can customise the checkout UI without touching the transaction logic. Lucas: Yeah, that was a key requirement. Shopify's larger merchants want to customise the checkout experience — think one-page checkout, upsells, loyalty points — but historically any front-end change risked breaking the payment flow. By exposing a set of well-defined APIs, the checkout service handles all the heavy lifting: cart validation, address verification, payment authorisation, fraud scoring, order creation. The front-end just calls those APIs. Luna: How did they handle state management? Checkout sessions can last minutes — someone might add a discount code, switch payment methods, come back from a 3D Secure redirect. The service has to maintain that state across multiple requests. Lucas: Good question. They moved from a database-backed session model to an event-sourced approach. Every action on a checkout — item added, address changed, payment selected — is recorded as an event. The current state is derived by replaying those events. That gives them a complete audit trail and makes debugging failed checkouts dramatically easier. But it also introduces latency: replaying events on every request would be too slow, so they maintain an in-memory snapshot that's updated as events come in. It's a classic CQRS pattern with event sourcing. Luna: That's interesting — in-memory snapshots are fast, but what happens if the service goes down? Do they lose state? Lucas: No, because every event is persisted to a durable event store — they used a partitioned Kafka-like log — so when a new instance starts, it rebuilds the snapshot by replaying the event stream for that checkout session. They tuned it so that rebuilding takes under 50 milliseconds for even the longest sessions. The trade-off is storage: each checkout generates dozens of events, and with millions of checkouts per day, the event store grows fast. They had to implement aggressive compaction policies, merging events into snapshots after a session completes. Luna: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making deep dives like this one possible. No ads, no sponsors — just listeners who find value in the show. Lucas: Yeah, and we really appreciate that. It lets us spend weeks researching a single architectural story without worrying about keeping an advertiser happy. So thank you to everyone who's part of that. Luna: Alright, back to the flow — one-click checkout. How did Shopify implement that with the new architecture? Lucas: One-click checkout — or Shop Pay as they brand it — depends on storing payment credentials securely and applying them with a single token. In the old monolith, the token was tied to a specific merchant's checkout session. In the new service, they introduced a 'payment method vault' that's merchant-agnostic: a buyer can save their card once and use it at any Shopify store with Shop Pay. The checkout service queries the vault via a gRPC call, and if the buyer has opted in, it pre-fills the payment method and initiates authorisation immediately. Luna: That reduces the checkout flow from maybe five steps to one. But it also means the service has to handle race conditions — what if the buyer clicks 'buy' twice? Lucas: Idempotency keys. Each checkout session generates a unique idempotency key at creation. Every payment authorisation attempt includes that key, so the payment gateway — Stripe, Adyen, etc. — can deduplicate. If the buyer clicks twice, the second request is rejected because the order is already in 'processing' state. The state machine ensures idempotency: a checkout can only transition from 'open' to 'completed' once. Any other transition is invalid. Luna: What about fraud detection? In a one-click flow, you don't have the traditional friction of typing in a CVV or address. So how do they prevent fraud? Lucas: They moved fraud scoring into the checkout service itself. Previously, fraud checks happened after the order was created — if it was flagged, the order would be cancelled, which is a poor experience. Now, the checkout service calls a fraud detection service — built on a machine learning model that evaluates hundreds of signals — before authorising payment. The model scores the risk in under 200 milliseconds. If the score is above a threshold, the service falls back to a two-step checkout, requiring CVV or 3D Secure. That way, legitimate buyers get the frictionless experience, and high-risk transactions get extra verification. Luna: So the architecture adapts the checkout flow dynamically based on risk. That's clever, but it also adds complexity — the checkout service now has to handle multiple flows depending on fraud score. Lucas: Exactly. The team modelled that as a state machine variant. The base checkout state machine has states like 'billing_address_collected', 'payment_method_selected', 'authorising', 'completed'. But if fraud detection requires extra verification, the service transitions to a 'requires_verification' state, then waits for the buyer to complete 3D Secure before moving to 'authorising'. The same event-sourcing infrastructure handles this gracefully — each state transition is an event, so the system knows exactly where every checkout is. Luna: One thing I'm curious about: Shopify has thousands of apps that hook into checkout — loyalty apps, subscriptions, upsells. How did they ensure those apps still work with the new headless checkout? Lucas: That was arguably the hardest part. They introduced a 'checkout extension' API that app developers can use to inject their own UI components into the checkout flow. But those components are rendered on the client side, not server-side. The checkout service provides a set of 'slots' — like after the shipping address, or before the payment button — where apps can render their content. The app's code runs in an iframe sandbox, so it can't access the checkout service's internals. The app communicates with the checkout service via postMessage calls, requesting discounts or modifying the cart — but the checkout service always validates those requests server-side. Luna: That sounds like a lot of backward compatibility work. Did they break any existing apps during the migration? Lucas: They had a phased rollout. First, they ran the new checkout service in shadow mode — duplicating live traffic to the new service but still serving checkout from the old monolith. They compared the output of both systems for every checkout, looking for discrepancies. After three months of shadow traffic with zero discrepancies, they started routing 1 percent of live checkouts to the new service. They gradually increased that over six months. Even then, a small number of apps that relied on direct database access to the old checkout tables broke. Those apps had to migrate to the new API, but Shopify gave them a six-month deprecation window. Luna: That's a responsible migration strategy. So what's the result now? How much faster is checkout? Lucas: According to Shopify's engineering blog, the median checkout time — from cart to order confirmation — dropped from about 4.5 seconds to under 1.2 seconds for Shop Pay users. That's a 73 percent reduction. And for merchants, conversion rates increased an average of 12 percent. But there's a trade-off: the new service is more resource-intensive. Each checkout session requires more in-memory state and more gRPC calls. They had to invest significantly in infrastructure — particularly in their event store and caching layer — to keep latency low. Luna: But for a platform that processes billions in transactions, that investment pays for itself if conversion improves even a fraction of a percent. Lucas: Absolutely. And the team is already working on the next iteration: multi-merchant checkout — where a single checkout can include items from multiple Shopify stores, with one payment and one shipment. That's only possible because the checkout service is now a standalone entity that can orchestrate across merchants. Luna: That's ambitious. It would basically turn Shopify into a mini marketplace aggregation layer. I imagine that brings a whole new set of challenges around inventory and tax calculations. Lucas: For sure. But if the checkout service is already handling state machines across payment methods, fraud, and apps, adding a 'multi-merchant' state is just a new composite state. The pattern scales. It's a great example of investing in architecture that enables future product features — not just fixing the current pain. Luna: Alright, so what's the one lesson for CTOs listening? If you're considering a similar checkout or critical-path rebuild, what should you take away? Lucas: Don't underestimate the state machine design. Shopify spent months just modelling the checkout state transitions — they had a whiteboard with over 30 states and 50 transitions. That upfront investment made the rest of the architecture straightforward. The second lesson: shadow traffic is your friend. You can't prove correctness by testing alone; you need real production traffic to validate the new system against the old one. And third, plan for the migration to take at least a year. Shopify's took 18 months from first commit to full rollout. Luna: Good advice. And it's a reminder that even the most critical system can be rearchitected with careful planning and a willingness to invest in the long term. Lucas: Exactly. If your checkout is still a monolith, you might want to start thinking about that state machine now.