Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Rebuilt Payment Routing for 99.999% Uptime
Transcript
- Lucas: If these conversations are useful for what you're building or running, that's exactly why we make them. Today, we're talking about Stripe's payment routing engine — the system that decides, in milliseconds, which bank or processor gets each transaction. And the architecture they built to keep that decision reliable. Luna: Because if that routing breaks, money doesn't move. And Stripe processes hundreds of billions of dollars a year. Lucas: Exactly. So a few years ago, Stripe realized their routing layer was becoming a bottleneck. They had grown from a handful of processor integrations to hundreds — every major card network, local payment methods, bank transfers, wallets. The routing logic had been a simple priority list: try processor A, if it fails try B, if that fails try C. Luna: Serial fallback. That's fine at small scale, but at Stripe's volume, the latency from that sequential retry becomes a real problem. Lucas: Right. And worse — if processor A has an outage, every transaction in that batch hits a failure before even trying B. So they rebuilt it as a state machine with a decision tree at the core. Luna: Can you describe that state machine? I think listeners will find the design surprisingly concrete. Lucas: Sure. Every transaction goes through six phases: initiation, routing, authorization, capture, settlement, and reconciliation. The routing phase itself is a state machine with three sub-states: evaluate, select, and execute. Evaluate is where the decision tree runs — it looks at the transaction attributes, the merchant's processor preferences, real-time signals from each processor's recent success rates, and latency. Luna: So it's not just a static priority list. It's a dynamic choice based on current conditions. Lucas: Exactly. The decision tree is trained on historical data but also factors in live health checks. Stripe publishes something called the 'processor health score' — a composite metric that blends success rate, response time, and error rate over a sliding window of, I think, the last five minutes. That score feeds into the tree. Luna: So a processor that's normally fast but starts spiking errors gets downgraded in real time. Lucas: Precisely. And the decision tree isn't a single monolithic model. They actually have separate trees for different payment methods — card-present, card not present, ACH, SEPA, each with its own branching logic. The card tree alone has over 200 leaf nodes. Luna: That's a lot of state. How do they keep the latency low? If you're evaluating a tree with 200 leaves on every transaction, that could be expensive. Lucas: They optimized the tree itself. Instead of evaluating every node, they use what they call 'lazy evaluation' — they only traverse branches that are relevant based on the transaction's features. And they precompute the most common paths. Something like 80 percent of transactions follow one of ten paths, so those are compiled down to a simple if-else chain. Luna: That's clever. So the hot path is hardcoded, and the cold path uses the full tree. Lucas: Right. And they also parallelized the health checks. Instead of checking processors sequentially, they have a sidecar service that pings all processors every 30 seconds and caches the results. So when a transaction comes in, the routing engine already knows the health score — it doesn't block on I/O. Luna: Which brings us to the circuit-breaker pattern. I've read that Stripe uses circuit breakers per processor, not just globally. Lucas: Yes. Each processor integration has its own circuit breaker. If the processor returns more than, say, five percent errors in a two-minute window, the breaker trips. Then the routing engine stops sending traffic to that processor for 30 seconds — it uses a half-open state where it sends a trickle of traffic to test recovery. Luna: So the circuit breaker is itself a state machine. Open, half-open, closed. Lucas: Exactly. And they have a hierarchy of breakers. There's a breaker per processor, but also per region, per merchant category. If a processor is failing only for high-risk merchants, the breaker isolates that segment without affecting low-risk traffic. Luna: That granularity is key for their five-nines target. You can't afford to blacklist a processor entirely if it's only partially degraded. Lucas: Right. The design goal was 99.999 percent uptime for the routing engine itself. That means less than about five minutes of downtime per year. The circuit breakers and the decision tree together help them achieve that by routing around failures before they become outages. Luna: And they test this. I remember a talk where one of their engineers said they run chaos experiments — randomly injecting failures into processor responses to verify the routing engine adapts. Lucas: They do. They have a dedicated Chaos Engineering team that simulates processor outages, latency spikes, and even data center failures. It's all automated, runs in production — on a small percentage of traffic — and the routing engine has to degrade gracefully every time. Luna: So what happens when the decision tree can't find a healthy processor? Is there a fallback? Lucas: There is. If all primary processors are unhealthy, the engine falls into what they call 'last-resort routing' — it uses a set of backup processors that are deliberately kept on a separate infrastructure stack. These are typically older integrations that are simpler but slower. They have higher latency but they're extremely reliable because they've been running for a decade. Luna: So reliability over speed at that point. Better a slow approval than a failed transaction. Lucas: Exactly. And Stripe monitors the fallback rate as a key metric. If the fallback rate exceeds 0.1 percent, that triggers an alert — it means the primary routing is under stress. Luna: Let's talk about how this architecture influenced the broader industry. A lot of fintech companies have adopted similar patterns because of Stripe's open-source contributions and talks. Lucas: Absolutely. Stripe open-sourced their circuit-breaker library, and their decision-tree framework has inspired several payment startups. But more than the code, I think the philosophy has spread: that routing is a real-time optimization problem, not a static config. Luna: It's interesting to compare it to other critical infrastructure — like how Cloudflare's DNS routing uses Anycast and health checks, or how CDNs route traffic to the nearest edge node. The patterns are similar: health scores, circuit breakers, fallbacks. Lucas: Right. The difference is that DNS and CDN routing is about data — payment routing is about money. The stakes are higher. A misrouted transaction can mean a double charge, a failed payment, or a compliance issue. That's why Stripe's design is so conservative in some ways — like the fallback stack being intentionally slow — and so aggressive in others, like the parallel health checks. Luna: We deliberately don't run ads on these episodes. If that kind of ad-free, deep-dive engineering content is valuable to you, supporting the show at buy me a coffee dot com slash fexingo helps keep it that way. No pressure, just the option. Lucas: Yeah, listener support is what makes this sustainable. We don't have to chase sponsors or optimize for clicks. We just get to talk about interesting systems. Luna: And back to Stripe — one thing I haven't asked: how do they handle the reconciliation side? After a transaction is routed, they have to make sure the money actually arrives. Lucas: That's the reconciliation state machine. After settlement, they run a matching process that compares Stripe's internal ledger with the processor's settlement report. If there's a discrepancy — say the processor reported a fee different from what was expected — it triggers an investigation workflow. Luna: So even with perfect routing, the back-end accounting is a whole other system. But that's a topic for another episode. Lucas: Exactly. For today, the key takeaway is that Stripe's routing architecture shows how to build reliability into a system where every decision has a financial consequence. The state machine, the decision tree, the circuit breakers, the fallback — each layer is designed to handle failure gracefully. Luna: And to think, it all starts with a single transaction. The routing engine makes a call in milliseconds, and billions of dollars move. Lucas: That's the beauty of it. It's invisible when it works. And when it breaks, everything stops. So getting it right matters.