Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Migrated Payment Routing to 99.999% Uptime
Transcript
- Lucas: So Stripe processes hundreds of billions of dollars in payments every year. And for a long time, their core routing engine that decides which bank or processor gets each transaction was a monolithic service. It worked, but it had a single point of failure. Back in 2022, they set a goal: 99.999% uptime for that routing layer. Luna: Five nines. That's less than six minutes of downtime a year. For something that touches every single payment. How do you even architect for that? Lucas: Exactly. The team realized they couldn't keep the monolith. If that one service went down, no new routes could be computed. So they broke it into a set of stateless, deterministic microservices. Each one handles a slice of the routing decision, and they're replicated across multiple regions and cloud providers. Luna: Deterministic meaning given the same input, you always get the same output. So you can have multiple instances make the same decision independently. No need for consensus. Lucas: Right. That's the key insight. Instead of having a single router that holds state about which bank is up or down, they encode the routing logic as a pure function of the transaction attributes and a global configuration that's versioned and cached. If one instance fails, another picks up the exact same request and produces the same result. Luna: But what about the configuration itself? If you need to update it because a processor goes offline, you have to push that change to every instance. That's a distributed state problem. Lucas: They use a separate configuration service that publishes immutable snapshots. Each routing instance pulls the latest snapshot on startup and refreshes it periodically. So the routing logic itself is stateless and deterministic, but the configuration is managed through a reliable, strongly consistent store. That split is what allows them to scale and survive failures. Luna: It's like separating the map from the navigation algorithm. The algorithm doesn't change, but the map gets updated. Lucas: Perfect analogy. And they didn't just rely on testing. They used formal verification to mathematically prove the routing logic was correct under all possible inputs. That's rare in production systems, but for something handling billions, it makes sense. Luna: Formal verification is usually associated with safety-critical systems like aerospace. What did that look like in practice for Stripe? Lucas: They used a tool called TLA+ to model the routing algorithm and check for invariants. For example, they proved that no transaction could be routed to a deactivated processor, and that routing decisions were deterministic across replicas. This caught subtle bugs that would have been hard to find in integration tests. Luna: Speaking of catching bugs, how did they roll this out without breaking payments for millions of merchants? Lucas: That's where the canary deployment came in. They started by sending one percent of traffic to the new system for a week. Then ramped to five percent, then ten. But they didn't just monitor error rates. They also compared routing decisions between the old and new systems in real time. If the new system chose a different processor, they flagged it for review. Luna: So they had a shadow comparison running alongside the old system. That's a really careful approach. Lucas: It was. And it took them nearly nine months from design to full production. But the result is a system that hasn't had a single routing outage since it went live in early 2023. They hit their five nines target. Luna: That's impressive. And it makes you think about the trade-offs. They prioritized determinism and formal verification over, say, dynamic load balancing based on real-time processor performance. Was there a latency cost? Lucas: Actually, the new system is faster. Because the routing decision is just a hash lookup on a local cache, it takes under a millisecond. The old system had to query a centralized database to check processor health. So they got better reliability and lower latency at the same time. Luna: That's the dream. A change that improves both. But I imagine the cognitive load on the team during design was intense. Lucas: Yeah, and that's a good point. If these conversations about architecture and trade-offs are useful for what you're building or running, consider supporting the show. Listeners like you keep it ad-free and going. You can buy me a coffee at buy me a coffee dot com slash fexingo. Luna: Absolutely. It's a small gesture that makes a big difference in keeping this content independent. And we really appreciate it. Lucas: So back to Stripe. One of the most interesting parts of the redesign was how they handled regional failover. They run in multiple AWS regions and also on Google Cloud, and they designed the routing instances to be completely stateless so that a request can land on any instance anywhere and get the same result. Luna: That requires that the configuration too is globally consistent. How do they handle that across clouds and regions without a single point of failure for the config store? Lucas: The configuration service is itself a distributed system with a quorum-based consensus protocol. They use a version of etcd with some customizations. Each routing instance pulls the latest configuration snapshot on startup and also subscribes to updates via a long-lived connection. But even if the config service is temporarily unreachable, the instances continue routing with the last known good config. Luna: So there's a built-in tolerance for config distribution delays. That's smart. What about when a processor goes down? The config needs to be updated to avoid routing to a dead endpoint. Lucas: Great question. They have a separate health-check system that detects processor outages and automatically updates the configuration. But here's the key: the update is not instantaneous. There's a propagation delay of up to a few seconds. So they also added a circuit-breaker pattern at the routing instance level. If a routing instance tries to send traffic to a processor and gets repeated failures, it marks that processor as temporarily unavailable in its local cache. Luna: That local cache acts as a fast fail-fast mechanism while the global config catches up. That's a nice layered defense. Lucas: Exactly. And they stress-tested all of this with chaos engineering. They simulated a AWS region failure by killing all instances in one region. Then they simulated a processor going down by blackholing traffic to a specific endpoint. The system maintained five nines throughout. Luna: I imagine those tests also revealed edge cases in the formal model. Formal verification is only as good as your specification. Lucas: Exactly right. They found that the model assumed all messages were delivered in order, but in reality, during a regional failure, some messages got reordered. They had to add a sequence number to the configuration snapshots to handle that. It's a good reminder that formal methods complement, but don't replace, real-world testing. Luna: And the team size? How many engineers were on this redesign? Lucas: The core team was about eight engineers, with support from a platform team and the SRE group. They weren't a massive army. It was a focused, disciplined effort with clear ownership. Luna: So what's the one takeaway for a CTO or engineering leader listening who's thinking about modernizing their own critical path? Lucas: I'd say: invest in making your critical path deterministic and stateless. That's what enabled all the other properties — replication, canary deployments, formal verification. If you can't model your core decision logic as a pure function, you're going to struggle to get to five nines. Luna: And don't skip the chaos engineering. Stripe's success came from proving the system under real failure conditions, not just hoping it would work. Lucas: Right. And one last thing: they didn't try to boil the ocean. They started with just the routing engine, leaving the rest of the payment pipeline untouched. That focused scope was what made the project manageable. Luna: A good reminder that even at Stripe, architectural transformation happens incrementally. Lucas: Exactly. Thanks for listening.