Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Runs a Global Payment Platform With 99.999 Percent Uptime
Transcript
- Lucas: So Stripe processes hundreds of billions of dollars in payments every year. And when you're that big, even a tiny fraction of downtime translates into millions in failed transactions and angry merchants. Luna: Right. And yet I don't think I've ever heard of Stripe having a major outage that actually stopped payments globally for hours. Lucas: That's by design. Stripe publicly targets five-nines reliability — that's 99.999% uptime. Which means less than about five and a quarter minutes of total downtime a year. Luna: That's an insane target for any system, let alone one that touches banking rails and has to handle spikes like Black Friday. Lucas: Exactly. And achieving it means you can't just throw more servers at the problem. You need a layered reliability strategy. So today I want to talk about three specific things Stripe does: circuit breakers, canary deployments, and multi-region database topology. Luna: Let's start with circuit breakers. That's a pattern we've seen elsewhere — Netflix popularized it with Hystrix. Lucas: Right. Stripe uses circuit breakers at the service level. If a downstream dependency — say, a credit card processor — starts returning errors or timing out, the circuit breaker trips. The calling service stops trying for a set period and fails fast. Luna: So instead of cascading failures where every request backs up and takes down the whole system, you contain the blast radius. Lucas: Exactly. And Stripe configures those breakers with very specific thresholds. For example, they might trip if 10% of requests to a dependency fail within a 30-second window. Then they'll wait 60 seconds before trying again. Luna: That's pretty aggressive. But it makes sense when you're dealing with payments. A delayed failure is better than a slow failure that spirals. Lucas: Right. And the other key piece is that Stripe doesn't just have circuit breakers on outgoing calls — they also have them on incoming requests. If a merchant's system is sending malformed or excessive requests, Stripe can throttle them without affecting other merchants. Luna: That's a nice isolation property. So the second layer — canary deployments. How does Stripe roll out changes without breaking the payment flow? Lucas: Stripe uses a multi-stage canary process. A new version of a service is first deployed to a single instance in one region, then gradually scaled to a percentage of traffic. The canary runs for a minimum observation period — typically 15 minutes — while automated monitoring checks error rates, latency, and business metrics like successful charge rates. Luna: And if something looks off, they auto-rollback. Lucas: Yes. And the interesting part is that Stripe's canary system is itself fault-tolerant. The deployment pipeline is designed so that a bad deploy can't take down the orchestration system. They also have a manual override for critical incidents, but the default is automated rollback. Luna: That's a big cultural commitment. A lot of companies say they do canaries, but then they let engineers bypass them during crunch time. Lucas: Stripe has a policy that any bypass must be approved by a designated reliability engineer. And those bypasses are rare — I think they're measured in single digits per quarter across all services. Luna: That speaks to their incident response culture too. They must have really good runbooks. Lucas: They do. Every service has a runbook that lists the exact steps for common failure modes. And they practice failure scenarios weekly. Literally, every week, a rotating team runs a 'chaos drill' where they deliberately inject a fault — like killing a database replica — and the on-call engineer has to respond. Luna: So by the time a real incident happens, the muscle memory is already there. Lucas: Exactly. And that brings us to the third layer: database topology. Stripe runs a global multi-region setup using a custom database layer on top of MySQL and Vitess. Luna: Vitess is the same technology YouTube uses to scale MySQL horizontally. Lucas: Right. Stripe has multiple read replicas in each region, and writes are sharded across multiple database clusters. But the key for reliability is that they can survive a full region failure. If us east goes down, us west can take over within minutes. Luna: And they don't lose data because they use synchronous replication for critical data? Lucas: They use a mix. For transaction-critical data — like the actual payment authorization — they use synchronous replication to at least two regions. For less critical metadata, they use asynchronous. And they have a recovery time objective of 30 minutes for the most critical services. Luna: Thirty minutes to recover everything from a full region failure. That's impressive. Lucas: But they've had to prove it works. A couple of years ago, a major cloud provider had a multi-hour outage that affected several services. Stripe's payment processing remained available throughout because traffic was diverted to other regions within minutes. Luna: That's the kind of story that makes merchants trust you with their payment flow. Lucas: Right. And it's not just about the technology — it's about the organizational habits. Every Stripe engineer goes through reliability training, and there's a dedicated SRE team that works with product teams to design for failure from the start. Luna: I think a lot of listeners who build or run systems will find these patterns useful, even if they're not at Stripe's scale. Lucas: If these conversations are useful for what you're building or running, that's exactly why we keep the show ad-free and listener-supported. We've had listeners say these deep dives have helped them rethink their own architecture. So if you find value in the show, you can support us at buy me a coffee dot com slash fexingo. Luna: Even a small contribution helps us keep doing episode after episode with no commercial breaks. Lucas: It does. And we appreciate it. Now, back to Stripe — one more thing I find fascinating is how they handle what they call 'planned downtime'. Luna: Planned downtime? I thought they never had downtime. Lucas: They do have planned maintenance windows, but they've built their system so that no single maintenance event ever causes a customer-facing outage. They use a technique called 'zero-downtime deployment' where they can upgrade database schemas or swap out hardware without taking anything offline. Luna: How does that work with schema changes? A lot of companies struggle with that. Lucas: Stripe uses online schema change tools that create new tables, copy data incrementally, and then atomically swap the old table for the new one. No locks, no downtime. They've open-sourced some of these tools, like gh-ost, which was originally developed at GitHub but Stripe contributes to. Luna: So it's not just internal engineering — they're also pushing improvements back to the broader community. Lucas: Right. And that contributes to their hiring pipeline too. Engineers want to work on systems that are both mission-critical and well-architected. Luna: I think the takeaway for me is that reliability isn't a feature you add at the end. It's a set of decisions you make at every layer — from code to infrastructure to culture. Lucas: Exactly. And Stripe's approach shows that it's possible to achieve five-nines even in a complex, globally distributed payment system. It just takes relentless investment in the right patterns.