Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Rebuilt Its Payment Infrastructure for 100 Million API Calls a Day
Transcript
- Lucas: Stripe processes over a hundred million API calls a day. That's not a flex from a press release — that's the actual number engineers there have to design for. And the most interesting part isn't the scale itself, it's how they rebuilt their core payment pipeline to stay correct at that volume. Luna: Correctness being the operative word. Because in payments, 'almost right' means double charges or failed transactions. Both of which lose you money and trust. Lucas: Exactly. So today I want to walk through three architectural decisions Stripe made that I think are genuinely instructive — not just for fintech, but for any system where data integrity is non-negotiable. First, the move from a monolithic Rails app to a service-oriented architecture. Second, how they built idempotency into the fabric of the API. And third, how they used deterministic simulation to test the whole thing. Luna: Let's start with the monolith. Stripe famously ran on a single Rails application for years, even as they processed billions. What pushed them over the edge? Lucas: The classic signal: deploy pain. Every change, even to an internal tool, required a full deploy of the entire app. A small team's PR could take down the whole payment pipeline. Around 2015 or 2016, they started carving out services — first the less critical ones like billing and reporting. But the core charge-creation path stayed monolithic for a long time because the risk was too high. Luna: Right. You don't refactor the heart of your business while it's beating. So how did they eventually do it? Lucas: They used the strangler fig pattern — gradually routing traffic from the monolith to new services. But the key detail is they didn't just split by domain. They split by workflow. For example, they extracted the tokenization service first, because credit card tokenization had a clean interface and clear ownership. Then they extracted the balance-check service. Each extraction came with a new, dedicated database. And they required an explicit migration plan for data, not just code. Luna: So the data migration was the harder part. That's often underestimated. Lucas: Massively. Stripe had to ensure that during the migration, no transaction was lost or duplicated. They built a dual-write system where both the old monolith database and the new service database received writes simultaneously, and a reconciliation job ran behind the scenes to catch discrepancies. That job ran for months before they flipped the switch. Luna: That's a lot of operational overhead. Did they ever consider a big-bang rewrite? Lucas: They explicitly rejected it. Their CTO at the time, Greg Brockman, wrote about this — the idea of freezing feature development for a year to rewrite is a fantasy. Your business changes, your API changes, your customers change. By the time you finish, the rewrite is obsolete. So incremental migration was the only viable path. Luna: And that brings us to idempotency. Stripe's API is famous for its idempotency keys. Tell me how that's built. Lucas: Idempotency is the property that retrying the same request produces the same result. For Stripe, this is fundamental because network failures happen constantly. If a client sends a charge request and doesn't get a response, they don't know if it succeeded or failed. With idempotency, they can safely retry. Stripe implements this by requiring an idempotency key in the header. The key is a unique string — often a UUID — that the client generates. The server stores the result of the first request keyed by that id. Luna: But storing that key and result needs to be consistent, right? What happens if the server crashes after processing the charge but before storing the key? Lucas: That's the critical race condition. Stripe's solution is to use a transactional outbox pattern. The charge is written to the database, and the idempotency key is written in the same transaction. If the server crashes after the transaction commits, the key is persisted. If it crashes before, the charge never happened. This gives you strong guarantees. But it also means every write goes through a single leader database, which is a bottleneck. Luna: So you trade some availability for consistency. That's a deliberate choice for payments. Lucas: Exactly. Stripe chose consistency over availability in the payment path. They'd rather reject a request than silently double-charge. And that's a business decision as much as a technical one. Luna: Let's talk about the simulation testing piece, because that's the most unusual part of their approach. Lucas: Yes. Stripe developed a custom deterministic simulation framework they call 'Morpheus' — though the name might have changed. The idea is you take the entire system, including network calls, database queries, and even clock time, and you model it in a simulated environment where you control all nondeterminism. That means you can replay the exact same scenario millions of times with different interleavings to find race conditions. Luna: So it's like property-based testing for distributed systems. Lucas: Exactly. They run these simulations as part of their CI pipeline. Every commit that touches critical payment logic triggers a simulation suite that can take hours. It's expensive, but it catches bugs that would never be found in production testing because the interleavings are so rare. They've talked about finding issues that only occur when a specific network timeout happens at the exact moment a database failover occurs. You'd never catch that with a load test. Luna: That's the kind of investment most companies don't make until they've been burned. Stripe was burned early on, I think. Lucas: They had a major outage in 2014 where a database migration caused a cascade of failures. That was a forcing function. After that, they invested heavily in fault injection and simulation. It's not cheap — you need engineers who understand both distributed systems and the business domain. But for a company processing billions of dollars, it's insurance. Luna: It's also a cultural statement. Stripe famously has a low tolerance for operational heroics. They prefer systems that are boringly predictable. Lucas: Right. And that's the thread that ties these three decisions together. The service extraction, the idempotency layer, the simulation testing — they all aim to reduce surprise. When you're handling a hundred million API calls a day, each one potentially moving money, you want the system to behave the same way every time. Surprises cost money and trust. Luna: And that's a lesson that applies well beyond fintech. Any system where data integrity matters — healthcare records, inventory systems, even multiplayer game state — could benefit from that mindset. Lucas: Absolutely. And on that note, I think conversations like this are really useful if you're building or running something where correctness is critical. Our goal with this show is to dig into the real engineering decisions, not the marketing fluff. That's also why we don't run ads — we want this to be a clean, focused resource for people who actually build systems. Luna: If you find that valuable and want to support it, there's a link — buy me a coffee dot com slash fexingo. No pressure, just an option if you want to help keep it ad-free. Lucas: Yeah, exactly. It's a small way to keep the conversation going. Back to Stripe — one more detail I find fascinating: their database architecture. They moved from a single Postgres instance to a sharded setup, but they didn't use a standard sharding library. They built their own because they needed deterministic behavior under failure. Luna: What kind of custom sharding? Are you talking about consistent hashing, or something more specific? Lucas: More specific. They shard by account ID, but they also introduced a concept called 'virtual shards' — logical partitions that can be moved between physical databases without downtime. This allows them to rebalance load without changing the routing logic. The routing layer uses a lookup table rather than a hash function, so they can remap virtual shards as needed. Luna: That gives them operational flexibility. You can move a hot tenant to a dedicated database without affecting other accounts. Lucas: Right. And they built tooling to monitor per-shard latency and throughput. If one shard starts degrading, they can split it or move it. The whole thing is designed to be boringly routine. No panicking at 3 AM. Luna: That's the ideal state for an infrastructure team. The system should handle scale changes without human intervention. Lucas: And Stripe's SRE model reinforces that. They don't have a central SRE team that gets paged. Instead, each service team owns its own reliability. They embed observability and runbooks into the service itself. If a service goes down, the team that wrote it fixes it — and they have the context to do so quickly. Luna: That requires a certain engineering maturity. Not every org can do that, especially if they're growing fast and hiring juniors. Lucas: True. Stripe invested heavily in onboarding and internal documentation. They have a culture of writing things down. Every decision, every outage postmortem, every architecture review is documented. New hires spend their first weeks reading past incident reports. It's a knowledge-sharing flywheel. Luna: So the key takeaways for me: incremental migration, idempotency as a first-class API concern, deterministic simulation testing, and a culture that treats reliability as a team responsibility rather than a separate function. Lucas: That's a good summary. And I think the biggest meta-lesson is that all of these choices are boring. There's no clever algorithm, no AI magic. It's just disciplined engineering applied over years. Stripe's advantage is not that they're geniuses — it's that they consistently make the boring, correct choice. Luna: Which is the hardest thing to do. Especially when your investors want you to move fast. Lucas: Exactly. The next time you hear about a startup's 'scalable architecture,' ask about their idempotency layer. That'll tell you how serious they are. Luna: Or ask about their testing strategy. If they don't have simulation tests for critical paths, they're flying blind at scale. Lucas: Great point. And that's our show for today. Thanks for listening.