Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Rebuilt Its Payment Infrastructure for 100 Million API Calls a Day
Transcript
- Lucas: Let's talk about a company whose infrastructure most of us interact with daily without thinking about it — Stripe. Specifically, the quiet but massive architecture overhaul they completed earlier this year to handle over a hundred million API calls a day. Luna: A hundred million API calls a day. And that's just the API calls — not the internal service to service traffic or the batch processing. What was driving the rebuild? Lucas: The short answer: the Payments monolith wasn't going to survive the next decade. Stripe's core Payments service — the thing that processes charges, refunds, disputes — had been around since 2011. By 2024, it was handling not just payments but subscriptions, Connect, Terminal, all sorts of products. The codebase had grown into what they internally called 'The Spaghetti Monster'. Luna: Right, I've heard that term before. And the real problem wasn't just code complexity — it was data contention. That shared PostgreSQL database. Lucas: Exactly. Every payment, every refund, every webhook event, every dispute update — they all touched a handful of core tables. The `charges` table alone had over two billion rows. And with a hundred million API calls, you'd get hotspots on the `account_id` index, lock contention on balance updates. They saw transaction retry rates hitting two percent during peak — which sounds small, but at that scale, two percent of a hundred million is two million retried requests. That's a lot of latency and a lot of user-facing failures. Luna: So they decided to break the monolith apart. But financial systems are notoriously hard to decompose because you can't lose a single cent. How did they approach it? Lucas: They started with the ledger. Stripe's ledger — the double-entry bookkeeping that records every money movement — was embedded inside the Payments monolith. Every charge creation would write to the charges table, then immediately write a ledger entry. That meant ledger writes were coupled to payment lifecycle events. So they extracted the ledger into its own service, with its own database, and importantly, its own consistency model. Luna: But if you decouple the ledger, how do you keep it in sync with payments? You can't have a payment succeed but the ledger miss the entry. Lucas: They built an event-sourcing layer. Every state change in Payments — charge created, refund issued, dispute filed — gets written as an immutable event to a dedicated event store. And the ledger service subscribes to those events. The trick is that the event store uses the same transactional boundary as the Payments database initially, so the event is only committed if the payment state change is committed. That gives you exactly-once semantics. Luna: That's clever — using the monolith's own transactional integrity to seed the event stream. But eventually they'd want to decouple the event store from the monolith, right? Lucas: Exactly. Phase two was to make the event store its own service. But now you have a distributed transaction problem. How do you guarantee that a payment state change and its event are written atomically across two services? Luna: Two-phase commit? That's usually a non-starter at scale. Lucas: Stripe avoided it entirely. Instead, they used what they call 'outbox pattern with idempotency keys'. Every write to the Payments service first inserts a row into an outbox table within the same local transaction. A separate process — they call it the Event Relay — reads that outbox and publishes to the event store. And the idempotency key ensures that if the relay crashes and retries, the event store doesn't get a duplicate. Luna: So the idempotency key is the linchpin. Stripe has had idempotency for years — every API request has an `Idempotency-Key` header. But now they're using it internally, between services. Lucas: Right. And that's actually a beautiful property: the same mechanism they exposed to users for retry safety became the foundation for their internal consistency. The key is a UUID that the client generates. The server keeps a cache of recently seen keys and their responses. If a duplicate request arrives, you return the cached response — no reprocessing. Luna: But that cache must be huge at a hundred million calls a day. How do you store all those keys without it becoming a bottleneck? Lucas: They use a tiered approach. A fast in-memory cache for the first few minutes — that's where most duplicates arrive — backed by a persistent key-value store for longer retention. The retention window is tied to the API timeout: if your client times out after thirty seconds, the idempotency key lives for at least sixty seconds. They also prune old keys aggressively. Luna: And this all works because the ledger is now event-driven. But what about the actual payments database — the one with the two-billion-row charges table? Did they split that too? Lucas: Yes, and this is where domain-driven design came in. They identified bounded contexts: Charges, Refunds, Disputes, Payouts, Connect sub-accounts. Each became its own service with its own database. But here's the thing — Stripe didn't go full microservices overnight. They started with the highest-contention tables, like the balance updates. That table was being written to by virtually every payment flow. They extracted it into a Balance Service with a dedicated PostgreSQL cluster, sharded by account ID. Luna: Sharded by account ID — that makes sense. Each account's balance is independent, so you can partition by that key. But what about cross-account transactions, like a Stripe Connect platform moving money from a buyer to a seller? That touches multiple shards. Lucas: That's the hard part. For those, they use a two-phase commit across shards, but they've heavily optimized it. They keep the transaction scope as narrow as possible — typically just a few milliseconds. And they've built a custom transaction coordinator that can handle failures gracefully. If a shard goes down mid-transaction, the coordinator can retry against a replica. Luna: Okay, so let's talk about the cost. This was a multi-year effort — how many engineers, how much downtime? Lucas: Stripe hasn't published exact numbers, but from what I've gathered, the core team was around fifty engineers, with another hundred or so across dependent teams. They did it over about eighteen months, with zero planned downtime. The migration strategy was strangler fig — they'd route a small percentage of traffic to the new services, compare results, and gradually ramp up. Luna: That's the ideal approach, but it requires a parallel infrastructure running for months. The compute and storage costs must have been significant. Lucas: Absolutely. They ran dual-writes for almost a year. Every write went to both the old monolith and the new services, with a background comparison job that flagged discrepancies. They had a team of engineers whose full-time job was investigating mismatches — most were race conditions in the comparison logic, not actual data corruption, but they took no chances. Luna: And honestly, if today's episode was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Keeps us ad-free and digging into these rebuild stories. Lucas: Yeah, it's a small gesture that makes a big difference for us. And we really appreciate it. Anyway — back to Stripe. So after eighteen months, they flipped the switch. What happened? Luna: I read that p99 latency dropped by forty percent for payment API calls. And the retry rate went from two percent to basically zero. Lucas: Correct. The biggest win was the elimination of lock contention. Before the split, a single busy account could slow down all other accounts sharing the same database node. After sharding, each account had its own resources. The forty percent latency improvement was mostly from removing those hot-spot waits. Luna: But there's a hidden cost: operational complexity. Now they have dozens of services, each with its own database, its own deployment pipeline, its own monitoring. The cognitive load on the on-call engineers is higher. Lucas: Absolutely. Stripe invested heavily in internal tooling — an automated runbook system called 'Autopilot' that can diagnose common issues and even roll back a service if it detects a regression. And they built a unified observability platform that traces requests across all services. Without that, the system would be unmanageable. Luna: So what's the lesson for CTOs listening? When should you consider a similar rebuild? Lucas: I think the trigger is when your database contention becomes the primary bottleneck. Not just CPU or memory — but when you see locks, retries, and timeouts growing faster than your traffic. That's a sign your data has outgrown your schema's ability to isolate workloads. The second trigger is when adding a new feature requires touching ten tables because the domain model has become too coupled. Luna: And the counterpoint: don't do it if you haven't solved the observability and operability problem first. Stripe had years of investment in monitoring before they attempted this. Lucas: Exactly. The rebuild itself was the last step, not the first. They had to have the foundation — idempotency, event sourcing, sharding patterns — already proven in smaller contexts before they could apply them to the core payments system. Luna: One more thing: Stripe used a custom event store rather than Kafka or Pulsar. Why? Lucas: They needed exactly-once semantics and strong ordering within an account's event stream. Kafka can give you ordering within a partition, but exactly-once is tricky with external systems. Stripe's event store is built on PostgreSQL itself — using logical replication and the outbox pattern. It's simpler, they control the consistency model, and it integrates with their existing backup and failover tooling. Luna: So the lesson is: don't automatically reach for the shiniest distributed system. Sometimes a well-tuned relational database with a clever pattern is the right tool. Lucas: Exactly. And that's the quiet truth behind Stripe's rebuild. It's not about using the latest technology — it's about applying the right patterns at the right time. Event sourcing, domain-driven design, idempotency — these are ideas from the early 2000s. The novelty is applying them at this scale with zero downtime. Luna: And that's what makes it a case study worth studying. Any predictions for what Stripe's next architectural challenge will be? Lucas: I think the next frontier is multi-region active-active. Right now, Stripe operates primary and replicas, but all writes go to a single region. With a hundred million calls a day, they'll eventually need to write to multiple regions with conflict resolution. That's a whole new level of complexity — and probably a future episode. Luna: We'll be watching. Thanks, Lucas. Lucas: Thanks, Luna.