Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Stripe Migrated Its Core Payments Stack
Transcript
- Lucas: Stripe processes hundreds of billions of dollars in payments every year. And for most of its history, the core engine that actually moves money from point A to point B was a Ruby-on-Rails monolith. Luna: Which is frankly amazing — that they got that far on Rails for something so mission-critical. Lucas: It is. But by 2024, the system was showing serious strain. The team called it 'the mover' internally — it handled every payment authorization, capture, and refund. And it was handling over 100 million API requests per day, with latency requirements in the low hundreds of milliseconds. Luna: So they decided to rewrite it. Which is usually a terrible idea in engineering. Lucas: Right, Joel Spolsky's famous dictum about rewrites being a trap. But Stripe didn't do a big bang rewrite. They did an incremental migration over about eighteen months, replacing the monolith piece by piece with services written in Rust — not Go, not Java, but Rust. Luna: Why Rust specifically? I know Stripe has been a Rust shop for infrastructure tools for years, but for the core payments logic — that's a different level of commitment. Lucas: Two main reasons, according to the engineering leads who spoke at their internal conference. First, Rust gives them memory safety without a garbage collector. Payment systems cannot afford unpredictable pauses from garbage collection when you're processing a transaction. Second, Rust's type system lets them encode business invariants at compile time. Things like 'you cannot refund a payment that hasn't been captured yet' become compile errors, not runtime bugs. Luna: That's huge for correctness. But how did they actually pull off the migration without downtime? Because you can't just flip a switch on 100 million requests a day. Lucas: They used a pattern called 'parallel run with comparison'. For each payment operation — say, authorizing a credit card — they would run the new Rust service alongside the old Rails service. The new service would process the request and produce a result, but the old service's result was the one actually returned to the user. Then they'd compare the two outcomes. Luna: So they were basically shadow-running the new system for months. Lucas: Exactly. They called it 'diff mode'. Every discrepancy was logged and investigated. Over time, the diff rate dropped to near zero. Then they flipped the switch for that specific operation to serve production traffic. But they did it behind feature flags, so if something went wrong, they could instantly revert. Luna: Feature flags for infrastructure changes — that's a common pattern, but Stripe took it further. I read they had per-account flags. Lucas: Right. They could route specific merchants to the new system while keeping everyone else on the old one. So they started with their own internal Stripe accounts, then moved to friendly beta merchants, then gradually ramped up. If a merchant saw a problem, they could flip them back in seconds. The whole process took about a year for all core operations. Luna: That level of granularity requires a really sophisticated feature flag infrastructure. Which they built themselves, obviously. Lucas: They did. And it's worth noting that the engineering org structure played a big role. Stripe organized the migration around 'pods' — small, cross-functional teams of about six to eight engineers each. Each pod owned one piece of the payments flow. One pod owned authorization, another owned settlement, another owned refunds. Each pod had full ownership of their Rust service and could make independent decisions. Luna: So the org mirrored the architecture they were building. That's Conway's Law in action. Lucas: Exactly. The old monolith was owned by one big team — the mover team — and that created bottlenecks. By breaking into pods, they increased parallelism. But they also had to invest heavily in shared infrastructure: a common Rust framework they called 'the plank', and a set of internal libraries for serialization, logging, and metrics. Luna: What about data consistency? Payments are all-or-nothing — you can't have partial states. Lucas: That was the hardest part. In the monolith, consistency was easy because everything was in a single Postgres database with transactions. In the new event-driven architecture, each service has its own database, and they communicate via an event bus. To maintain consistency, they used a pattern called 'outbox with exactly-once delivery'. Each service writes both the business record and an event to its own database in a single transaction. A separate process reads the event and publishes it to the message bus. Luna: So the events are guaranteed to be published exactly once, because they're part of the database transaction. Lucas: Right. And consuming services are idempotent — they can handle seeing the same event twice without double-processing. That's been a standard pattern in distributed systems for a while, but Stripe had to retrofit it into existing flows. Luna: Did they see any performance improvements after the migration? Lucas: Significant ones. The median latency for a payment authorization dropped by about 40 percent. And the p99 latency — the worst-case — dropped even more. Because Rust's performance is more predictable than Ruby's, especially under load. Plus, they could now scale individual services independently. If settlement was getting slow, they could spin up more settlement replicas without touching authorization. Luna: What about the team morale? Eighteen months of migration is grueling. Lucas: The engineers I've talked to said it was actually energizing compared to maintaining the monolith. They had clear ownership, they were learning Rust, and they could see their changes in production quickly. But there were definitely painful periods — especially when the diff mode showed mysterious discrepancies that took weeks to debug. Luna: Any lessons for CTOs considering a similar migration? Lucas: I think the big lesson is: don't rewrite, migrate incrementally. Stripe didn't start from scratch. They took one piece of the monolith — the simplest operation, which was retrieving a payment's status — and built a Rust service for that first. That gave them confidence in the infrastructure. Then they tackled refunds, then captures, then authorizations last. Each piece validated the approach before they moved to the next. Luna: And they invested heavily in testing. I heard they had a simulation environment that could replay production traffic against the new system. Lucas: Yes, they built a traffic replay tool that recorded real requests from production and replayed them against the new Rust services. That caught a lot of edge cases before they ever hit live traffic. And they had chaos engineering experiments — they would randomly kill services in the new system to make sure the fallback worked. Luna: So it's not just about the technology. It's about the process and the culture of safety. Lucas: Exactly. The CTO of Stripe at the time, David Singleton, has said that the migration was as much about organizational change as technical change. They had to shift from a mindset of 'one team owns the monolith' to 'many teams own their services'. That required new communication patterns, new on-call procedures, and new ownership models. Luna: I think that's the real takeaway for engineering leaders. The tech choices — Rust, event-driven, feature flags — are all interesting. But the org design and the incremental approach are what made it work. Lucas: Absolutely. And it's a reminder that even the best-engineered companies face these challenges. Stripe's payments system was already world-class, but they knew it wouldn't scale indefinitely. So they invested in the migration before it became a crisis. That proactive stance is rare — and it's probably why Stripe continues to be the dominant player in online payments. Luna: So for any CTO listening who's staring at a monolith that's starting to creak — what's the first step? Lucas: Find the smallest, safest piece you can extract. Something read-only if possible, like a status endpoint or a reporting query. Build a new service for it, run it in parallel, compare results, and gain confidence. Then pick the next piece. Don't try to eat the whole elephant at once. Luna: And give your engineers something fun to work with. Rust was a draw for Stripe's team. Lucas: That helps too. A migration is a chance to modernize your tech stack, not just your architecture. If you can use a language or framework that excites your engineers, you'll get more buy-in and better outcomes. Luna: Interesting. So Stripe's migration is a case study in technical leadership — balancing risk, speed, and team motivation. Lucas: Exactly. And it's ongoing. They're still migrating some long-tail operations. But the core payments path is fully on Rust now, and the results speak for themselves. Luna: I look forward to seeing what they tackle next. Lucas: Same here. For The CTO Podcast, I'm Lucas. Luna: And I'm Luna.