Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Nubank Keeps Its Banking Platform Running
Transcript
- Lucas: Luna, you ever think about how we just expect our banking apps to work — like, never go down — but the engineering behind that is almost invisible. Luna: All the time, actually. Especially after that Nubank outage in 2023 that lasted maybe forty minutes and it was front-page news in Brazil. Lucas: Right. That tells you the standard. A forty-minute downtime is a scandal. So today I want to dig into how Nubank keeps its core banking platform running — because they're not just another fintech. They're a full digital bank with over a hundred million customers, and they run on their own stack, not some legacy mainframe. Luna: And they built that stack from scratch, starting around 2013. That's still fairly young for a bank. Lucas: Exactly. Their CTO, and the engineering team, made a bet early on that they'd own the entire transaction pipeline. They didn't want to bolt on top of some existing core banking system. So they wrote their own transaction engine in-house. Luna: That is a huge bet. I mean, companies like Thought Machine and Mambu exist to provide core banking as a service. Nubank said no, we'll build it. Lucas: And they did. The engine is called Nubank Core — not super creative naming — but it handles every debit, credit, transfer, and payment authorization in real time. And it's built on a microservices architecture, but with a twist: they run a kind of distributed monolith for the transaction path itself. Luna: A distributed monolith? That sounds like an oxymoron. Lucas: It kind of is. What I mean is, all the services involved in a single transaction — authorization, ledger update, fraud check, notification — they're deployed separately, but they communicate synchronously over a single request scope. No async queues in the critical path. That gives them consistency, but it means the whole chain is only as strong as its weakest link. Luna: So they need every single service to be rock solid under load. That's a different reliability challenge from, say, a social network where you can tolerate a few failed likes. Lucas: Totally different. A failed payment is money gone or a customer stuck. So Nubank invested heavily in what they call 'production readiness' — a formal process every service goes through before it can handle real traffic. It's like a security review, but for operational resilience. Luna: What does that include? Like chaos engineering? Lucas: That's part of it. They run a program they call 'Chaos Friday' — every other Friday, a rotating team chooses a production service and deliberately injects failures. Network latency, instance crashes, database connection pool exhaustion. They observe how the system behaves and fix the weak spots. Luna: And that's in production, not a staging environment? Lucas: Yes, but they're careful. They start with low-impact experiments — say, injecting fifty milliseconds of latency into one service — and escalate only after they've verified the system gracefully degrades. They've been doing this since 2019, and their mean time to recovery has dropped from about four hours to under thirty minutes. Luna: That is a massive improvement. And I know from talking to engineers there that the key wasn't just the tests — it was automating the recovery itself. Lucas: Right. They built an incident response framework called 'Rapid Response' that automatically detects anomalies, runs diagnostics, and in some cases triggers remediation without a human in the loop. For example, if the database replica lag exceeds a threshold, it automatically promotes a standby and reroutes read traffic. Luna: And that's where the engineering culture matters. A lot of teams would be terrified to let a machine make that call. Lucas: Absolutely. It takes trust in your monitoring and your runbooks. And if today was actually useful to you — a peek into how a real hundred million user bank operates under the hood — the way these episodes stay ad-free is directly through listener support. You can find that at buy me a coffee dot com slash fexingo. Luna: Yeah, it's how we keep the show running without sponsors. Really appreciate everyone who chips in. Lucas: So back to that database replica promotion — what makes it safe is they've practiced the scenario in Chaos Friday for months. They know the exact conditions under which promoting a replica is safe versus risky. Luna: And they've also invested heavily in schema change management. Because one of the biggest causes of bank outages is a bad database migration. Lucas: That's the other piece I wanted to highlight. Nubank treats database schema changes like code changes — they go through code review, they're tested in CI, and they're deployed using a tool they built called 'SchemaShip' that applies migrations in a phased, reversible way. Luna: I've heard of it. The idea is you never run an ALTER TABLE that locks the entire table. They add columns as nullable first, backfill data in batches, then set the NOT NULL constraint. Lucas: Exactly. And they do all of this while keeping the database online. They've gotten to the point where a schema migration almost never causes a blip in uptime. That's a huge engineering achievement. Luna: And it's not just the database — they also have a custom distributed tracing system they call 'TraceFlow' that gives them end to end visibility into every transaction. So when something does go wrong, they can pinpoint the exact service and the exact request. Lucas: TraceFlow is actually built on top of OpenTelemetry, but they added their own correlation logic to handle the synchronous chain I mentioned earlier. Every transaction gets a unique trace ID that follows it through all the microservices, and they store the full trace for thirty days. Luna: That must generate a massive amount of data. How do they query it quickly? Lucas: They built a custom indexing layer on top of Cassandra — they compress the traces and use a time-based partition key. So when an incident happens, the on-call engineer can query 'show me all traces that had a latency spike in the last ten minutes' and get results in under a second. Luna: That's the kind of investment that separates a bank from a startup. It's not glamorous, but it prevents the firefights. Lucas: Exactly. And it's worth mentioning that Nubank also runs a 'Game Day' once a quarter — a full simulation of a major disaster, like losing an entire AWS availability zone. They literally take down a zone and see if the platform survives. Luna: I imagine that gets pretty intense. Especially because they're a regulated bank — the Brazilian central bank has specific uptime requirements. Lucas: Yes, they need to maintain 99.95% uptime for their core services, or they face regulatory penalties. They actually target 99.99% internally, which gives them a buffer. And they've hit that for the past two years straight. Luna: That's incredible. I mean, 99.99% is about four minutes of downtime per month. For a system that processes millions of transactions a day. Lucas: And they've done it by weaving reliability into every layer — from the architecture to the culture to the tooling. It's not about any single silver bullet. It's a thousand small decisions that add up. Luna: So what's next for them? They've already conquered the core banking platform. What's the next frontier? Lucas: I think the next big push is around real-time fraud detection at the edge. They've been experimenting with running machine learning models directly on the payment authorization path — so the model scores a transaction in under ten milliseconds before it's even committed to the ledger. Luna: That's a whole new set of reliability and latency requirements. If the model takes too long, the customer gets a timeout. Lucas: Right. And they're solving that by deploying the models on GPUs inside their own Kubernetes clusters, not calling out to an external inference service. It's a fascinating example of how reliability evolves as the product grows. Luna: I'd love to see a deep dive on that in a future episode. Lucas: Noted. For now, the takeaway is that Nubank's engineering team has proven that a born-in-the-cloud bank can be more reliable than many legacy institutions — if you invest in the right disciplines from day one. Luna: And if you're willing to turn off an AWS zone on a Friday afternoon just to see what happens. Lucas: Exactly. Thanks for listening, and we'll catch you next time on The CTO Podcast.