Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Monzo Rebuilt Its Core Banking Engine for Real-Time
Transcript
- Lucas: So Monzo, the UK digital bank, quietly rebuilt the core engine that powers every single transaction and balance check. Not a front-end redesign — the actual ledger. Luna: That sounds terrifying for a bank with seven million customers. What drove them to touch the core? Lucas: The old system was built on a traditional batch-processing model. When you checked your balance, it wasn't real-time — it was the last settled snapshot. As Monzo grew, the latency became a competitive problem. Luna: So they wanted instant balance updates, but banking is supposed to be the domain of double-entry accounting and immutability. How do you reconcile speed with correctness? Lucas: Exactly. They made a very specific architectural bet: use Apache Kafka as the single source of truth for all transaction events, and build a stream processor that derives account balances on the fly. No more settling in batches overnight. Luna: That's a radical shift. Traditional core banking systems are relational databases with ACID transactions. Kafka is log-based, eventually consistent. Did they sacrifice consistency? Lucas: They didn't. They implemented what they call 'eventual consistency with immediate visibility'. The key insight is that the event log itself is immutable — once a transaction event is committed to Kafka, it's permanent. The balance is a materialized view of that log. Luna: So they're effectively running a stateful stream processor in front of Kafka to compute balances. That's a huge engineering lift. What was the migration like? Lucas: They did it gradually, running both systems in parallel for over a year. Every transaction was written to both the old batch ledger and the new Kafka log. They had reconciliation jobs running every few seconds to compare outputs. Luna: That must have been expensive — double the infrastructure, double the operational load. Lucas: It was, but they treated it as a safety net. Monzo's engineering blog mentions they caught several edge cases during the dual-run phase — things like race conditions in the stream processor when a customer had multiple transactions hitting the same account in milliseconds. Luna: Right, because in a streaming model, ordering matters. If two debits arrive in the wrong order, the balance could temporarily go negative. Lucas: Precisely. They solved that by keying every account to a specific Kafka partition, so all events for that account are processed in strict order by a single stream processor instance. It's a classic partitioning strategy — but with the twist that they also had to handle rebalancing when partitions split. Luna: And the performance gains? What did real-time actually get them? Lucas: Monzo now processes over 500 transactions per second at peak, with balance updates visible to the customer in under 50 milliseconds. That's from card swipe to app notification. The old batch model had a lag of minutes, sometimes hours. Luna: And they achieved that while maintaining the same regulatory requirements for audit trails and financial reconciliation? Lucas: Yes, because the Kafka log itself is the audit trail. Every event has a unique ID, a timestamp, and a cryptographic hash linking it to the previous event. Regulators can replay the entire history from the log. In fact, they argue it's more auditable than a database where data can be overwritten. Luna: That's a compelling argument. But what about the human side — the engineers who had to learn stream processing? Lucas: Monzo invested heavily in internal training. They had a dedicated platform team that built abstractions on top of Kafka — essentially a framework where product engineers could define business logic in a declarative way, without having to manage partition assignment or offset commits. Luna: So they treated the complexity of stream processing as infrastructure, not something every team needed to master. Lucas: Exactly. And they invested in tooling to simulate failures — they'd deliberately kill Kafka brokers during non-peak hours to make sure the system recovered gracefully. That's a sign of engineering maturity: not just building the system, but hardening it. Luna: I'm curious about the downside — what are they missing by not having a traditional database as the source of truth? Lucas: The big one is ad-hoc queries. If a product manager wants to run a complex SQL query across all accounts, they can't just query the Kafka log directly. Monzo built a separate analytics pipeline that materializes the event stream into a columnar store for reporting. That's an extra system to maintain. Luna: So they traded the convenience of a relational database for real-time accuracy. That's a deliberate trade-off, and one that seems to have paid off for their use case. Lucas: It's a pattern we're seeing more of in fintech. The question is whether it scales to larger incumbents with hundreds of millions of accounts. Monzo is still relatively small in the banking world. Luna: And that's exactly the kind of conversation we love having on this show — where operators and builders can learn from real engineering decisions. Lucas: Look, if you're a CTO or a senior engineer, these deep dives into architecture are the reason we record every week. We deliberately don't run ads on these episodes. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to keep the conversations ad-free and focused on the technical details that matter. Lucas: Back to Monzo — one thing I find fascinating is how they handled the concept of 'available balance' versus 'actual balance'. In a streaming system, you can see a pending transaction before it settles. Luna: Right, because the event log contains both authorized and settled events. So they can show you a pending debit immediately, even before the merchant batch settles. Lucas: Exactly. That's a better user experience, but it also introduces new financial nuances. For example, if a merchant authorizes a transaction but never settles it — like a hotel deposit that gets released — the system needs to handle that expiration correctly. Luna: So they had to model the lifecycle of a transaction as a state machine within the stream. That's non-trivial. Lucas: It is. And they did it using a combination of Kafka Streams and a lightweight state store backed by RocksDB. Each account has a tiny state machine that tracks all pending authorizations and their expiration timers. Luna: That's clever. It also means they can offer features like instant refund notifications — the moment a refund event hits the log, the balance updates. Lucas: Yes, and that's become a competitive differentiator. Monzo's app shows you a real-time feed of transactions with merchant logos and maps. That's all driven from the event stream. Luna: So the architecture directly enables the product experience. That's the kind of alignment every CTO dreams of. Lucas: It's a great example of why 'just use a database' isn't always the right answer. For Monzo, the database became a bottleneck, and the stream became the engine. Luna: And the risk of that approach is that you're building infrastructure that most banks haven't figured out yet. That's both an opportunity and a liability. Lucas: Right. But Monzo's bet seems to be paying off. Their system handles peak loads during Black Friday and New Year's Eve without hiccups. And they've open-sourced some of their tooling, which gives back to the community. Luna: I think that's the takeaway — sometimes the right engineering decision is to challenge a decades-old pattern, even in a regulated industry. Lucas: And the proof is in the production metrics: sub-50-millisecond balance updates, 500 transactions per second, and a system that's been running for over two years without a core outage. That's real evidence. Luna: I'd love to see how they handle the next order of magnitude — say 50 million accounts. Lucas: That's the next episode. Maybe we'll invite one of their engineers to walk us through it.