Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Robinhood Cut App Latency by 50 Percent
Transcript
- Lucas: So, Robinhood built its whole brand on speed. You open the app, you place a trade, and it feels instant. But behind that instant feel, there's a fascinating engineering story about cutting latency to the bone. Luna: Yeah, and I've heard they've had to make some pretty radical changes to get there. I'm curious what actually moves the needle when you're already that fast. Lucas: Well, and this is a good moment to mention something honest. We're able to make episodes like this every day because a small group of listeners chips in monthly through buy me a coffee dot com slash fexingo. It's not a big ask, but it literally keeps the lights on and pays for the time we put into the research. Luna: Yeah, and it keeps us ad-free, which is a big deal for us. We're not hawking VPNs or meal kits, so if that's worth something to you, even a couple bucks a month helps. Lucas: Exactly. And today's topic is a perfect example of the kind of thing we dig into. So, Robinhood's latency story really came to a head in 2021, during the GameStop frenzy. They had to halt trading, which was a PR disaster, but behind the scenes, their systems were struggling with unprecedented volume. That was the wake-up call to fundamentally rethink their architecture. Luna: So what did they actually change? I mean, you can't just throw hardware at a problem like that and expect the latency to drop. Lucas: Right. So, one of the big moves was replacing their order routing system. They moved from a Java-based monolith to a set of microservices, with the critical path rewritten in Rust. Rust gives you that low-level control over memory and threads without the garbage collection pauses you get in Java. That alone cut latency significantly. Luna: Rust for a trading system. That's a bold choice. I've heard it's got a steep learning curve, but for this kind of work, the performance payoff must be huge. Lucas: It is. And they also built a custom in-memory matching engine. Instead of going through a traditional database for every order, they keep the order book in memory and use a highly optimized matching algorithm. That's where the fifty percent latency reduction really came from. They went from around a hundred milliseconds to under fifty for order placement. Luna: Fifty milliseconds still seems like a lot for a modern trading system, but I guess when you're dealing with millions of orders a day, every millisecond counts. Lucas: Exactly. And it's not just the placement. They also overhauled how they stream market data to the app. They moved from a rest based polling model to WebSockets for real-time updates. That cut the perceived latency for users, making the whole app feel snappier. Luna: So they're tackling latency on two fronts: the backend matching and the frontend data streaming. But what about the network? Even with fast code, you're still bound by physics. Lucas: That's where they got smart about colocation. They placed their servers in the same data centers as the exchanges, like the New York Stock Exchange's data center in Mahwah, New Jersey. That cuts the physical distance, so data travel time drops from milliseconds to microseconds. It's a classic move for high-frequency trading firms, but Robinhood's doing it for retail orders too. Luna: And that must have been a huge infrastructure shift. I can't imagine just picking up and moving your entire backend. Lucas: It was a massive project. They had to rearchitect their data replication and ensure they could maintain consistency while running in multiple data centers. They use a custom consensus protocol to keep their in-memory state in sync across sites, so even if one data center has a hiccup, they can failover without losing orders. Luna: That sounds like a distributed systems nightmare. How do they handle consistency when you have orders coming in on both coasts? Lucas: They use a leader-based approach. All orders for a particular stock go to a single leader node for that stock, so there's no split-brain. The leader processes the order, updates the state, and replicates to the followers. That ensures that every order is processed in a consistent order, even if there's a delay in replication. Luna: So they're trading some availability for consistency, which makes sense for a broker. But that must add latency, right? You're replicating every order to multiple sites. Lucas: That's the clever part. They use synchronous replication within a data center, but for cross data center, they use asynchronous, with a short window where they can failover. That way, they get the low latency of a single site, but with the durability of having copies elsewhere. It's a pragmatic tradeoff. Luna: And how does that hold up on a day like when GameStop spiked? I know they had to halt trading, but I'm guessing they learned a lot from that experience. Lucas: Oh, absolutely. The halt wasn't because their system crashed, but because they had to meet regulatory capital requirements. Their systems actually handled the volume, but the risk team had to step in. So they've since invested heavily in making sure that side is automated and scalable. Luna: So the technology was fine, it was the risk management that needed an upgrade. That's a good reminder that it's not all about speed. Lucas: Right. And they've also improved their monitoring and observability. They use a distributed tracing system so they can see where every order spends its time, from the app to the matching engine and back. That's how they found the bottlenecks in the first place. Luna: It must be hard to debug a system that's processing millions of orders a day. You can't just add a log statement and hope for the best. Lucas: They've built a sophisticated testing pipeline, including chaos engineering. They inject failures into their systems on a regular basis to make sure they can handle outages gracefully. They even have a 'game day' where they simulate a massive market event to see how the system reacts. Luna: That's the kind of thing that separates the serious players from the wannabes. But let's get back to the performance numbers. You said fifty percent latency reduction. What's the actual target now? Lucas: They're aiming for under fifty milliseconds for order placement, and under ten milliseconds for market data updates to the app. That's pretty aggressive for a retail broker. And they've actually achieved that in their internal benchmarks, though real-world numbers can vary due to network conditions and the user's device. Luna: And what about the user experience? Does cutting latency actually change how people trade? I'm skeptical that a few milliseconds matters to your average Robinhood user. Lucas: It's more about perceived responsiveness. If you tap a button and see a confirmation in under a hundred milliseconds, it feels instant. If it takes half a second, you start to doubt whether the app is working. That can lead to user frustration and churn. So it's not just about high-frequency trading, it's about making the app feel alive. Luna: That makes sense. And I've noticed that Robinhood has been adding more advanced features, like options and crypto. Does that complicate the latency story? Lucas: It does. Different asset classes have different requirements. Crypto trading is 24/7, so they need systems that never sleep. Options are more complex, so they've built separate microservices for pricing and risk. But the core matching engine is shared, and they've designed it to be asset-agnostic. So they can add new instruments without rewriting the whole thing. Luna: So the architecture is really about flexibility and speed. You've got a high-performance core with modular extensions. It's like building a race car that can be reconfigured for different tracks. Lucas: That's a great analogy. And the team is constantly tuning that race car. They're experimenting with new algorithms, like using field-programmable gate arrays for some of the matching logic. FPGAs can process orders in nanoseconds, but they're expensive and hard to program. So they're testing them for specific high-volume stocks. Luna: FPGAs. That's getting into serious high-tech territory. I can't imagine the skill set required to maintain that. Lucas: It's a niche area, but they've hired some of the best in the industry. And they're also looking at using machine learning to predict order flow and pre-position resources. It's a constant arms race to be faster. Luna: But at some point, you hit the laws of physics. You can't beat the speed of light. So what's the ultimate limit for latency in a trading system? Lucas: Well, if you're co-located with the exchange, you're looking at microseconds for the network hop. The main bottleneck is the software processing time. So the ultimate limit is probably a few microseconds per order, if you're using custom hardware and a tightly optimized codebase. But that's not realistic for a retail broker with millions of users. So they're aiming for the sweet spot where the perceived performance is excellent, but they're not over-engineering. Luna: It's a balance between performance and cost. You could build a system that's ten times faster, but it would cost ten times more to run. That's not a good business decision. Lucas: Exactly. And that's what makes this a great business and technology story. They're not just chasing speed for speed's sake. They're being strategic about where to invest their engineering resources to get the most bang for the buck. And the result is a platform that's both fast and economically viable. Luna: So, what can other companies learn from Robinhood's approach? Is there a universal takeaway? Lucas: The big one is that you need to measure before you optimize. They spent a lot of time building observability and understanding where their latency was actually coming from. Then they made targeted changes to the critical path. It's not about rewriting everything at once. It's about identifying the bottlenecks and fixing them in the right order. Luna: And sometimes that means making hard choices, like moving to a completely new programming language. That's not something you do lightly. Lucas: Absolutely. They took a risk with Rust, but it paid off. And they did it incrementally, starting with the most performance-critical components. It's a lesson in measured engineering courage. Luna: Well, I feel like I've got a much deeper appreciation for what goes into making a trading app feel instant. It's not just a pretty UI. Lucas: Exactly. And next time you tap that buy button, just think about all the nanoseconds that are flying around behind the scenes. It's a pretty remarkable system.