Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Spotify Rebuilt Its Recommender System for 600 Million Users
Transcript
- Lucas: If these conversations are useful for what you're building or running, I'm glad. Today, I want to look under the hood of how Spotify rebuilt its recommender system for six hundred million users. Luna: Six hundred million — that's a huge number. I remember their old system was pretty batch-oriented, right? Nightly model retraining on Spark? Lucas: Exactly. Up until about 2022, Spotify's main recommendation pipeline was built on Apache Spark with collaborative filtering — think alternating least squares, user-item matrices. They'd train a model overnight, push it to production, and that model would serve recommendations for the next twenty-four hours. It worked well enough when they had a hundred million users, but by the time they hit four hundred million, the limitations became obvious. Luna: The biggest one being cold start? New songs or new users would have to wait a full day before the model even saw them? Lucas: Right. A new release from an indie artist would get zero recommendations until the next day's model run. And user behaviour changes in real time — someone listens to three hip-hop tracks in a row, the old system couldn't adapt until tomorrow. So Spotify's machine learning team decided to move to a real-time graph neural network architecture, or GNN for short. Luna: I've heard GNNs are the hot thing in rec sys right now. But what made them choose graphs over something like transformers or deep learning on sequences? Lucas: Great question. The key insight is that user listening behaviour is inherently graph-structured. You have users, songs, albums, artists, playlists — all connected by listen events, skips, saves, shares. A graph neural network can naturally capture these relationships by propagating information along the edges. Transformers treat everything as a flat sequence, but a graph lets the model learn that two songs are similar because they share listeners, or because they appear in the same playlist. That's hard to capture with sequential models alone. Luna: Makes sense. So how did they actually replace the Spark pipeline? Lucas: They built a streaming pipeline using Apache Kafka for event ingestion, then a TensorFlow-based GNN model that updates embeddings in near real time. Instead of a single nightly training job, they have a continuous training loop that ingests user actions as they happen and updates the graph embeddings incrementally. The model itself is a two-layer GraphSAGE architecture — a variant of GNN that samples neighbors to keep computation manageable. Luna: Wait, incremental updates — doesn't that mean they're retraining the entire graph every time a new event comes in? That sounds computationally insane. Lucas: That's the clever part. They don't retrain the whole graph. They use a technique called 'graph streaming with stale embeddings'. The core idea is that the graph structure changes slowly — users add songs to playlists, artists release new tracks — but user interactions change fast. So they maintain two sets of embeddings: a static set that gets recomputed every few hours for the structural part, and a dynamic set that updates in real time for user preferences. The GNN then combines both during inference. Luna: So cold start latency dropped from hours to seconds? Lucas: Under thirty seconds, according to their engineering blog. A new song gets uploaded, within half a minute the graph has created an initial embedding based on its audio features and the artist's existing connections, and it starts appearing in recommendations. For new users, they use a lightweight onboarding model that asks a few questions about genre preference, then maps those answers to graph embeddings instantly. Luna: That's impressive. But I have to imagine the compute cost went up significantly. Real-time inference on a GNN for six hundred million users — that's not cheap. Lucas: You're right. Spotify published some numbers — they're running the inference on a fleet of GPU instances, and the total cost per recommendation is about three times higher than the old batch system. But they argue the improvement in user engagement — things like session length and discovery rate — more than justifies it. They A/B tested the new system against the old one and saw a twelve percent increase in weekly active users discovering new music. Luna: Twelve percent — that's huge for a mature platform. I'm curious about the A/B testing itself. How do you even test a recommender change when the model influences what users see, which influences the training data for the next iteration? Lucas: That's the classic exploration versus exploitation problem. Spotify uses what they call 'interleaved experiments' — instead of showing one user the new model and another the old, they show each user a mix of recommendations from both models, then track which ones they click or listen to. This gives a much cleaner signal because it eliminates user-level variance. They also run long-term holdout experiments where they freeze a small percentage of users on the old model for weeks to measure delayed effects like churn or discovery fatigue. Luna: Interesting. Let me ask about the engineering org side — how did they structure the teams to pull this off? Lucas: Spotify is famous for squads, but for this initiative they created a 'Recommendation Platform' tribe with about sixty engineers split into six squads. Each squad owned a piece of the pipeline — data ingestion, graph construction, model training, inference serving, experiment platform, and monitoring. The key decision was to make the squad boundaries aligned with the data flow, not the model components. So the squad that owned graph construction also owned the schema for listener events, which forced them to think about data quality from the start. Luna: That makes a lot of sense. If you align squads with model layers, you end up with handoff problems. Aligning with data flow means each squad has an end to end view of their part of the pipeline. Lucas: Exactly. And they also had what they call 'you build it, you run it' — each squad was responsible for the production reliability of their component. So the inference serving squad had on-call rotation for the GPU inference cluster. That created strong incentives to build robust systems with good monitoring and gradual rollout capabilities. Luna: Speaking of gradual rollouts — how did they migrate from the old Spark system without disrupting the user experience? Lucas: They used a technique called 'shadow scoring'. For several months, every recommendation request was sent to both the old and new models in parallel. The old model's output was served to users, but the new model's output was logged and compared offline. They validated that the new model's recommendations were at least as good on metrics like click-through rate and diversity. Only after they had weeks of consistent improvement did they start serving the new model to a small percentage of users, ramping up over a month. Luna: Shadow scoring — that's a smart way to de-risk a model change. And it avoids the problem of taking a gamble on user experience. Lucas: Right. One more thing — they also had to handle the fact that the graph itself is constantly growing. New songs, new users, new playlists. They built what they call a 'graph compaction' process that runs nightly and prunes stale nodes — for example, songs that haven't been listened to in six months get removed from the active graph but archived. This keeps the graph size manageable: they went from about two billion nodes to around eight hundred million active nodes. Luna: Eight hundred million is still enormous, but cutting more than half shows how much noise there can be. It's a good reminder that bigger isn't always better for ML. Lucas: Absolutely. And that's actually a good segue — one of the design principles they emphasized was 'respect the sparse signals'. Most recommendation systems focus on popular items, but Spotify wanted to surface niche content. The GNN architecture inherently handles sparsity better because it can propagate information from similar users or songs even if the direct interaction history is thin. Luna: You know, this level of depth in engineering decisions is exactly why people listen to this show. And it's also why a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo — that's literally what funds making this many of these episodes. Lucas: Yeah, it's a small group, but it keeps the show ad-free and lets us spend time on the real technical details. So thank you to anyone who's part of that. Luna: Alright, back to Spotify. One thing I want to understand better — how did they handle the training data bias? If the model is trained on user behaviour, it tends to reinforce existing preferences, not encourage exploration. Lucas: Great point. They added an explicit exploration component. During training, they inject random 'exploration noise' into the graph — a small percentage of edges are randomly rewired to simulate discovery. And during inference, they blend the GNN's top recommendations with a separate 'diversity sampler' that picks from less likely candidates. The blend ratio is adaptive: for users who have a high diversity score — meaning they listen to many genres — the exploration weight is higher. Luna: So the model itself doesn't just optimize for click-through rate; it's optimizing for long-term user satisfaction. Lucas: Exactly. They measure something called 'session satisfaction' — not just whether the user clicked, but whether they listened to the entire song, saved it, or came back later. That's a much better proxy for true engagement than a raw click. And the GNN architecture turned out to be particularly good at predicting session satisfaction because it captures the context of the listening session — what was played before, what came after, the time of day. Luna: I love that they're thinking beyond short-term metrics. A lot of companies optimize for the wrong thing and end up with echo chambers. Lucas: Spotify has actually published research showing that their old collaborative filtering system was creating filter bubbles — users were getting stuck in a narrow set of genres. The new GNN system increased the diversity of recommendations by about thirty percent measured by the number of unique genres per user per week. That's a meaningful change. Luna: Thirty percent is huge. So the architecture decision had a direct product impact. That's the kind of story that makes a CTO's case for investing in infrastructure. Lucas: For sure. And the timeline is worth noting — from initial research to full production rollout took about eighteen months. They started with a small team of four engineers exploring GNNs, then grew to sixty once they had proof of concept. That's a classic approach: incubate with a small group, then scale the team once you have conviction. Luna: What's next for them? Are they already thinking about the next generation? Lucas: They're exploring what they call 'multimodal graphs' — incorporating not just listening behaviour but also audio features, lyrics, and even social signals like shares and comments. The idea is to build a richer representation of songs that doesn't rely solely on user interactions, which helps even more with cold start. They're also looking at online learning where the model updates continuously without any batch retraining at all. Luna: That would be the holy grail — a model that learns from every single user action instantly, no latency at all. But the engineering challenges must be immense. Lucas: Definitely. Consistency, staleness, resource usage — all those problems get harder. But if anyone can do it, the team that already moved from batch to near real time has a head start. I think we'll see more platforms adopt graph-based recommenders in the next few years. It's not just Spotify — Pinterest, Uber, and even some e-commerce companies are moving this direction. Luna: It makes sense. The world is graphs, whether we model them that way or not. Spotify just decided to stop pretending otherwise. Lucas: Exactly. And that's the takeaway I hope listeners hold onto: the best architecture often comes from choosing a representation that matches the actual structure of your problem. Spotify's problem is a graph, so they built a graph. Simple in hindsight, but it took years of batch thinking to get there.