Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Spotify Rebuilt Its Recommendation Engine for Podcasts
Transcript
- Lucas: If these conversations are useful for what you're building or running, today's episode gets into one of the hardest recommendation engineering problems I've seen tackled in recent years. Luna: Which one? Lucas: Spotify's podcast recommendation engine. Not the music side — that's well documented. The podcast side is a fundamentally different problem, and they rebuilt it from scratch about two years ago. I want to understand what they did and why. Luna: Okay, I'm curious. Because when I think about recommending music, you have a dense signal — you know what songs a user listened to, skipped, saved, added to playlists. With podcasts, it's sparse. You might listen to one episode of a show and never come back, or binge ten hours. How do you even train a model on that? Lucas: That's exactly the core challenge. Before the rebuild, Spotify's podcast recommendations were basically bolted onto the music recommender. They used collaborative filtering on user-item matrices, treating a podcast episode like a song. But the listening patterns are totally different. Lucas: For music, repeat listening is common — people listen to the same track dozens of times. Podcasts? Almost zero repeat listening per episode. And the consumption session is much longer — thirty minutes to an hour, versus three minutes for a song. So the models kept recommending the same popular shows, and new or niche podcasts had no chance. Luna: Right, so the cold-start problem is brutal. If a show has only a few hundred listeners, how does the system even know what it's about? Lucas: Exactly. So the engineering team, led by a director of ML engineering named Ziad Sultan, decided to move from collaborative filtering to a two-tower neural network architecture. One tower encodes the user's listening history and preferences. The other tower encodes the content of the podcast episode itself — not just metadata, but the actual audio. Luna: Wait — the actual audio? They're processing spoken content? Lucas: Yes. They use a speech to text model to transcribe every episode, then run NLP embeddings on the transcripts. This gives them a dense vector representation of the content. So even if a podcast has zero listening history, the system can match it to users who have listened to similar topics. Luna: That makes sense. But transcribing every episode in real time must be expensive. How many podcasts are on Spotify now? Lucas: Over five million shows, and they add thousands every day. They don't transcribe everything in real time. They have a batch pipeline that processes new episodes within a few hours of upload. The transcription model is a fine-tuned version of Whisper — optimized for efficiency. They claim it costs about one cent per hour of audio. Luna: One cent per hour? That's cheap enough to scale. But then you have to store and index all those embeddings. What's the infrastructure look like? Lucas: They use a vector database — I believe it's a custom-built solution on top of Annoy, which Spotify open-sourced years ago. Each episode gets an embedding of 256 dimensions. For five million shows, that's about five billion floating point numbers. They shard across multiple nodes and use approximate nearest neighbor search to find candidates in under fifty milliseconds. Luna: Okay, so the content tower is clear. But the user tower — how do they represent a user's listening history? Especially when the signals are so sparse? Lucas: They aggregate the embeddings of all the episodes a user has listened to, weighted by completion rate and recency. Then they use an attention mechanism to give more weight to recent listens and episodes they finished. The output is a user vector. Then they compute the dot product between the user vector and the candidate episode vectors, and rank by similarity. Luna: So it's basically a semantic matching system. But what about the behavioral signals — like, do they use skip rates or rewind patterns? Lucas: That's where it gets interesting. They do incorporate implicit feedback. If you skip within the first thirty seconds, that's a negative signal. If you listen to more than seventy percent, it's positive. They also look at rewind patterns — if you rewind to replay a segment, that indicates high engagement. All of that feeds into the training labels. Lucas: The model is trained on a massive dataset: over 500 million daily listening events. They use a loss function called Bayesian Personalized Ranking, which optimizes for ranking the episodes you listened to higher than random episodes you didn't. Luna: 500 million events a day — that's a lot of data. How do they handle the training pipeline? Is it batch or online? Lucas: They do daily batch retraining. Online learning would be too noisy given the sparse feedback. Each day they compute new user embeddings based on the previous day's listens, then update the model weights. The training runs on a cluster of TPUs — about 32 TPU v3 cores per run. They told me the full training cycle takes about four hours. Luna: That's surprisingly fast for a model trained on half a billion events. But let me push back on something. Luna: If the system is purely matching content embeddings, doesn't it just recommend more of the same? Like, if I listen to one true crime podcast, I'll only ever see true crime. Where's the serendipity? Lucas: That's a great point, and they actually addressed that. They added an exploration component — a small percentage of recommendations come from a random sample of popular episodes in broader categories. They also use a diversification algorithm that limits how many episodes from the same show or genre can appear in a user's feed. Lucas: Additionally, they have a separate model for 'because you listened to X' that finds episodes with similar listener patterns, not just similar content. So if two shows are listened to by the same cohort of users, they'll get cross-recommended even if the topics differ. Luna: That's a hybrid approach — content-based and collaborative filtering combined. So they didn't completely abandon collaborative filtering, they just moved it to a different layer. Lucas: Exactly. The two-tower model handles the primary ranking, but they blend in results from a separate co-visitation matrix. That matrix captures which episodes are frequently listened to by the same users in a session. It catches patterns like 'people who listen to a tech news podcast also listen to a comedy podcast about tech.' Luna: So the engineering challenge isn't just building the model — it's orchestrating all these data pipelines, the vector search, the training infrastructure, and the serving layer. That's a lot of moving parts. Lucas: Absolutely. And the serving layer is critical because recommendations have to be fast. If a user opens the Spotify app, they expect their podcast feed to load in under a second. So the model inference is done server-side, precomputing the top 200 recommendations for each user every few hours. Lucas: They cache those in a distributed key-value store — I believe they use Cassandra. When the user opens the app, the feed is fetched from cache. The actual ranking happens offline, but the freshness is within a few hours. Luna: That makes sense for a daily refresh. But what about when a new episode drops from a show I subscribe to — does that get surfaced immediately? Lucas: Subscriptions are handled separately — those are deterministic. The recommendation model only affects the 'discover' section. New episodes from subscribed shows appear instantly via a separate pipeline. Luna: Okay, so the architecture is clear. But how do they measure success? Is it just listening time, or are there other metrics? Lucas: They track several metrics. Primary is listener session length — do people listen longer? Secondary is diversity — are they recommending a broader range of shows? And they also measure the number of first-time listens for new shows. Since the rebuild, they've seen a 20 percent increase in podcast listening time, and a 30 percent increase in the number of shows discovered per user. Luna: That's a significant improvement. I wonder how much of that is due to the model versus the fact that podcast content itself has grown. Lucas: It's hard to isolate, but they did A/B tests — the new model outperformed the old collaborative filtering baseline by 15 percent on listening time, holding content constant. So the architecture matters. Lucas: And you know, talking about what keeps these shows going, a couple of dollars a month is genuinely what keeps these episodes ad-free. If you've gotten something out of this deep dive into recommender systems, consider supporting at buy me a coffee dot com slash fexingo. Luna: Yeah, it really helps. We don't run ads, and that's because of listeners who chip in. It's that simple. Lucas: So back to the architecture — one more piece I find fascinating is how they handle the cold start for brand new shows. If a podcast has zero listening history, the content embedding is all they have. But they also use a 'podcast category' embedding that's pre-trained on similar shows. So if a new show is categorized as 'technology', it gets a head start from the aggregate embedding of all tech shows. Luna: That's a smart bootstrapping technique. But what about the user cold start? A brand new user with no history? Lucas: They use a onboarding flow where the user selects a few interests — topics like 'true crime', 'comedy', 'news'. Then the system generates an initial user vector as a weighted average of the category embeddings. After the first few listens, the model takes over. Luna: So basically, the system is a stack of embeddings — content, category, user — all learned and combined. It's elegant, but the complexity is high. Lucas: Right. And it's not perfect. They still have issues with stale recommendations — if a user's interests change, the model can lag by a few days. They're working on a online learning layer that updates user embeddings in real time based on the last hour of activity. That's the next frontier. Luna: That would be a huge engineering challenge — real-time model updates at Spotify's scale. But it would make the recommendations much more responsive. Lucas: Exactly. And it's a reminder that even after a major rebuild, the work is never done. Recommender systems are living products.