Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Spotify Migrated to Google Cloud Without Breaking Discovery Weekly
Transcript
- Lucas: When Spotify announced in February 2016 that it was moving its entire infrastructure to Google Cloud, a lot of people thought they were crazy. I mean, this was a company serving over 100 million active users at the time, with a real-time streaming product that basically cannot have any hiccups. Luna: Right, because if the music skips for even a second, users notice immediately. It's not like a website where a page load takes an extra 500 milliseconds and maybe nobody complains. Lucas: Exactly. And the migration ultimately took four years — they only shut down the last of their own data centers in 2020. So this is a story about how you move one of the largest music platforms on earth while it's running at full speed. Luna: Honestly, if this episode helps you think about your own infrastructure challenges, that's the kind of thing where we'd love your support. If today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Lucas: Yeah, listener support is what keeps this show ad-free and focused on real engineering stories. And speaking of real stories, let's get into how Spotify actually pulled this off. Lucas: Spotify's original infrastructure was built on-premise, running on a mix of bare metal servers and some virtualization. They had data centers in places like Stockholm, London, and San Jose. And by 2015, they were running something like 1,200 different microservices. Luna: That's a lot of services for a company that wasn't even ten years old at that point. Did they have a monolith before? Lucas: They actually started with a more monolithic architecture, but around 2013 they began breaking things into services because the engineering team was growing fast and they needed to decouple deployments. So by the time the migration decision came, they already had this pretty complex distributed system. Luna: What was the actual catalyst? Was it cost, or scalability, or something else? Lucas: It was mostly about growth — they realized that their own data centers couldn't scale at the rate they needed. They were adding users in new regions all the time, and building out new data centers from scratch would take months. Google Cloud gave them access to global regions immediately. Luna: But also, Google had some unique technology that Spotify wanted to use, right? Like Bigtable and Pub/Sub? Lucas: That's right. In particular, Google's Cloud Bigtable — which is their managed NoSQL database — was a big draw. Spotify was already using Cassandra heavily, and Bigtable offered a similar model but with better operational simplicity. And they wanted to use Google's network backbone for low-latency audio delivery. Luna: So they didn't just lift and shift. They actually re-architected some parts to take advantage of Google-specific services. Lucas: Correct. And that's one of the lessons here — they treated the migration as an opportunity to modernize, not just a data center move. For example, they built a custom tool called 'Sisyphus' — named after the Greek myth of pushing a boulder up a hill — to handle the incremental migration of their storage systems. Luna: Sisyphus? That name tells you something about how hard it was. Lucas: Exactly. Sisyphus was designed to continuously copy data from their on-premise Cassandra clusters to Google Cloud Bigtable in near real time. It would replicate every write, and if there was a conflict or inconsistency, it would flag it for manual resolution. That way, they could migrate one service at a time without a big cutover. Luna: So they were basically running dual infrastructure for years — both their own data centers and Google Cloud simultaneously. Lucas: Yes, and that's expensive. But it's the only way to do it safely. They had a rule: no service could be migrated until it had been running on both platforms in production for at least a month, passing all the same traffic, and with monitoring showing no degradation. So for each of those 1,200 services, there was a multi-week validation period. Luna: That sounds incredibly tedious. How did they prioritize which services to move first? Lucas: They started with the simplest, stateless services — things like API gateways and content delivery middleware. Services that could be killed and restarted without data loss. Then they moved on to stateful services, like the playlist storage and user profiles. And the very last thing they moved was the audio streaming pipeline itself, because that's the most latency-sensitive part. Luna: What about Discover Weekly? That's one of Spotify's most beloved features, and it's heavily dependent on machine learning models that process huge amounts of listening data. Lucas: Discover Weekly was one of the most complex parts of the migration. The recommendation engine runs on Google Cloud's Dataflow — which is their stream and batch processing service — but it's also tightly coupled with their storage layer. The team had to rebuild the entire ML pipeline to work with the new infrastructure, and then they had to validate that the recommendations didn't get worse during the transition. Luna: So they had A/B testing to compare recommendations from the old and new systems? Lucas: Precisely. They ran shadow tests where a percentage of users would get recommendations generated by the new pipeline, and they measured metrics like engagement and user satisfaction. If the new pipeline wasn't performing as well, they'd roll back the changes for that user segment and debug. Luna: That level of caution is impressive. But it also means the migration took forever. Four years is a long time. Lucas: It is, but Spotify's CTO at the time, Oskar Stål, said that they never had a major outage during the migration. That's a remarkable track record. The key was that they never forced a cutover. They let each team work at their own pace, as long as they met the validation criteria. Some teams took six months, some took a year. Luna: Did they run into any unexpected issues? I remember reading about some problems with Google's network configuration early on. Lucas: Yeah, one of the early hiccups was around virtual private cloud networking. Spotify's services are extremely chatty — they have internal RPC calls happening constantly — and the default Google Cloud VPC quotas weren't designed for that kind of traffic. They had to work with Google to increase limits and tune the network topology. Luna: So it wasn't just a technical migration, it was also a relationship management exercise with Google. Lucas: Absolutely. They had dedicated support from Google's customer engineering team, and they were one of Google Cloud's flagship customers. That meant they got early access to features and direct engineering time. But they also had to be careful not to over-couple their architecture to Google-specific APIs. Luna: Right, because if you build too deeply into Bigtable, say, and then later want to move to another provider, you're stuck. Lucas: Exactly. Spotify tried to abstract some of their data access layers. For example, they built a generic storage interface that could work with both their old Cassandra and new Bigtable. That way, if they ever needed to switch again, they could. Luna: What's the final outcome? Now that it's been a few years, are they happy with the move? Lucas: By all accounts, yes. Spotify has been able to launch in new markets much faster because they can spin up services in any Google Cloud region. They've also reduced their operational overhead — they no longer have to manage physical servers or worry about data center capacity planning. And they've been able to adopt newer Google services like BigQuery for analytics and Vertex AI for machine learning. Luna: So the four-year pain was worth it for the long-term flexibility. Lucas: That's the takeaway. And I think the lesson for other engineering teams is: don't underestimate the complexity of a cloud migration, but also don't be afraid to use it as a catalyst for modernization. Spotify didn't just move servers; they rethought their architecture, their data pipelines, and their operational practices. Luna: And they kept Discover Weekly running the whole time. That's the real miracle. Lucas: Yeah. It's easy to forget that behind every feature like Discover Weekly is an infrastructure story that took years to build. And now, when you open Spotify and your playlist is ready, you're seeing the result of thousands of engineering decisions. Luna: Makes you appreciate the next song a little more. Lucas: It really does.