Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Shopify Slashed Build Times with Remote Caching
Transcript
- Lucas: So Shopify's engineering team publishes over a thousand code changes every single day. And just a few years ago, each one of those changes could take over an hour to build. Luna: An hour just to compile? That would drive me insane. How did they fix that? Lucas: They didn't just fix it, they cut the median build time down to under five minutes. The key was moving to Bazel and rolling out remote caching, which lets engineers share build outputs instead of rebuilding everything from scratch. Luna: So the first person builds it, and everyone else just downloads the result? That sounds almost too good to be true. Lucas: It is, in practice, but it took a massive cultural shift. They had to migrate hundreds of services and tens of thousands of files into a single monorepo, and they had to get their developers to buy into a completely different way of working. Luna: What was the old setup? Why were builds so slow in the first place? Lucas: They were using a polyrepo with a custom build tool that didn't do any caching. Every developer, every CI job, every branch essentially compiled the whole world on its own. Multiply that by a thousand changes a day, and you've got a lot of wasted compute and a lot of frustrated engineers waiting on progress bars. Luna: So they went all-in on monorepo. That's a big bet. What made Bazel the right call? Lucas: Bazel gives you hermetic builds and fine-grained caching out of the box. It tracks every input and output, so if you've built a service once, the next time it uses the same inputs, it just pulls the cached result. And with remote caching, that cache is shared across everyone. Luna: But moving to a monorepo with Bazel is not exactly a weekend project. How did they sell it internally? Lucas: They started with a small team of engineers who were feeling the pain most, and they got them to prototype the migration on a few services. Once those teams saw their build times drop from forty minutes to four, word spread pretty fast. Luna: Word of mouth from engineers who actually hate waiting. That's the best marketing you can get. Lucas: Exactly. But it wasn't just about the tool. They had to build a whole culture around developer experience. They treat their internal tooling as a product, with a dedicated team that listens to feedback and iterates on it. Luna: So what was the actual infrastructure like? How do you scale a remote cache for a company that size? Lucas: They run a cluster of Bazel remote cache instances, backed by object storage, and they've tuned it to handle peak load during business hours. The cache hit rate is typically above ninety percent, which means most builds are just downloading artifacts rather than compiling. Luna: Ninety percent hit rate. So the first build is the expensive one, and everyone else gets a free ride. Lucas: Right. And that first build is still way faster than before because Bazel only builds what changed, not the whole graph. They went from building everything, every time, to building only the services affected by a change. Luna: That's a fundamental shift. But I imagine it was painful to get there. What were the biggest hurdles? Lucas: The migration itself was the biggest hurdle. They had to move hundreds of repositories into a single monorepo, and they had to handle conflicts, merge strategies, and a lot of legacy code that didn't comply with Bazel's strict rules. Luna: How did they not break everything? Lucas: They did it incrementally. Instead of a big bang migration, they built what they call an 'intermediate mode' where they could run Bazel alongside their old tool. That let teams move over one at a time, and they could roll back if something went wrong. Luna: So you're telling me they kept two build systems running in parallel? That sounds like a nightmare to maintain. Lucas: It was, but it was the safest way. They also automated a lot of the migration using custom codemods, so engineers didn't have to manually rewrite their build files. And they had a dedicated migration team that supported each service as it moved over. Luna: It's almost like a surgery, you don't just rip out the heart and replace it, you keep the patient alive the whole time. Lucas: Exactly. And the payoff has been massive. Beyond the build time, they've seen a big drop in CI compute costs, because they're not recompiling the same code over and over. They estimate they've saved tens of millions of dollars in infrastructure spend. Luna: Tens of millions. That's not just developer happiness, that's a business case. Lucas: Which is probably why they've been able to keep investing in it. And the lesson for other CTOs is that developer experience is not a soft metric. It's directly tied to your bottom line. Luna: So if someone's listening and their build is taking twenty minutes, what's the first step they should take? Lucas: First, measure. You can't fix what you don't understand. Look at your build graph, identify the biggest bottlenecks, and see where you can introduce caching at a smaller scale. You don't need to go full monorepo overnight. Luna: But if you're starting from scratch, would you recommend Bazel or something else? Lucas: Bazel is powerful but it has a steep learning curve. For smaller teams, tools like Nx or Turborepo might be a faster win. But if you're at Shopify's scale, Bazel is a proven path. Luna: And a proven path to saving tens of millions, which is nothing to sneeze at. Lucas: Right. And honestly, just the fact that they went from an hour to five minutes, that's the kind of thing that changes how a company ships. Developers can iterate faster, ship smaller changes, and catch bugs earlier. Luna: Speaking of catching bugs, does this also improve their CI pipeline? Lucas: Massively. Since builds are faster, they can afford to run more tests. They've expanded their test suite significantly, and they can run targeted tests on just the affected services, which gives them confidence without the wait. Luna: So they're not only faster, they're also more reliable. That's a winning combo. Lucas: And it's a great example of how a technical investment can ripple across the whole organization. It's not just about the build tool, it's about the culture of treating your developers as customers. Luna: If today was useful to you, the way these stay ad-free is listener support. You can buy us a coffee at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small way to keep the conversation going, and we really appreciate it. Now, back to Shopify. They also had to deal with the human side of this change, right? Luna: I imagine there was some resistance. You're telling engineers who've been doing things a certain way for years that they have to learn a new tool. Lucas: There was definitely pushback. Some engineers saw it as overhead, something that wasn't directly related to their product work. But the team won them over by showing real results and making the migration as painless as possible. Luna: What's the one thing you'd want a CTO to take away from this episode? Lucas: It's that build times are a signal, not a nuisance. If your developers are waiting, you're losing money and momentum. Investing in the tooling and culture to fix that is one of the best investments you can make. Luna: And it's not just for the tech giants. Even a small team can start with caching and see big wins. Lucas: Absolutely. Start small, measure, and scale. That's how Shopify did it, and that's how any team can start to chip away at their build times.