Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Shopify Cut Build Times with Remote Caching
Transcript
- Lucas: So the other day I was waiting on a build that took fifteen minutes, and I thought, there has to be a better way. And then I remembered Shopify's story of cutting build times by up to ninety percent with remote caching. That's the kind of win that changes how a team ships. Luna: Ninety percent? That's not just a little faster, that's a different ballgame. How did they pull that off? Lucas: Well, the core idea is simple: instead of each developer rebuilding the same dependencies and intermediate artifacts locally, you share them across a central cache. So when you push a commit, the build system checks what's already been built and only compiles what's actually changed. But the implementation is where it gets interesting. Luna: Right, because caching sounds easy until you have thousands of developers all pushing at once. Lucas: Exactly. Shopify had to build a cache that could handle massive scale. They use a distributed object store, and they rely on a content-addressable system where the cache key is a hash of the inputs. That way, if two developers build the same thing, they get the same key and can share the result. Luna: But what about cache invalidation? That's the classic nightmare. If you get that wrong, you're shipping stale artifacts and wondering why nothing works. Lucas: That's the crucial part. Their cache keys are incredibly precise. They hash not just the source files, but also the compiler version, the flags, the environment variables, even the operating system. So if anything changes, the key changes, and you get a miss. It's a bit more storage, but it's worth it for correctness. Luna: So they're trading storage for speed. That makes sense if storage is cheap and developer time is expensive. Lucas: And that's the trade-off they made. They estimated that every minute of build time saved per developer, times hundreds of developers, adds up to an enormous amount of productivity. So the storage cost is trivial compared to the time savings. Luna: It's like a compounding investment. The more developers you have, the more you benefit from the cache. Lucas: Right. And the funny thing is, they had to change their culture too. Developers had to trust the cache, and that meant having good tooling to inspect what was cached and why. They built a dashboard where you could see your cache hits and misses, and that transparency built confidence. Luna: So it wasn't just a technical fix, it was an organizational one. You had to get people to change their workflow. Lucas: Absolutely. And that's a lesson for any CTO: sometimes the biggest wins aren't about writing more code, but about removing friction. If your builds take forever, you're paying for it every single day. Luna: I'm curious, did they have any security concerns? I mean, if you're sharing build artifacts across the company, you're exposing source code or proprietary logic. Lucas: They did. They had to make sure the cache was only accessible within their network, and they used encryption at rest and in transit. Plus, the content-addressable nature means you can't accidentally access something you don't have the key for. It's like a secure library where only the right people have the card. Luna: That's a good analogy. So what's the takeaway for a smaller team? Can they benefit from remote caching too? Lucas: Definitely. Even a team of five can use something like a self-hosted cache or a service like Gradle Enterprise. The principles are the same: precise cache keys, shared storage, and a culture that trusts the cache. You don't need Shopify's scale to see big wins. Luna: It's like one of those things that seems like it's only for the big players, but the fundamentals are universal. And I love that they shared the details publicly, so we can all learn from them. Lucas: Before we get too deep into the technical weeds, I want to take a quick pause here. If you're finding these conversations useful for what you're building or running, and if today was worth a coffee to you, that's the link: buy me a coffee dot com slash fexingo. Just a small way to keep this show ad-free and independent. We'll get back to the caching story in a second. Luna: Yeah, it's a small gesture that goes a long way. And speaking of going a long way, that remote cache is exactly the kind of thing that pays off for years. Lucas: Exactly. One thing that surprised me was the impact on their CI system. With remote caching, they reduced the load on their CI runners dramatically, which meant they could run more parallel builds and reduce queue times. It's a virtuous cycle. Luna: So they weren't just saving developer time, they were also saving infrastructure costs. That's a double win. Lucas: Right. And they actually measured that too. They saw a significant drop in their CI usage, which translated to lower cloud bills. So it's a cost-saving measure that also improves developer experience. Luna: It's rare to find a change that makes developers happier and the finance team happier at the same time. Lucas: And that's why I think this is such a great case study for technical leaders. It's not just about the technology, it's about the metrics. They tracked build time percentiles, cache hit rates, and they used those to justify the investment. Luna: Data-driven decision making. That's the kind of thing that gets buy-in from the C-suite. Lucas: Exactly. So, what do you think is the biggest challenge for a team trying to implement this? I mean, beyond the initial setup. Luna: I'd say it's trust. If developers don't trust the cache, they'll be tempted to bypass it or do clean builds all the time, which defeats the purpose. So you need that transparency and tooling to build confidence. Lucas: That's a great point. And Shopify did that by making cache metrics visible to everyone. They also made it easy to manually invalidate the cache if something went wrong, which gave developers a safety valve. Luna: So it's about making the system feel safe and reliable, not just fast. Lucas: And that's the leadership lesson. You have to think about the human side of infrastructure changes. It's not just about the technical architecture, it's about how people interact with it. Luna: I think we could talk about this for hours, but let's wrap up with the big question: what's the one thing you want our listeners to remember? Lucas: If you're not using remote caching, you're probably wasting time and money. If you have a CI system, that's the first place to look. Even a small team can see huge gains. It's one of the highest-leverage changes you can make. Luna: Yeah, and it's a change that pays off every single day. Thanks for being here, everyone. See you next time.