Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How GitHub Migrated 100 Million Repos to a New Storage Layer
Transcript
- Lucas: If these conversations are useful for what you're building or running, stick around — we're going to talk about one of the most quietly ambitious infrastructure projects in the last few years. GitHub just finished moving over 100 million repositories off a storage system that had been stretched to its limit and onto something completely new. Luna: A hundred million repos — that's not just a technical migration. That's a decade and a half of data, every pull request, every commit, every fork. How do you even start moving something like that without breaking the thing everyone relies on? Lucas: That's exactly the question. And the answer involves a lot more than just copying files. Before we get into the weeds though — a quick note about this show. We deliberately keep these episodes free of sponsor interruptions. It's a choice we made early on because we think technical conversations like this deserve your full attention without a mid-roll ad break. If you find value in that, and you want to support the work that goes into researching and producing each episode, there's a way to do that. It's a simple, no-pressure link: buy me a coffee dot com slash fexingo. That's it. No tiers, no perks, just a way to keep the show ad-free. Luna: Yeah, I think that matters. A lot of podcasts start with good intentions and then the ads creep in. We wanted to avoid that, and listener support is what makes it possible. Lucas: Exactly. So, back to GitHub. The old system was essentially a layer on top of git's native storage — every repository stored as a pack file on a filesystem. That worked fine for a long time, but by 2020, the sheer volume of repos was creating real pain points. Luna: What kind of pain points? Were we talking about performance degradation, or something more structural? Lucas: Both, honestly. The filesystem had to handle billions of small files — pack files, index files, bitmaps. And the way git references objects internally, with SHA-1 hashes, meant that checking if an object existed required scanning directories. At GitHub's scale, that scanning became a bottleneck. Worse, the system had no built-in way to handle bit rot or disk failures gracefully. A single bad sector could corrupt an entire repository. Luna: So they needed something more resilient. Did they consider just throwing more hardware at it, or was that not viable? Lucas: They considered it — more SSDs, faster arrays — but the growth curve was exponential. Every year, the number of repositories grew by double digits. And the cost of storing everything on high-end flash was mounting. So they decided to build a custom object storage layer, internally called 'GitHub Object Store' or GOS. The idea was to decouple the git data from the filesystem. Luna: Decouple how? Git's data model is pretty tightly tied to how objects are stored on disk. You can't just abstract that away without changing how git works. Lucas: Right, and that's the clever part. They didn't change git at the protocol level. They built a proxy layer that intercepts git's read and write operations and maps them to objects in a separate key-value store. The proxy handles the translation transparently — git thinks it's talking to a regular filesystem, but really it's talking to a distributed object store backed by something like S3-style storage. Luna: So the migration itself — how did they actually move 100 million repos without taking GitHub down? That's the part that fascinates me. Lucas: They used a technique called 'lazy migration'. Instead of copying everything upfront, they migrated repositories on demand. When a user accessed a repo that was still on the old system, the proxy would fetch the data, store it in the new system, and then redirect subsequent reads to the new store. Over time, the most accessed repos migrated naturally, and they did background sweeps for the long-tail repos that weren't being accessed. Luna: That's elegant, but it introduces a consistency question. What if a write happens during the migration? Say someone pushes a commit while the old data is being copied over. Lucas: They handled that with a two-phase locking mechanism at the repository level. During migration, the repo is briefly locked — no pushes, no fetches — just for the duration of the copy. The lock window is typically under a second. They also had a rollback plan: if the new store failed to write, the system would revert to the old store without losing data. They tested this extensively on shadow traffic before any production cutover. Luna: And the cutover itself? Was it a big-bang switch or gradual? Lucas: Gradual. They started with 1% of traffic, then 5%, then 10%, and so on. At each stage, they monitored error rates, latency, and data integrity. If anything looked off, they could flip the switch back. The whole process took about 18 months from first prototype to full migration. Luna: That's a long time. But I guess when you're GitHub, you can't afford a single incident. Even a five-minute outage would affect millions of developers. Lucas: Exactly. And there's a deeper lesson here about technical debt. The old system wasn't broken — it was working — but the team recognized that the cost of not migrating was going to grow faster than the cost of migrating. That's a hard call to make, especially when the system is still running fine. Luna: I think about that a lot. A friend of mine runs a small SaaS company, and they hit a similar storage bottleneck last year. They had a MySQL database that was ballooning because they were storing blobs in it instead of moving to S3. They knew they should migrate, but it felt like a huge effort for a system that was 'fine'. They ended up doing it after a near-outage, but it was stressful. Lucas: That's the classic trap. The pain of migration is immediate and visible, while the pain of not migrating is deferred and invisible. GitHub's story is a good example of how to do it right — with careful planning, incremental rollout, and a clear understanding of the trade-offs. Luna: So what's the one thing you'd want a CTO to take away from this? Lucas: I'd say: when you're building for scale, design your storage layer to be replaceable from day one. Even if you don't plan to migrate, the ability to swap out the underlying storage without changing the application logic is a superpower. GitHub's proxy layer gave them that flexibility, and it's what made the migration possible without a full rewrite. Luna: And the second thing — don't underestimate the cultural challenge. A migration like that requires buy-in from engineering, operations, and leadership. It's not just a technical problem. Lucas: Absolutely. GitHub had to coordinate across dozens of teams, freeze certain features during the migration, and communicate constantly with users about potential impacts. That's as much a leadership challenge as an engineering one. Luna: So, looking ahead — now that they have this new storage layer, what does it enable? Are there new features or capabilities that were previously impossible? Lucas: Great question. The new system allows them to do things like instant repository cloning — because objects are stored in a globally distributed cache, new clones can be served from the nearest edge. They're also experimenting with more granular access controls and better support for large files. The architecture is built to handle 10x the current scale without a fundamental redesign. Luna: That's the kind of future-proofing that most companies only dream about. It's a good reminder that sometimes the most important infrastructure work is invisible to users. Lucas: Right. And if you're curious about the technical details, GitHub published a series of engineering blog posts breaking down the design decisions. I'll link to them in the show notes. For now, I think the takeaway is clear: don't wait until your storage is on fire to think about the next architecture. Luna: Wise words. And thanks for keeping this conversation ad-free — if anyone wants to support that, you know where to find it. Lucas: Yeah, buy me a coffee dot com slash fexingo. That's it. Thanks for listening.