Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How GitHub Actually Migrated 100 Million Repos to a New Storage Engine
Transcript
- Lucas: If you have ever pushed a commit to GitHub, you've interacted with a piece of infrastructure that, until very recently, was basically running on the same storage model that Git was designed for in 2005. Luna: You mean the whole 'just dump it on a POSIX filesystem' approach? Lucas: Exactly. And that worked fine when you had a few thousand repos. But GitHub crossed 100 million active repositories last year. And the storage backend — a layer called 'Gitaly' that they built a few years ago — was still ultimately backed by NFS or local disks holding raw Git objects. The scaling limits were becoming a real bottleneck. Luna: So they decided to replace the entire storage engine underneath without anyone noticing. Lucas: That's the headline, but the reality is more nuanced. They started in early 2024 with a project called 'GitHub Storage Service' — GSS for short. The core idea was to move Git object storage off POSIX filesystems and onto an object store — think S3-like, but custom-built for Git's access patterns. Luna: What's so special about Git's access patterns that a generic object store couldn't handle? Lucas: Great question. Git reads objects by their SHA hash — you'd think that maps perfectly to key-value storage. But the problem is that Git operations often need to enumerate all objects in a packfile, or walk the entire commit graph. Object stores are terrible at sequential scans. So GitHub had to invent a new data structure on top of the object store that indexed objects in commit order, not just by hash. Luna: So they basically built a custom indexing layer. Did they also change the Git protocol? Lucas: They made a deliberate decision not to. The protocol — the way 'git push' and 'git clone' talk to the server — stayed exactly the same. All the complexity was hidden inside Gitaly. That's the beauty of it: from a user perspective, nothing changed. But underneath, every object was being stored in a completely new system. Luna: How do you even begin a migration like that? You can't just flip a switch on 100 million repos. Lucas: They did a phased rollout over about two years. Phase one was 'shadow mode' — for every read or write, Gitaly would also send the data to GSS, but the authoritative source was still the filesystem. They verified consistency, measured latency, and fixed bugs. Phase two was 'dual-write' — both systems wrote every new object, and reads could come from either. Phase three was the cutover: make GSS the primary, and start backfilling historical data from the filesystem into GSS. Luna: I love the shadow mode approach — it's basically a parallel test environment without risk. But the backfill must have been enormous. How many objects are we talking? Lucas: GitHub has somewhere on the order of tens of billions of Git objects. The backfill took months. They had to be careful not to saturate the network or the filesystem I/O during peak hours. They built a scheduler that throttled backfill based on global load. Luna: And during all this, zero user-facing incidents? That's impressive. Lucas: According to the engineering blog, yes — zero downtime or data loss. But there were some interesting edge cases. For example, repos with millions of objects in a single commit — think a game engine or a large monorepo. The old system would handle that by streaming the packfile directly to disk. GSS had to be able to reassemble that on the fly without buffering the entire object set in memory. Luna: So they had to handle terabyte-scale monorepos without blowing up memory. That's a hard constraint. Lucas: Right. They introduced a streaming packfile builder that reads objects one at a time from GSS, compresses them, and sends them to the client as a continuous stream. The memory cost per clone is now constant — around 50 megabytes — regardless of repo size. That's a big improvement over the old system, which would sometimes need to hold a full packfile index in memory. Luna: And the performance numbers? You said 40% faster clone times for large repos? Lucas: Yeah, that's the headline metric. For repos over a gigabyte, clone times dropped from about two minutes to under 75 seconds. For smaller repos, the improvement is less dramatic — maybe 10 to 15 percent. But the real win is consistency: because GSS uses object storage with a global namespace, the latency is predictable no matter which region you're in. The old nfs based setup had huge tail latency when a filer was overloaded. Luna: So they also solved the 'hot filer' problem. Every ops team knows that dread — when one NFS server starts choking. Lucas: Exactly. And that's part of why they made the investment. The old system required manual rebalancing — moving repos between filers — which was fragile and labor-intensive. With GSS, capacity is elastic. You can add storage nodes without any data migration. Luna: What about the ai driven surge? GitHub has Copilot, and there's been this explosion of ai generated commits in the last couple years. Did that factor into the design? Lucas: Absolutely. One of the engineering leads mentioned that the rate of new object creation had been growing faster than they could add NFS capacity. ai generated code means more commits, more branches, more objects. GSS was designed to handle that growth curve — object stores scale horizontally in a way that filesystems just don't. Luna: It's fascinating how a storage engine migration can become a competitive advantage. But let me ask — what did they have to give up? There's always a trade-off. Lucas: The main trade-off is latency on small reads. Reading a single Git object from an object store is slower than reading it from local disk — we're talking maybe 5 milliseconds versus 1 millisecond. But GitHub mitigated that with aggressive caching at the Gitaly layer. They also tuned the object store to prioritize low-latency reads over write throughput. The net effect is that most operations feel faster, but the raw latency per object is actually higher. Luna: So the design was holistic — they optimized for the end to end user experience, not for any single metric. Lucas: Right. And they were disciplined about not over-engineering. One thing I really liked: they didn't try to build a general-purpose storage system. GSS is hyper-specialized for Git. It doesn't support arbitrary key-value operations or SQL queries. That narrow scope let them make decisions that would be terrible for a generic system but perfect for this use case. Luna: What's one specific design decision that reflects that? Lucas: They store objects in groups based on the commit graph topology. When you do a fetch, you typically want objects that are close to each other in the DAG. So GSS physically co-locates objects from the same branch or tag. That's a terrible idea for a general object store — it creates hot spots — but for Git, it reduces the number of I/O operations per clone by a factor of ten. Luna: That's clever. It's like organizing books in a library by topic instead of by author name, but only for the books that people actually check out together. Lucas: Exactly right. And this is the kind of deep systems thinking that makes for a great CTO podcast episode. Which reminds me — if you're finding these deep-dive conversations useful for your own work, there's a small way to keep them going ad-free. Luna: Yeah, we've kept this show free of sponsorship interruptions because a bunch of listeners chip in a couple of dollars a month. It genuinely makes a difference. Lucas: If you've gotten value from unpacking how systems like GitHub's storage migration actually work, you can support the show at buy me a coffee dot com slash fexingo. That's buy me a coffee dot com slash fexingo. It helps cover hosting and production, and keeps these episodes open to everyone. Luna: And it lets us focus on the engineering details without cutting to a commercial break. So, back to GSS — I want to know about the rollback plan. Did they ever have to use it? Lucas: They did test rollbacks during the dual-write phase. The plan was to keep both systems in sync for at least six months. If GSS had a catastrophic failure, they could flip the primary back to the filesystem. But they never actually needed to in production. The shadow mode caught enough bugs that by the time they cut over, the system was solid. Luna: That's a good lesson: invest in testing infrastructure so you don't have to use the emergency plan. Lucas: Absolutely. And the whole project is a case study in incremental migration. They didn't rewrite the world overnight. They added a layer, verified it, switched traffic gradually. It's boring engineering — and that's why it worked.