Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How GitHub Migrated 100 Million Repositories to a New Storage Engine
Transcript
- Lucas: If these conversations are useful for what you're building or running, you might want to hear this one. GitHub quietly finished migrating over 100 million repositories to a brand-new storage engine they built from scratch. They call it GitV2. Luna: GitV2 — and this isn't just a new file system layer, right? This is replacing the heart of how Git stores objects. Lucas: Exactly. Git's original storage model is surprisingly simple: each object — commit, tree, blob — is stored as a separate file, compressed individually, with loose objects eventually packed into packfiles. For a single developer repo, that's fine. But GitHub runs about a hundred million repos, many with years of history, large binaries, and massive monorepos. The filesystem approach breaks down at that scale. Luna: So what was the actual bottleneck? Inodes? Seek times? Lucas: Both. Plus the overhead of managing millions of tiny files across a distributed file system. GitHub's engineering team published some numbers: they had cases where a single `git clone` would open tens of thousands of file descriptors, and the metadata operations — stat, readdir — were consuming more CPU than the actual data transfer. They needed something that treated Git objects as a database, not as files. Luna: And that's where GitV2 comes in. What's the core data structure? Lucas: It's a content-addressed store built on top of RocksDB, which is an embedded key-value store from Facebook. Each Git object hash maps directly to a key, and the value is the object content plus some metadata. But the interesting part is how they handle packfiles: instead of storing each object individually, they keep the packfile as a single blob, and then use RocksDB's built-in bloom filters and prefix compression to make lookups fast. They also added a custom caching layer for the most frequently accessed objects — think HEAD commits, tree objects for popular branches. Luna: So the old system was filesystem-bound, and the new one is essentially a distributed database with a Git frontend. How did they migrate 100 million repos without downtime? Lucas: That's the really impressive part. They did a shard-by-shard migration — each shard covers a range of repository IDs. They built a shadow read system: every write went to both the old storage and GitV2, but reads first tried the new engine, and if it didn't have the data, fell back to the old one. Over months, they enabled GitV2 for reads on one shard at a time, monitoring latency and error rates. If something went wrong, they could flip the read back instantly. Luna: Right, so canary testing at the shard level. What was the most common failure mode? Lucas: Cache misses. Some repos have extremely specific access patterns — like a CI system that clones the same commit hundreds of times per minute. The old filesystem cached those objects in the OS page cache, but GitV2 had its own cache that wasn't always warm. They had to fine-tune the cache sizing and pre-warm it for high-traffic repos. Another issue was partial clones — where you only fetch a subset of objects. GitV2's packfile representation wasn't optimized for that initially, so they had to add a new index structure to quickly find which objects belong to which path. Luna: Makes sense. And the performance gains? I read somewhere clone times dropped by 40%. Lucas: Yeah, for full clones of repos with deep history, they saw about a 40% reduction. For shallow clones, the improvement was smaller — maybe 10–15% — because the overhead is mostly in the network, not the storage. But the storage savings were huge: a 30% reduction in disk footprint, mainly due to better deduplication and compression in RocksDB. They now store a single copy of an object even if it appears in multiple repos, which the filesystem couldn't do easily. Luna: That's a big win for GitHub's infrastructure costs. But I'm curious about the engineering org side — how did they convince the team to undertake a multi-year rewrite of the storage layer? That's a bet the company kind of project. Lucas: And we should say, this kind of deep infrastructure work is exactly why we make these conversations. We deliberately don't run ads on these episodes — not a single one. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. No pressure, just if the show is valuable to you. Luna: Yeah, it's a small way to keep this independent and ad-free. So back to GitV2 — how did they get buy-in? Lucas: They started with data. The engineering team ran benchmarks that showed the filesystem approach would hit a hard wall within 18 months — not in terms of total storage, but in terms of metadata operations per second. They projected that by late 2025, a routine `git push` could take over a minute due to filesystem contention. That gave the leadership a clear urgency. Then they built a prototype that handled 10,000 repos with the new engine and showed it was faster and more reliable. That convinced the core infrastructure teams. Luna: So it was a data-driven case, not a 'let's rewrite because it's fun' kind of thing. And the rollout took how long? Lucas: About 20 months from first prototype to full migration. But the interesting organizational detail is that they didn't create a separate GitV2 team. Instead, they embedded engineers from each of the major teams — the git team, the storage team, the operations team — into a tiger team that reported to the VP of engineering. That way, when they needed changes in, say, the replication layer, the person who owned that code was already in the room. Luna: That's smart — avoids the 'us vs. them' dynamic that plagues cross-team rewrites. So what's next for GitV2? Are they going to open-source it? Lucas: GitHub hasn't announced plans to open-source GitV2 itself. But they've published a few blog posts on the design, and some of the ideas are already influencing Git's upstream development. For example, the concept of a 'commit graph' that speeds up reachability checks — Git 2.40 introduced a similar feature based on what GitHub learned. So the community benefits indirectly. Luna: One last thing — how does this affect the developer experience for the average GitHub user? Do they notice anything different? Lucas: If you're cloning a repo today, it's faster — especially for large repos like the Linux kernel or React Native. Also, 'git log' and 'git blame' operations that traverse history are noticeably snappier because the new engine caches commit graphs. But the goal was transparency: you shouldn't have to change your workflow. The best infrastructure migrations are the ones users never see. Luna: Right. And that's the mark of a successful rewrite — when it just works, and the only difference is speed. Thanks, Lucas.