Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How npm Cut Package Publish Failures by 80 Percent
Transcript
- Lucas: So, there's this moment every JavaScript developer knows: you run npm publish, and for a split second, you hold your breath. Will it go through? For years, that uncertainty was a real thing, and npm's team felt it too. Luna: I remember that anxiety. And it wasn't just about the few seconds of waiting; when it failed, you'd have to bump your version and republish, which was a pain. Lucas: Exactly. So today we're looking at how npm cut publish failures by 80 percent. And the story isn't just about a tweak here or there—it's a full re-architecture of their publish pipeline, moving from a monolithic database to a distributed, event-driven system. Luna: That sounds like a massive change. Where did the original system start to break down? Lucas: The old system was built around a single PostgreSQL database that held the entire registry—every package, every version, every metadata blob. It worked fine for years, but as the ecosystem exploded, the write path became a bottleneck. Every publish had to update the main row for that package, then update the version-specific rows, and then invalidate caches. Under heavy load, that serialized work would queue up, and timeouts would start to pile on. Luna: So it was a classic hot-spot problem. All the traffic for a popular package like lodash would funnel onto one database row. Lucas: Right. And they tried to scale vertically for a while—bigger instances, more memory—but it was a losing battle. The team knew they needed to change the data model fundamentally. They opted for an event-sourced approach, where every publish is captured as an immutable event, and then that event fans out to update the various projections that serve reads. Luna: So instead of one giant database, you have a stream of events that multiple services can consume independently. That makes sense. Lucas: They built a dedicated publish service that owns the event stream, and then a set of workers that handle things like updating the metadata store, invalidating CDN caches, and triggering downstream notifications. The key was that the publish operation itself only needs to write to the event log, which is append-only and can be sharded. That decoupling took the latency out of the critical path. Luna: But sharding an event log—how did they decide what goes where? I imagine you can't just shard on package name because that brings back the hot-spot problem for popular packages. Lucas: Good question. They actually shard on a hash of the package name and version, so each publish event is assigned a random shard. That distributes the load evenly, even for lodash. And because the event log is append-only, there's no contention on a single row. The workers then read from all shards and process events asynchronously. Luna: So you trade immediate consistency for eventual consistency. Isn't that risky? If a user publishes and then immediately tries to install, the package might not be there yet. Lucas: That was their biggest concern, and they had to think about it carefully. They made a deliberate trade-off: the event is written synchronously, so the client gets a success response, but the metadata store updates asynchronously. To mitigate the read your writes problem, they added a mechanism where the publish API returns a token that the client can use to check readiness, and the CLI actually polls until the package is visible. It's a bit like a distributed transaction, but simpler. Luna: So the user experience doesn't change, but the backend is fundamentally different. That's the kind of migration that keeps engineering leaders up at night. Lucas: Absolutely. And we're lucky to have sponsors who keep us ad-free. If you find these deep dives useful for what you're building or running, you can support us at buy me a coffee dot com slash fexingo. It's a small way to keep this podcast independent. Luna: Yeah, it's because of listeners like you that we can stay focused on the technical details without any sponsor noise. Appreciate anyone who considers it. Lucas: Now, back to the migration. They rolled this out incrementally, starting with a shadow-read phase where they mirrored traffic to the new system but didn't switch over. That let them compare the outputs and catch data inconsistencies. Luna: How long did that phase last? It's got to be tempting to rush it when you're under pressure to fix the problem. Lucas: I think they spent a few months in that mode. They built a diff tool that would compare the metadata returned by the old and new systems for a sample of reads, and they had alerts for any mismatches. It wasn't until they saw a near-zero diff rate that they felt confident enough to flip the switch. Luna: So you're saying they had a safety net. But what about the human side? How did the team convince themselves—and the community—that this was the right move? Lucas: A lot of that came down to communication. npm is a public good, so they published blog posts about the architecture, ran beta programs, and even had a status page during the migration. It built trust. And they had a rollback plan—they could flip back to the old system within minutes if something went wrong, because they kept the old stack running in parallel for a while. Luna: That's a huge operational cost, though. You're running two systems that both need to be maintained and kept in sync. Lucas: It is, but they saw it as the cost of de-risking a critical migration. And honestly, for a service that powers millions of developers, it's worth it. The result? Publish failures dropped by 80 percent. And the system is now horizontally scalable, so they can handle spikes from big releases without breaking a sweat. Luna: I love that they focused on the developer experience throughout. It's not just about performance metrics; it's about not making developers anxious when they share their code. Lucas: Exactly. And that's a lesson for any CTO: when you're rebuilding a core piece of infrastructure, the user's perception of reliability matters as much as the actual uptime numbers. npm's story shows that with careful planning, a willingness to make trade-offs, and a strong communication plan, you can evolve even the most critical systems. Luna: So, what's the one takeaway you'd want a CTO to store from this? Lucas: I'd say it's the value of decoupling the write path from the read path. By moving to an event-driven model, npm turned a fragile, monolithic bottleneck into a resilient, scalable system. It's a pattern that applies beyond package registries—any system with hot spots can benefit from that thinking. Luna: And the fact that they cut failures by 80 percent is a testament to the approach. Thanks for listening, and we'll catch you in the next episode. Lucas: Stay curious.