Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Wikipedia Survives With Almost No Tech Debt
Transcript
- Lucas: So, Wikipedia. It's one of the top ten most visited websites in the world. It runs on MediaWiki, a codebase that's been around since 2002. And yet, the engineering team is famously tiny, and the tech debt is... surprisingly manageable. Luna: I've heard that phrase, 'boring technology,' thrown around. Is that really the secret? Lucas: Partly, yeah. But it's more than just picking boring tech. It's a whole discipline around not letting that boring tech become a mess. Let me give you a concrete example. In 2023, Wikipedia's engineering team was about 200 people, and they serve billions of page views. That's an insane ratio. How do they do it? Luna: I'd love to know. I mean, I've worked in startups where we had more engineers than that and we still had a mess. Lucas: Right. So one of the first things you notice is their strict policy on backward compatibility. MediaWiki has been around for over twenty years, and they have extensions and customizations that were written a decade ago and still work. They've never had a major breaking change. That's a huge deal. Luna: Never? I find that hard to believe. Every codebase I've seen has had some 'clean-up' that broke things. Lucas: I know, it sounds insane. But they've made backward compatibility a non-negotiable. They'd rather carry a bit of extra code than break something for their users. And they have a formal process for deprecating anything: you have to announce it years in advance, provide a migration path, and even then, they rarely remove it unless absolutely necessary. Luna: So they just accumulate cruft? That doesn't sound like low tech debt. Lucas: That's the counterintuitive part. They do accumulate some, but they keep it in check. They have a 'code stewardship' role in the community that actively reviews old code and refactors it. But they never do a big bang rewrite. They do incremental improvements. And that's a deliberate choice. Luna: So what's the actual metric for tech debt then? Is it just about not breaking things? Lucas: For them, it's about the cost of change. They measure how easy it is to add a new feature or fix a bug without touching a million other things. And they've kept that cost low for years. There's a great example from 2020 when they rolled out a new skin called Vector 2022. That was a huge change, but they did it with feature flags and gradual rollouts, so the risk was minimal. Luna: Feature flags, sure. But I've seen those become a mess too. How do they avoid that? Lucas: They have a disciplined approach. They use a configuration system that allows them to enable features per user, per wiki, or globally. And they actually run beta features in production. So they're dogfooding all the time. Their users are their testers, and they're quite open about it. There's a 'beta features' section on every account. Luna: Wait, so they are using production as a test bed? That sounds risky. Lucas: It is, but they're careful. They have a dedicated 'beta cluster' as well, but the real testing happens on Wikipedia itself with a small percentage of users. And they have excellent monitoring. If something goes wrong, they can roll back instantly. They've had a few incidents, but they've never had a major outage that lasted long. Luna: I read somewhere that they actually do maintain a huge test suite. Is that true? Lucas: Oh, absolutely. They have hundreds of thousands of unit and integration tests. But they don't rely on them as a safety net. They rely on their process. The tests just tell you if you broke something; they don't tell you if you built the right thing. Luna: Right. And their 'right thing' is all about stability and simplicity. I mean, they could have rewritten MediaWiki in a modern language, like Rust or Go, but they didn't. Why? Lucas: Because that would be a massive risk with no clear benefit. MediaWiki is written in PHP, and it works. It's well understood. The libraries are mature. And the community of developers knows it. Rewriting in a new language would alienate contributors and introduce a ton of new bugs. So they stay with PHP, and they don't feel embarrassed about it. Luna: That's refreshing. Most CTOs I know would be itching to rewrite everything in the latest hotness. Lucas: And that's exactly why they succeed. They have a culture that values 'boring technology.' They even have a page on their wiki that says 'Boring technology is a feature.' They'd rather use a slightly less elegant solution that everyone understands than something clever that only a few people can maintain. Luna: But doesn't that stifle innovation? I mean, they are a top-ten website. They need to innovate to stay relevant. Lucas: They do innovate, but they do it in layers. They keep the core stable and boring, and they experiment on the edges. For example, they've been working on a new search system, and they've added a machine learning-based recommendation system. But those are optional modules, not core rewrites. They also have a whole program for 'Community Tech' where they build features requested by editors. That's innovation, but it's constrained and managed. Luna: That makes sense. So the lesson for other engineering orgs is: don't rewrite, but do refactor incrementally. And keep your core boring. Lucas: Exactly. And there's another piece: they have a very flat hierarchy. There aren't many 'architects' or 'principal engineers' who design grand systems. Instead, decisions are made by consensus among a small group of maintainers. That prevents a lot of over-engineering. Luna: But how do they avoid analysis paralysis? Consensus can be slow. Lucas: It is slow sometimes. But they've been doing this for twenty years, and they've got a process called 'Request for Comments' where any proposed change is discussed. But for day-to-day decisions, they have a few key maintainers who can approve changes quickly. They have a good balance. And the slow pace is actually a feature, not a bug. It means they don't make hasty decisions. Luna: I've also heard that they have a very strong 'do no harm' policy. Is that true? Lucas: Yes, they literally have that in their principles. 'Do no harm' is one of their guiding principles. They are extremely cautious about any change that might affect the user experience or the community. That's why they have a 'Wikimedia Engineering Principles' page, and one of the items is 'We plan for the future by building for the present.' They don't build speculative features. Luna: That's a good principle for any startup. But I guess it's easier when you're a non-profit and you don't have the pressure to ship 50 features a quarter. Lucas: True, but even in the for-profit world, you can apply that. Look at how many companies have suffered from a major rewrite that went wrong. There are countless horror stories. Wikipedia is a shining counterexample. Luna: And they've been around since 2001. That's two decades of continuous operation. What do you think is the single most important thing they do that other companies don't? Lucas: I'd say it's their commitment to backward compatibility and their willingness to carry a little bit of 'ugly' code for the sake of stability. Most engineering teams are quick to remove deprecated code, but Wikipedia treats that code as a promise to their users. And that promise builds trust. Luna: So the next time I want to refactor my codebase, I should remember: think of your users, not just your code. Lucas: Exactly. And if you're listening and you want more conversations like this one, we keep the show ad-free, and the only thing that supports that is the listener community. So if you find value in what you hear, consider supporting the show at buy me a coffee dot com slash fexingo. It's a small thing that keeps us going. Luna: And we appreciate every bit of it. But back to Wikipedia. I want to ask about their incident management. Have they had any notable outages? Lucas: They've had a few, but they're rare. One notable one was in 2019 when they had a DDoS attack that took them down briefly. But they recovered quickly. Their infrastructure is spread across multiple data centers, and they have a robust failover system. Luna: So they are prepared. I read that they use a custom built CDN for static assets. Is that true? Lucas: They do have a global CDN, but it's not something they built from scratch. They use a network of caching servers, but they also have a custom caching layer for their API. They're not afraid to build what they need if off-the-shelf doesn't fit. Luna: Interesting. So they're not dogmatic about 'buy vs build' either. Lucas: Not at all. They use open source as much as possible, but they've also built their own tools when necessary. For example, they have a custom deployment system called 'Deploy' that's been in place for years. It's not fancy, but it works. Luna: So what's the biggest risk they face now? Is it tech debt or something else? Lucas: I think the biggest risk is attracting and retaining engineers. Since they're a non-profit, they can't pay huge salaries. But they offer a mission-driven culture and the chance to work on a project that impacts millions. That's a trade-off that some engineers love. Luna: And for someone listening who's a CTO, what's the one takeaway you'd give them? Lucas: If you're facing a legacy system that seems like a liability, remember that legacy can be an asset. Wikipedia has shown that a twenty-year-old codebase can be maintainable, stable, and even delightful to work on, if you have the right principles. You don't have to rewrite everything. You just need to manage complexity carefully. Luna: And maybe embrace boring technology once in a while. Lucas: Exactly. Boring is beautiful.