Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Architecture Decision Records Shaped Spotify's Engineering Culture
Transcript
- Lucas: In 2018, Spotify's playlist infrastructure hit a wall. The team supporting the feature that millions of people open every day was spending about thirty percent of its sprint cycles just coordinating changes across three different backend services. So they did something that wasn't another rewrite—they wrote a document. That document was an Architecture Decision Record, number forty-two, and it changed how they thought about every architectural move after that. Luna: I remember when ADRs first started popping up in engineering blogs. But I've never really heard the inside story of one that actually shaped a major system. What was ADR forty-two about? Lucas: It proposed shifting the playlist update flow from a synchronous request-response pattern to an event-driven one. At the time, every time a user added a song to a playlist, the client would call one service, which then called two others synchronously. If any one of those services was slow, the whole flow degraded. The ADR laid out the context, the options they considered—like sticking with synchronous but adding caching, or moving to an async event bus—and then the decision. They went with Kafka as the event backbone. Luna: And that reduced latency by something like forty percent, right? I think I saw a conference talk about that. Lucas: Exactly. Forty percent drop in p95 latency for playlist updates. But what's more interesting to me is not the technology choice—it's how they made the choice. The ADR forced them to write down the trade-offs, the consequences, and who disagreed and why. That document became a reference point for every related decision for the next two years. Luna: So it's not just about the content of the ADR, but the process around it. How did they get a whole team to adopt that discipline? I imagine some engineers pushed back on the paperwork. Lucas: Yeah, there was definitely skepticism. The playlist team started by using a lightweight template: context, options, decision, consequences. They kept it to one page if possible. And they made the ADR a required step only for decisions that had a duration of more than a quarter—things that would affect the team's architecture for months. That boundary helped. It wasn't 'write an ADR for every tiny thing,' it was 'if this could cause a headache later, document it now.' Luna: That's a smart heuristic. I could see a rule like 'anything that touches a data model or a network boundary' being the threshold. Lucas: Exactly. And over time, the team started seeing unexpected benefits. The ADRs became onboarding material. New engineers could read through the last twelve months of ADRs and understand not just the current architecture, but why it evolved that way. That's huge for a team that's growing fast. Luna: Right, because tribal knowledge gets lost when people leave. And ADRs are explicit, searchable, and they capture the rationale—which is often more valuable than the decision itself. If this is the kind of conversation that helps you think about what you're building or running, we keep this show ad-free thanks to listener support. If today's episode was worth a coffee to you, that's all we ask—buy me a coffee dot com slash fexingo. It's a small thing that lets us keep digging into stories like this. Lucas: Yeah, it really does make a difference. So back to ADR forty-two—one of the biggest cultural shifts came from the way they handled disagreement. In that ADR, one of the senior engineers argued for a different event platform, and they actually recorded that dissenting opinion in the document. It wasn't a footnote; it was a section called 'Rejected Alternatives.' That gave everyone permission to have opinions without derailing the decision. Luna: I love that. It's basically a formalized way of saying 'we heard you, we considered it, here's why we went another way.' Which is way better than someone just feeling ignored and then resenting the decision for months. Lucas: Exactly. And the team at Spotify started sharing ADRs across squads. Before long, you'd have a frontend team looking at a backend team's ADR to understand why an API changed. It broke down silos. By 2020, Spotify had over two hundred ADRs in their repository, covering everything from client storage to service mesh. Luna: Two hundred is a lot. Did they ever have to prune or retire old ones? I imagine some decisions become obsolete. Lucas: They did. They added a status field—'proposed,' 'accepted,' 'deprecated.' If a decision was superseded by a later ADR, they'd link the new one and mark the old one as deprecated. That way, the repository stays honest. No one's reading a five-year-old ADR about a system that no longer exists. Luna: That structure reminds me of how you'd run an RFC process, but more lightweight. What would you say is the one thing a CTO should steal from Spotify's approach? Lucas: I think it's the emphasis on consequences. Most teams stop at 'we chose X.' But the real value is in the anticipated consequences—both positive and negative. In ADR forty-two, they wrote: 'Switching to event-driven will increase eventual consistency, meaning a user might see a slightly stale playlist for up to one second after adding a song. We accept this because the latency improvement outweighs the risk.' That clarity let product management set expectations with users. Luna: So it aligns engineering and product. Which is often the hardest part of any decision. Were there any downsides to the ADR culture? I've heard complaints that writing docs slows teams down. Lucas: Sure, there's a cost. Spotify's own data showed that writing an ADR took about two to four hours for a moderately complex decision. But they also found that decisions made with an ADR were reversed or revisited forty percent less often than decisions made in Slack or a meeting. So the upfront cost saved them rework later. The key is not to mandate ADRs for trivial choices. If the decision can be undone in a day, skip it. If it's going to affect the next six months, write it down. Luna: Two to four hours seems like a fair investment for a decision that lasts years. I'd be interested in how the template evolved. Did they use a standard like Michael Nygard's format or something custom? Lucas: They started with Nygard's format—title, status, context, decision, consequences—but they added a couple of fields: 'assumptions' and 'risks.' And they made it a point to write in plain English, not jargon. Because an ADR might be read by a product manager or a VP who doesn't know the internals. That forced engineers to articulate the trade-offs in business language. Luna: That's a great touch. It elevates the document from a technical note to a communication tool. So what's the state of ADRs at Spotify in 2026? Is the practice still going strong? Lucas: It's scaled with the company. They now have a dedicated ADR review board for cross-team decisions—a rotating group of senior engineers who meet once a week to review the most impactful ADRs. But for internal squad decisions, the process remains lightweight. They also built a simple web interface where you can browse ADRs by service or by topic. It's become an indispensable part of their engineering knowledge base. Luna: It sounds like a pattern that could work in any org of fifty or more engineers, as long as leadership enforces the discipline without being bureaucratic. How do you keep it from turning into a process sinkhole? Lucas: That's the tension. The best advice I've seen is to start small. Pick one team, one significant decision, write the ADR in an hour. See how it feels. Then iterate on the template. Don't try to roll it out to the whole company on day one. Spotify's playlist team didn't invent the process in a vacuum; they adapted it over several months. And they used the results to sell it to other teams. Nothing convinces like a forty percent latency improvement. Luna: True. Numbers are persuasive. So for CTOs listening, what's one action you'd recommend they take this week? Lucas: Find a decision that's been lingering—maybe a choice between two databases or a new service boundary—and write a one-page ADR for it. Don't worry about making it perfect. Just get the context, the options, the decision, and the consequences down. Then share it with a colleague. The act of writing alone will clarify your thinking. And if you want a starting template, search for 'ADR Markdown Any Decision'—there's a popular GitHub repo that's a good reference. Luna: I'll check that out. It's one of those practices that seems obvious in hindsight but takes intention to start. Thanks for breaking down Spotify's story, Lucas. I have a feeling ADR forty-two is going to inspire some new documents in a lot of offices this week. Lucas: That's the hope. A good decision record doesn't just capture the past—it shapes the future.