Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Spotify Fixed Its Testing Culture
Transcript
- Lucas: Most software teams think they have a testing problem when they actually have a ownership problem. You see it in every quarterly review where deployment frequency drops but bug counts stay flat. The team blames the tools, the CI pipeline, or the budget for QA engineers. Luna: Right, because you can buy all the automation suites you want, but if no one feels responsible for the code they ship, the tests just become decorative. It’s theater. Lucas: Exactly. And that is why the Spotify engineering model is such a sharp counter-example here. They didn’t fix their quality issues by hiring more testers or buying better monitoring tools. They fixed it by dismantling the idea that quality was someone else’s job. Let’s look at the specific structural change they made around two thousand sixteen. Luna: I remember reading about that shift. They moved away from having dedicated squad-level QA roles into a shared responsibility model? Lucas: They did, but it wasn’t just about removing titles. It was about redefining what 'done' means for a feature. Before that shift, developers would write code, hand it off to QA, wait for a pass, then maybe fix something late in the cycle. That creates a feedback loop that is measured in days, not minutes. Luna: And that delay is exactly where bugs hide. They get buried under integration work and only surface in production. Lucas: Precisely. So Spotify introduced what they called the Quality Guild. It sounds bureaucratic, but it was actually a support network, not a gatekeeper. The guild provided frameworks and best practices for automated testing, but each squad decided how to implement them based on their specific risk profile. Luna: So instead of a central policy dictating fifty unit tests per feature, they let the team decide what actually matters for their users? Lucas: That is the core insight. A payment squad might need rigorous transactional integrity tests, while a content recommendation squad might prioritize latency and edge-case handling. When QA is centralized, they force a uniform standard that often misses the actual risks. When it’s decentralized, the people building the feature own the risk assessment. Luna: But doesn’t that lead to inconsistency? One team might be super rigorous while another cuts corners because they’re under pressure. Lucas: It does, unless you have strong engineering norms. The Quality Guild at Spotify acted as a mentorship layer. Senior engineers rotated through squads to help calibrate their test strategies. It wasn’t policing; it was coaching. If a squad’s test coverage was low, they didn’t get blocked from deploying. They got invited to a workshop on how to improve their local verification. Luna: That is a fascinating distinction. Blocking deployment is punitive, but coaching is developmental. Most companies default to blocking because it’s easier to measure compliance than competence. Lucas: Spot on. Compliance is easy. Competence is hard. By removing the block, Spotify forced developers to care about their own output. You cannot outsource accountability to a tester who has never seen your code architecture. The moment you make the developer the first line of defense, your defect rate drops because the feedback loop shrinks from days to seconds. Luna: I have seen teams try this and fail because they lacked the initial discipline. Developers hate writing tests until they realize how much time they save debugging later. Lucas: Which brings us to the metric that actually tracks this success. Spotify stopped measuring QA throughput. They started measuring Mean Time To Recovery. If a bug slips through, how fast can the team fix it and redeploy? That metric aligns incentives perfectly. It encourages robust testing to prevent escapes, but also encourages fast remediation when prevention fails. Luna: Because if you are spending half your sprint fixing old bugs, you aren’t shipping new features. Fast recovery protects the roadmap. Lucas: Exactly. It turns quality into a velocity multiplier rather than a speed bump. Now, I know what some listeners might be thinking. This sounds like it requires mature teams. What about smaller companies or startups that don’t have a Quality Guild? Luna: That is a fair pushback. Startups usually can’t afford dedicated QA resources anyway, so they just wing it until something breaks. Lucas: The principle still applies regardless of size. Even a three-person team can adopt the mindset shift. The key is to stop treating testing as a separate phase. Integrate verification into the daily workflow. Code reviews should include test plans. Feature flags should limit blast radius. These are small tactical changes that mimic the structural benefits of a larger guild. Luna: So it’s less about the org chart and more about the rhythm of the work. You embed quality checks into the natural flow of development. Lucas: Yes. And that rhythm is what prevents the legacy code decay we talked about in episode one hundred and ninety-six. When everyone owns the code, no one lets it rot. There is a constant, gentle pressure to keep the system clean because you are the one who will have to debug it next Tuesday. Luna: It really comes down to empathy for your future self. Writing tests is an act of kindness to the person who inherits your ticket. Lucas: Beautifully put. But there is a trap many teams fall into when they try to replicate this. They automate everything immediately. They spend months building a massive test suite that runs for hours before every deploy. Luna: Oh, yes. The slow test hell. You lose the benefit of immediate feedback because nobody wants to wait forty minutes to know if their change broke anything. Lucas: Right. And that slowness kills the culture. Developers start ignoring failures because they are too tired of waiting. The lesson from Spotify is to start small. Focus on the critical path. Automate the things that hurt most when they break. Keep the rest manual or exploratory until the cost of failure justifies the automation effort. Luna: So prioritize risk over completeness. Test what matters, not just what is easy to script. Lucas: Exactly. Completeness is a vanity metric. Reliability is the only thing that keeps customers paying. Now, before we dive deeper into how to structure those critical paths, I want to take a quick moment to talk about the show itself. Luna: Yeah, this topic really resonates with anyone who has ever been stuck in a slow release cycle. Lucas: If today's tech conversation gave you something usable, consider supporting the Fexingo Business network directly. Listener contributions are what keep these deep dives ad-free and independent. It helps us bring you more specific, data-driven analysis like this every day. Luna: You can find our donation page at buy me a coffee dot com slash fexingo. Any amount helps us keep producing this content. Lucas: Thank you for considering it. Back to the engineering culture. Let’s talk about the role of observability in this decentralized model. Luna: Observability becomes the safety net. If you aren't checking every commit manually, you need real-time data to catch regressions. Lucas: Correct. In a high-velocity environment, logging and metrics are your secondary eyes. Spotify uses extensive canary releases. They roll out a change to one percent of traffic and watch the error rates. If the numbers look good, it expands. If not, it rolls back automatically. Luna: That automates the safety check without adding human delay. It’s like a circuit breaker for software defects. Lucas: A perfect analogy. And that automation allows the human element to focus on creativity and complex problem solving, rather than repetitive verification tasks. It frees up mental bandwidth. Luna: Which leads to better product outcomes. Engineers who aren't bogged down by QA bureaucracy can innovate faster. Lucas: Absolutely. The bottom line is that quality is a design choice, not a testing outcome. You build it in from day one, or you pay for it in debt later. Luna: So the question isn't how do we test better, but how do we design systems that are harder to break. Lucas: Now that is a question worth asking your own team tomorrow.