Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Documentation Fails and What to Do About It
Transcript
- Lucas: Luna, I want to start with a number that keeps bugging me. A 2023 survey by SmartBear found that 67 percent of QA teams cite outdated documentation as a top challenge. Not test flakiness, not tooling — documentation. That's nearly seven out of ten teams. Luna: That's huge. And I bet most of us have been there — staring at a test case document that describes a feature that was deprecated two releases ago. Lucas: Exactly. And the default fix is usually 'write better docs.' But I think the problem is structural. Documentation becomes a separate artifact — it's created once, then it rots. The real solution is to stop treating documentation as a deliverable and start treating it as a byproduct of how you define tests. Luna: Are you talking about behavior-driven development? Gherkin scenarios and all that? Lucas: Exactly. BDD, or specification by example — the idea that your test scenarios are written in a structured natural language that non-technical stakeholders can read. In theory, those scenarios become the living documentation. They're always up to date because they're executed as tests. If the scenario changes, the test changes. Luna: In theory. But in practice, I've seen teams where the Gherkin files are just as neglected as the Word documents they replaced. Lucas: That's the key nuance. Gherkin alone doesn't solve it. The discipline is that the scenarios must be the single source of truth for what the system does. If a product manager updates a feature, the first thing that changes is the given when then scenario. Not the code, not a spec sheet — the scenario. Luna: So you're saying the process has to embed documentation into the definition of done. It's not a separate phase. Lucas: Right. I worked with a fintech startup that was struggling with onboarding. New engineers spent two weeks reading outdated confluence pages. They switched to writing all acceptance criteria as Cucumber scenarios before any sprint. After three months, new-hire ramp-up time dropped by 30 percent. Because the scenarios were executable and always reflected the actual system behavior. Luna: Thirty percent is a big deal. But I imagine the product team had to buy into writing Gherkin. That's not always easy. Lucas: It's a cultural shift. Product managers often think 'I'll write scenarios later.' But if you make scenario writing the gate for starting development, it forces the conversation early. The side effect is that you get much clearer requirements. I've seen teams catch 40 percent more ambiguities in the scenario-writing phase than they ever did in traditional requirement reviews. Luna: That number sounds familiar — there was a study from a few years ago that showed BDD teams detect requirement defects earlier, which reduces rework cost by something like 60 percent. Lucas: Yep. But here's the thing — living documentation only works if the scenarios are written at the right level. Too many teams write scenarios that are either too high-level — 'user logs in successfully' — or too technical — 'when the API returns HTTP 200.' The sweet spot is the language of the business domain. Luna: So for a banking app, instead of 'when the balance endpoint returns zero', you'd say 'Given a customer with a zero balance, when they attempt to withdraw five dollars, then they see an insufficient funds message.' Lucas: Exactly that. And that scenario becomes documentation for customer support, for QA, for the next developer inheriting the feature. It's readable, testable, and it doesn't go stale because it's run every time the code changes. Luna: But what about teams that don't use Cucumber or SpecFlow? Can you still have living documentation? Lucas: You can, but it's harder. Some teams embed structured comments in their unit tests that feed into a documentation generator. But the problem is that unit tests are written in code — they're not readable by non-developers. The power of Gherkin is that it's a shared language. Your business analyst can write a scenario, your developer can implement it, and your tester can verify it. All without translation. Luna: I want to go back to the 67 percent number. That means most teams are still stuck with stale docs. What's the first step you'd recommend for a team that's in that situation? Lucas: Start with one feature. Pick something that changes frequently — maybe a checkout flow or a search algorithm. Write three or four core scenarios in Gherkin. Make them pass. Then, instead of updating the old requirement document, just link to the scenario file in your version control. Watch whether people actually read it. Nine times out of ten, they will, because it's concise and accurate. Luna: And if the scenarios don't get updated? That's a people problem, not a tools problem. Lucas: Absolutely. But here's a trick — make the scenario update part of the code review. When a developer changes behavior, the review checklist should include 'did the scenario change?' If no, flag it. That builds the habit. After a sprint or two, it becomes automatic. Luna: I've also seen teams use something called 'documentation linters' — tools that check if Gherkin files are present for every new feature. That kind of automation helps enforce the discipline. Lucas: Great point. You can even have a CI check that fails if a feature file doesn't exist for a new story. It sounds draconian, but it's better than a 67 percent documentation failure rate. Luna: So living documentation isn't about writing less — it's about writing once, in a place that stays alive. Lucas: Exactly. And that's the message I want listeners to take away. The next time you're updating a README or a Confluence page, ask yourself: could this be a Gherkin scenario? If yes, you're probably spending time maintaining a dead artifact. Luna: On that note, I think this is a good moment to mention something about this show. We work hard to keep these episodes ad-free because we believe the content should stand on its own. No sponsors, no interruptions. Lucas: And that's only possible because of listeners who choose to support us. If you've gotten value from episodes like this one — maybe a technique you applied at work — you can help keep it going at buy me a coffee dot com slash fexingo. Luna: It's a small way to say this matters. And we appreciate every single person who does. Lucas: Alright, back to living documentation. One more angle I want to touch on — how do you handle living documentation for legacy systems that don't have any tests yet? That's a common scenario for teams inheriting old code. Luna: I'd say start with characterization tests. Write scenarios that capture the current behavior, even if it's buggy. Then you have a baseline. The documentation becomes the truth of what the system does today, not what it should do. Lucas: Right. That's the Michael Feathers approach from 'Working Effectively with Legacy Code.' You write tests that document the actual behavior, then you can refactor safely. And those scenarios become the living documentation for that module. It's how you climb out of the documentation debt hole. Luna: And the beauty is that once you have those scenarios, you can start having conversations with stakeholders: 'The system currently does X. Is that correct?' That's when living documentation becomes a communication tool, not just a test artifact. Lucas: Yes. And that's the ultimate goal — documentation that informs and aligns the team, not just a shelf-ware PDF that nobody reads. So if you're in the 67 percent, take heart. You can start small, with one feature, one scenario, one CI check. And let the documentation grow from the code. Luna: I think that's a great place to wrap. Next episode, we'll look at something a bit different — how chaos engineering can inform your testing strategy in unexpected ways. Lucas: Looking forward to it. Until then, keep your scenarios green and your documentation alive.