Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / The Case for Structured Server Documentation
Transcript
- Lucas: So, let’s talk about something that almost every server team I’ve ever worked with has a complicated relationship with: documentation. Luna: Ah, the thing everyone says they want but nobody has time to write. Lucas: Exactly. And I want to make a specific case today: that structured, living documentation — not a stale wiki page, not comments in a config file, but a real runbook — is probably the single highest-leverage investment a sysadmin can make. Luna: Define 'living documentation' for me. Because we've all seen the Confluence graveyard. Lucas: Right. A living document is one that's embedded in your workflow. It lives alongside your code, it gets reviewed in the same pull request that changes a server configuration, and it gets tested — yes, tested — the same way you'd test a deployment script. Luna: Tested how? You can't unit test a paragraph. Lucas: No, but you can automate checks. For example, you can have a CI job that runs every time someone updates a runbook in your repository — it checks that every command in the runbook actually parses, that every referenced environment variable exists in your vault, that every URL returns a 200. I've seen teams do this with a simple shell script and a Markdown linter. Luna: Okay, that's interesting. What's the concrete payoff here? Give me a real example. Lucas: Sure. There's a well-known story from a mid-size SaaS company — I'm not naming them but you'd know the product — where a senior engineer spent two solid weeks building a runbook repository for their core platform. The team was skeptical. Two weeks is a lot of time when you're heads-down on feature work. Luna: I can imagine the pushback. 'We'll document it later.' Lucas: Exactly. But six months later, they had a PagerDuty escalation in the middle of the night. A critical database migration had stalled, and the on-call engineer — who was new to the team — pulled up the runbook, followed the step-by-step recovery procedure, and had the system back up in under forty minutes. The senior engineer estimated that without the runbook, diagnosis alone would have taken two to three hours. Luna: So the runbook saved, what, maybe forty person-hours in a single incident? That's a pretty good return on two weeks of investment. Lucas: That's the math. And the thing is, most teams don't have that story because they never make the upfront investment. They rely on tribal knowledge — 'Bob knows how the load balancer works' — and Bob is on vacation or Bob left the company. Luna: Or Bob is the one getting paged at 3 AM and he's too tired to explain it. Lucas: Right. So the argument isn't really about documentation for documentation's sake. It's about reducing the mean time to recovery. Every minute you spend writing a runbook is a minute you buy back during the next incident — and you will have a next incident. Luna: What's the best format for this? I've seen teams use everything from plain text files to elaborate wikis. Lucas: I'm a fan of Markdown in a Git repository. It's version-controlled, it's reviewable, it's searchable, and it doesn't require a special tool to read or edit. You can generate a static site from it if you want, but the source of truth is just files in a repo. Luna: That's basically the 'docs as code' philosophy. But what about teams that aren't comfortable with Git? Not every sysadmin is a developer. Lucas: Fair point. But I'd argue that if you're managing servers in 2026, you probably should be comfortable with Git. Even if it's just the basic clone, commit, push cycle. The overhead is tiny compared to the benefit. Luna: Alright, let's talk about the social challenge. How do you get a team to actually maintain documentation? Because the hardest part isn't writing it the first time — it's keeping it current. Lucas: The most effective pattern I've seen is to make documentation a part of the definition of done for any operational change. If you're changing a cron job, you update the runbook. If you're adding a new port to the firewall, you update the runbook. The pull request that changes the infrastructure also changes the documentation. Luna: That's a cultural shift. It requires buy-in from the team lead, maybe even from management. Lucas: It does. And the way to sell it is to frame it as an investment in team resilience. Not as 'we need to write more docs' but as 'we need to reduce the time it takes to recover from an outage.' That's a metric management cares about. Luna: Okay, let's get practical. What should a runbook actually contain? Can you sketch a minimal template? Lucas: Sure. Every runbook should start with a one-line summary of what service or system it covers. Then a list of prerequisites — what access you need, what credentials, what tools installed. Then a step-by-step recovery procedure, written in the imperative. 'SSH into the bastion host. Run this command. Check this log file. If you see this, do that.' No ambiguity. Luna: Should it include troubleshooting trees? Like 'if X doesn't work, try Y'? Lucas: Absolutely. The best runbooks anticipate common failure modes. For a database failover, for example, you might have a section for 'primary is unreachable', another for 'replica lag is too high', another for 'quorum lost'. Each with its own steps. Luna: That's a lot of writing. But I guess once it's written, it's reusable. Lucas: Exactly. And you don't have to write it all at once. Start with the most common incidents. The ones that page your team at 2 AM. Document those first, then expand over time. Luna: Alright, I'm sold on the concept. But there's a practical question: how do you make sure the runbook is actually correct? It's a living document, but if no one tests the steps, they might be outdated or wrong. Lucas: That's where automation comes in. Some teams schedule a 'runbook drill' — once a quarter, they deliberately simulate an incident and have an on-call engineer follow the runbook start to finish. They time it, they note where the runbook was unclear or wrong, and they file a pull request to fix it. Luna: That's like a fire drill for your servers. I love that. Lucas: Yeah, and it doubles as training for new team members. They get to practice the recovery procedure in a safe environment instead of learning it during a real outage. Luna: Let me push back a little. Isn't there a risk that runbooks become crutches? That people stop thinking critically and just follow steps blindly? Lucas: That's a valid concern. But I'd argue that a runbook is a tool, not a replacement for understanding. The goal is to handle the routine stuff efficiently so you have mental energy left for the novel problems. If you're following the runbook for a standard failover, great. If something unexpected happens, the runbook can't help you — but at least you're not wasting brainpower on the basics. Luna: Fair. And it's better than the alternative — frantically searching through old emails or Slack messages for 'that one command Bob used last time'. Lucas: Exactly. That's the chaos that structured documentation eliminates. Luna: You know, before we go further, I want to mention something. If today's episode was useful to you and you want to keep it ad-free, buy me a coffee dot com slash fexingo helps. Lucas: Yeah, it really does. We don't run ads — that's a deliberate choice. If you want to support that, buy me a coffee dot com slash fexingo is the place. And we mean it, even a couple of dollars a month makes a difference. Luna: Alright, back to the docs. So we've talked about the what and the why. Let's talk about the how — specifically, how do you get started if your team has zero documentation right now? Lucas: Start small. Pick one service — the most critical one, the one that pages most often. Write a one-page runbook for that service. Just the essentials: how to log in, how to check status, how to restart it, how to escalate. Put it in a Markdown file in a Git repo. That's it. Luna: And then what? Hope people use it? Lucas: No. You make it the default. When the next incident happens, the on-call engineer's first step is to open that runbook. If they can't find the answer, they update it. You build the habit. Luna: What about tools? Are there any specific ones you recommend for hosting these runbooks? Lucas: There are plenty. You can use GitHub or GitLab with a static site generator like MkDocs or Docusaurus. There's Read the Docs for hosted documentation. Some teams like Notion or Coda for a more collaborative feel. The tool matters less than the discipline of keeping it version-controlled and reviewed. Luna: I've heard of teams using something called 'runbook automation' — where you can actually execute the steps from the runbook via a chat bot or a web UI. Is that the same thing? Lucas: That's a step beyond what we're talking about. That's turning your runbook into executable code — like using Rundeck or StackStorm. It's powerful, but it's also a bigger investment. For most teams, a well-maintained Markdown file is a huge improvement over nothing. Luna: So the bar is low. That's encouraging. Lucas: It really is. The hardest part is the first page. Once you have one runbook that works, you have a template. Then it's just a matter of expanding coverage over time. Luna: Let me ask about a specific pain point: credentials. How do you document which commands need which credentials without putting secrets in the runbook? Lucas: Smart question. You never put secrets in the runbook. Instead, you reference a secret store. Say 'Run vault login, then fetch the token for service X.' Or 'Use the admin credentials from the team password manager, entry named Y.' The runbook tells you where to find the secret, not what the secret is. Luna: And if the password manager goes down? Then you're stuck. Lucas: That's a valid edge case. Some teams have a break-glass procedure — a physical envelope with emergency credentials, or a separate offline backup. That should also be documented in a separate, highly restricted runbook. Luna: Alright, I'm convinced. Let's talk about how to sell this to a team that's resistant. What's your pitch? Lucas: I'd say: 'Look, we all hate being paged at 3 AM. We can't prevent every incident, but we can make sure that when it happens, we have a clear, tested path to recovery. Investing a few hours now will save us hours of sleep later.' It's hard to argue with that. Luna: And if they still don't buy in? Lucas: Then you start documenting your own workflows. Even if it's just for yourself. Eventually, someone else will benefit from it, and the value will become visible. It's a bottom-up change. Luna: So the key takeaway: start small, start now, and make it a habit. Lucas: Exactly. One runbook for one service. That's all it takes to start building a culture of operational resilience. Luna: And if you need inspiration, there are plenty of open-source runbook templates out there. I'll link to a few in the show notes. Lucas: Good idea. And if you have a runbook war story — good or bad — we'd love to hear it. Drop us a note. Luna: Alright, that's our case for structured server documentation. Until next time, keep your servers running and your runbooks fresh.