Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Using Linux Server Etckeeper for Automated Config Version Control
Transcript
- Lucas: Alright, let's talk about something that feels almost too simple to be this useful — etckeeper. Luna: I've heard the name, but I'll admit I've never actually set it up. What's the core idea? Lucas: It's a tool that automatically puts your entire /etc directory under version control — Git, Mercurial, Bazaar, whatever you prefer — and commits changes every time a package manager touches a config file, or whenever you manually make a change. Luna: So it's like a safety net for the one directory that, if you break it, your server breaks. Lucas: Exactly. And the thing is, most sysadmins already know they should version-control their configs, but they don't because it feels like overhead. Etckeeper removes that friction entirely. You install it, it hooks into apt or yum or pacman, and from that point on, every pre-install and post-install event triggers a commit. Luna: Let's walk through the setup. I want to know how fast this actually is. Lucas: On a Debian-based system, it's literally 'apt install etckeeper'. Then you edit '/etc/etckeeper/etckeeper.conf' to set your VCS — I use Git. That's it. The first run automatically initializes a Git repo in /etc and commits all existing files. The whole thing takes under two minutes. Luna: Wait — the first commit includes everything in /etc? That could be thousands of files, including stuff like shadow and passwd. Lucas: It does, but by default it respects '.gitignore' patterns. Etckeeper ships with a sensible ignore list — things like '.pid', and certain cache directories. It also automatically runs 'git add -A' after commits, so you never have to manually stage. Luna: What about sensitive files? I don't want passwords or keys in a repo. Lucas: Good point. The default config actually includes '/etc/shadow', '/etc/gshadow', and '/etc/security/opasswd' in the ignore list. But if you have custom sensitive files, you can add them to '.gitignore' inside '/etc'. Or you can use etckeeper's commit hooks to encrypt certain files before they get committed. There's a built-in option to use 'gpg' if you really need that. Luna: So you're saying I can set this up in a couple minutes, and from then on, every time I run 'apt upgrade' or edit an Nginx config, it auto-commits? Lucas: Yes. And not just package manager events. If you manually edit a file with 'vim' or 'nano', etckeeper hooks into the shell — if you have the bash-preexec hook installed — to detect the change. But more robustly, it runs a daily cron job that checks for uncommitted changes and commits them. Luna: That's actually smarter than what a lot of people do, which is manually copying /etc to a backup directory and hoping they remember which copy is the good one. Lucas: Right. And the real power shows up when something goes wrong. Let me give you a concrete example. I had a production Nginx server where I was tweaking the SSL config. I changed 'ssl_protocols' and accidentally disabled TLS 1.2, leaving only TLS 1.3, but one of our upstream APIs still needed 1.2. Suddenly, half the proxied requests started failing. Luna: Oof. And if you didn't have etckeeper, you'd be trying to remember what the original file looked like. Lucas: Exactly. Instead, I just ran 'cd /etc/nginx && git log' and saw the commit from five minutes ago: 'auto-commit: /etc/nginx/nginx.conf'. Did a 'git show' to see the diff, then 'git checkout HEAD~1 -- nginx.conf' and reloaded. Everything back to normal in under thirty seconds. Luna: That's a huge time saver. But what about the case where you accidentally mess up SSH config and lock yourself out remotely? Lucas: That's the nightmare scenario. And etckeeper can help even there, but you need a fallback access method — like a BMC, IPMI, or a console. If you have that, you can boot into single-user mode, go to /etc/ssh, do a 'git checkout HEAD~1 -- sshd_config', and restart sshd. The commit history is local, so you don't need network access. Luna: So as long as you have out-of-band access, you can roll back SSH config even if you broke the service entirely. Lucas: Yes. And there's another trick: you can use etckeeper's 'git bisect' to find which change broke something. If you have a long series of config changes over a week and suddenly a service stops working, you can bisect through commits to find the exact change that introduced the bug. It's like debugging code, but for server configs. Luna: That's really elegant. But I'm wondering — does etckeeper cause any performance issues? Especially on a busy server with lots of package updates? Lucas: No, because the commits are incremental and Git handles them very efficiently. Even on a server with thousands of files in /etc, the initial commit might take a couple seconds, but subsequent commits are nearly instant. The disk usage is also minimal — we're talking a few megabytes for months of history. The real overhead is basically zero. Luna: How does it handle things like permissions and ownership? Git doesn't track those by default. Lucas: Etckeeper has a metafile system. It stores metadata like permissions, ownership, and ACLs in a separate file called '.etckeeper' that's committed alongside the files. When you checkout a previous commit, etckeeper's 'pre-install' hook restores those metadata. So it's not just file contents — it's the full state. Luna: That's a killer feature. I can think of a dozen times I've restored a config file from a tarball only to find the permissions were wrong and the service still wouldn't start. Lucas: Right. And the tool integrates with your existing workflow. If you're already using configuration management like Ansible or Puppet, etckeeper isn't a replacement — it's a safety net underneath. It captures changes that happen outside of your CM, like manual fixes during an incident or changes made by another admin who forgot to use the playbook. Luna: So it's essentially a 'last resort' audit trail. And since it's so lightweight, there's really no reason not to set it up on every server. Lucas: I'd go further — I'd say it should be part of your baseline server image. Just like you install SSH and a monitoring agent, you should install etckeeper. It costs nothing and can save you hours of downtime. Luna: Alright, you've sold me. Let's do a quick setup walkthrough for someone listening who wants to try it today. Lucas: Sure. On any Debian or Ubuntu system: 'sudo apt update && sudo apt install etckeeper git'. Then open '/etc/etckeeper/etckeeper.conf' and set 'VCS="git"'. That's it. The first commit happens automatically. On RHEL or CentOS: 'sudo yum install epel-release && sudo yum install etckeeper git', same config step. Luna: What about the shell hook for manual edits? Does that need extra setup? Lucas: The shell hook requires installing 'etckeeper' with the 'bash-completion' package and sourcing the hook in your bashrc. But honestly, the cron job is sufficient for most cases. It runs daily and commits any uncommitted changes. If you want immediate commits on manual edits, you can configure the 'pre-edit' and 'post-edit' hooks, but I find the cron approach plus the package manager hooks cover 99% of changes. Luna: One thing I want to clarify: if I use etckeeper on a server that's also under full configuration management with Ansible, will the Ansible runs cause tons of git commits? Lucas: Yes, but that's actually useful. Each Ansible run that changes a config file will create a commit. That gives you a granular history of exactly what Ansible changed and when. If a playbook update breaks something, you can see the commit tied to that run and roll it back independently. It's like having an audit log. Luna: That makes sense. So it's not redundant — it adds visibility. Lucas: Exactly. And if you're in a regulated environment that requires change logs, etckeeper gives you a tamper-evident history. The git log is immutable unless someone forces-pushes to a remote. And you can push the repo to a remote server for off-site backup. Luna: Wait — you can push /etc as a Git repo to a remote? That seems like a security risk. Lucas: If you're careful, it's fine. You'd want to use a private repository and ensure sensitive files are ignored. Many organizations push to an internal GitLab or Gitea instance with strict access controls. The remote gives you a backup if the local disk fails. But even without remote, the local git history is resilient — it's stored in '/etc/.git'. Luna: So worst case, if the server dies completely, you lose that history. But if you have remote, you don't. Lucas: Correct. And you can automate the push with a cron job. Etckeeper even has a 'push' hook that runs after each commit if you configure it. But I'd recommend using a separate SSH key with limited permissions for that. Luna: Let's talk about a real rollback scenario. Suppose I have a server where I changed the Nginx config, reloaded, and now the site is broken. Walk me through the exact commands. Lucas: Okay. First, you run 'cd /etc/nginx' and then 'git log --oneline' to see recent commits. You'll see something like 'abc1234 auto-commit: /etc/nginx/nginx.conf'. Then you check the diff with 'git show abc1234'. If that's the bad commit, you restore the previous version of just that file: 'git checkout abc1234^ -- nginx.conf'. Then 'sudo nginx -t' to test, and 'sudo systemctl reload nginx'. Luna: And that works even if you've made multiple changes since then, because you're only reverting that one file. Lucas: Exactly. Git's checkout with a file path is a restore operation. It doesn't change the commit history, it just updates the working tree. If you want to revert multiple files from that commit, you can do 'git revert abc1234' — that creates a new commit that undoes the changes. That's cleaner because it preserves history. Luna: I like that. It's like an 'undo' button for sysadmins. Lucas: It really is. And once you start using it, you'll wonder how you lived without it. I've had it on every server I manage for the last five years. Luna: Alright, I'm convinced. Let me ask — are there any gotchas or limitations I should know about? Lucas: A few. First, etckeeper doesn't handle binary files well — the diffs are useless. But most config files are text, so that's minor. Second, if you have a huge number of files in /etc — like thousands of package-specific configs — the initial commit might take a while, but it's a one-time thing. Third, if you use 'etckeeper commit -m' manually, it merges with the auto-commits, which can be confusing. But honestly, these are small trade-offs. Luna: Sounds like the benefits far outweigh the downsides. I'm going to set this up on my test server today. Lucas: Do it. And once you have it, you'll start noticing situations where it saves you. Like when you're troubleshooting a DNS issue and realize someone changed resolv.conf two days ago — you can see exactly what and when. Luna: That's a perfect example. The kind of thing that's nearly impossible to track down without version history. Lucas: Exactly. So, I'd say etckeeper is one of those tools that, once it's in place, you never think about it until you desperately need it. And when you do, it's a lifesaver. Luna: Honestly, if this episode gave you something you can use — maybe it saves you that one headache someday — that's the point. If today was worth a coffee to you, that's the link: buy me a coffee dot com slash fexingo. Just a small way to keep this show ad-free and focused on practical stuff like this. Lucas: Yeah, we don't do sponsors or ads, so listener support is what keeps it going. And we appreciate everyone who chips in, no matter the amount. Luna: Alright, back to the tech. So, Lucas, you mentioned that etckeeper works with other VCS like Mercurial. Is there a reason to use something other than Git? Lucas: Not really, unless you already have a strong preference or your organization uses Mercurial. Git is the most common and well-supported. The etckeeper project itself recommends Git. But the tool is agnostic — you set 'VCS' in the config and it adapts. I've never met anyone who regretted using Git for this. Luna: Fair enough. One more thing — what about containers? If I'm running a containerized app, should I use etckeeper inside the container? Lucas: Generally no. Containers are ephemeral, and you should manage their config through Dockerfiles or orchestration. Etckeeper is for persistent servers — physical or virtual machines that have a long lifespan. For containers, you'd use something like a GitOps workflow. Luna: Good distinction. So it's really for the servers that live for months or years. Lucas: Exactly. And those are the ones where config drift happens. Etckeeper gives you a way to detect and reverse that drift without needing a full CM system. Luna: Alright, I think we've covered the essentials. Any final pro tip? Lucas: Yes — run 'etckeeper gc' periodically to clean up unused Git objects and keep the repo small. And if you're pushing to a remote, set a cron job for that. Other than that, just let it run. Luna: Great. Thanks for walking us through etckeeper — definitely adding it to my toolkit. Lucas: Glad to. Next time, we'll talk about a tool that helps you visualize those commit histories in a more user-friendly way. But for now, go set up etckeeper.