Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / The Case for Filesystem Snapshots Before Package Updates
Transcript
- Lucas: Let me ask you something, Luna. When was the last time you ran apt upgrade or yum update on a production server and didn't feel a tiny knot in your stomach? Luna: Honestly? Every single time. Even with staging environments, there's always that moment of 'please don't break networking, please don't break networking.' Lucas: Right. And most of us rely on package manager rollback — which, let's be honest, is a fragile safety net. Today I want to make the case for something that's saved my skin more times than I can count: filesystem snapshots taken right before any package update. Luna: So you're talking LVM snapshots, ZFS snapshots, that kind of thing? Lucas: Exactly. And I'm not talking about replacing backups. Snapshots are not backups. But they are the fastest way to undo a failed update with zero dependency on whether the package manager can figure out its own mess. Luna: I've definitely heard the mantra 'snapshots are not backups' but I've also seen people treat them as such. What's the real distinction? Lucas: A backup is your safety net for hardware failure, data corruption, accidental deletion. You store it off-box, ideally off-site. A snapshot is a point-in-time image of your filesystem on the same storage. If your disk dies, the snapshot dies with it. But for the specific use case of 'I just ran an update and now Apache won't start' — a snapshot is perfect. It's instant, it's low-cost, and the recovery is literally a reboot into the snapshot. Luna: So the window of protection is narrow — just that update window. But within that window, it's incredibly reliable. Lucas: Exactly. Let me give you a concrete example. A few years ago I was managing a fleet of Debian-based web servers. One Tuesday, the standard apt update pulled in a new kernel and a new version of systemd. The update finished fine, but on reboot, the network interfaces didn't come up. The driver module had silently changed its naming scheme. No SSH, no console access except through a remote KVM. I was staring at a bricked server. Luna: Ouch. And you had a snapshot? Lucas: I did. Before every update, I had a cron job that ran a simple LVM snapshot. On that server, I just rebooted into the snapshot volume, which reverted both the kernel and systemd. Five minutes later, the server was back online with the old kernel. Then I could investigate the driver issue in a controlled way. Luna: So the snapshot saved you from a potentially hours-long recovery — maybe even a reinstall. Lucas: Exactly. And the snapshot took maybe two seconds to create. That's the cost to benefit ratio I want people to appreciate. A couple of seconds of I/O, and you get a guaranteed rollback path. Luna: Alright, I'm sold. But how do you actually set this up? Let's talk mechanics. Lucas: Sure. There are two main paths: LVM and ZFS. Let's start with LVM since it's more common on enterprise Linux. You need a volume group with free space — at least as much as the size of the logical volume you're snapshotting, though thin provisioning can reduce that. Then the command is literally lvcreate --snapshot --size 10G --name pre-update-snap /dev/vg_root/root. Luna: And that 10 gigabyte — that's the space allocated for changes during the snapshot's lifetime. If the original volume changes more than 10 gigabytes of data, the snapshot becomes invalid, right? Lucas: Exactly right. So you need to size it large enough to cover the expected writes during the update. For a typical package update, a few gigabytes is plenty because you're mostly touching binaries and libraries, not user data. But if you have a busy database server, that's a different story — you'd want a bigger snapshot or use a different strategy. Luna: And ZFS — that's even more elegant because it has built-in snapshots as a first-class feature. No pre-allocation needed. Lucas: ZFS snapshots are basically instant and consume zero space initially — they only grow as data changes. So command is zfs snapshot pool/dataset@pre-update. That's it. And rolling back is zfs rollback pool/dataset@pre-update. But here's the gotcha with ZFS: you can't roll back to a snapshot if there are newer snapshots, unless you destroy the newer ones. So you need a clean snapshot hierarchy. I usually delete old snapshots before creating a new pre-update one. Luna: That's a good practice. So for both LVM and ZFS, the workflow is: before update, create snapshot. After update, if everything's fine, delete the snapshot. If something breaks, roll back. Simple. Lucas: Exactly. And you can script this. I have a bash script called pre-update-snap that I run before any manual or automated update. It checks the filesystem type, creates the snapshot with a timestamp in the name, and logs it. Takes about 30 seconds to write. Luna: I want to pause on something. A lot of sysadmins I've talked to say 'I just use package manager rollback' or 'I have backups anyway'. Why do you think snapshots aren't more common? Lucas: I think it's a combination of things. First, the perceived complexity — especially with LVM, people are intimidated by volume groups and logical volumes. Second, the 'it won't happen to me' bias. But the biggest reason is probably that most sysadmins learn this the hard way, after a failed update causes an outage. This episode is my attempt to help people learn the easy way. Luna: And I think there's also a misconception that snapshots are expensive in terms of disk space or performance. But as you said, for the update window, the cost is minimal. Lucas: Right. And if you're using thin provisioning in LVM or ZFS, the space consumption is even lower. The real cost is the discipline to remember to take the snapshot before every update. That's why I automate it. Luna: You know, speaking of discipline and small habits that make a big difference — that reminds me of something I've been meaning to touch on. Lucas, we run this show ad-free, and honestly, that's because of listeners who support us directly. If today's tech conversation gave you something usable — a command, a workflow, a habit — consider buying us a coffee. It's buy me a coffee dot com slash fexingo. A couple of dollars a month genuinely keeps these episodes going. No pressure, just an option. Lucas: She's right. Every bit helps cover hosting, editing, and keeps the show independent. And we really do appreciate it. Alright, back to snapshots. Lucas: So let's talk about one more real-world scenario that snapshots handle perfectly: the 'I accidentally ran update on the wrong server' situation. Luna: Oh, that's terrifying. You SSH into a production box instead of the staging box and hit upgrade before you realize. Lucas: I've done it. We've all done it. And if you have a snapshot from, say, an hour ago, you can roll back cleanly. But you need to have that snapshot already. So the habit is: take a snapshot before any operation that could change system state. Not just package updates — think configuration management runs, manual config edits, even some scripted maintenance tasks. Luna: That's a good rule of thumb. Any operation that could break the server, take a snapshot first. But what about the snapshot's lifetime? Do you keep them around for a while? Lucas: No. Snapshots are short-lived. If the server runs stable for, say, 24 hours after the update, I delete the snapshot. They're not meant for long-term retention. For that, you have backups. The snapshot is just a quick undo button. Luna: And you can automate the cleanup too. A cron job that deletes snapshots older than a day. Lucas: Exactly. So the full workflow is: pre-update snapshot, perform update, verify health, delete old snapshot. All scripted. And for extra safety, you can even have the snapshot automatically removed if a health check passes. Luna: I want to ask about recovery time. If you do need to roll back, how long does it take? Lucas: For LVM, you need to unmount the filesystem, run lvconvert --merge, and then remount. That's maybe a minute. For ZFS, zfs rollback is almost instant. But you might need to reboot if the kernel or critical libraries changed. In my kernel example, I had to reboot into the snapshot, which took about two minutes. Compare that to restoring from backup — which could be hours. Luna: So the time savings are enormous. And the confidence boost is even bigger. I think I'm going to start implementing this on my own servers. Lucas: Do it. Start with one non-critical server, script it, test the rollback process. You'll sleep better. And once you have the habit, it becomes second nature. Luna: One last question: are there any cases where snapshots won't help? Lucas: Sure. If the update corrupts your data files — like a database migration that rewrites tables — a snapshot won't help because the data has already changed. Snapshots are for reverting system state, not application data. Also, if you have multiple volumes and only snapshot one, a rollback might leave you with inconsistent state across volumes. So you need to snapshot all relevant volumes together. Luna: Good point. Consistency groups matter. But for the common case of a package update affecting system binaries and libraries, snapshots are a superb tool. Lucas: Exactly. And that's the takeaway: for the specific high-risk, low-duration event of a server update, a snapshot is your best friend. It's cheap, fast, and reliable. Add it to your sysadmin toolkit if it's not there already. Luna: I'm convinced. I'll be writing that pre-update-snap script tonight. Lucas: Good. And if you do, you can buy me a coffee to celebrate. Just saying. Luna: Ha. Alright, that's all for this episode. Until next time, keep your filesystems snappy.