Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Rescue a Server With a Broken Root Filesystem
Transcript
- Lucas: You SSH into a server and it's gone. Not a network issue — the box is powered on — but nothing responds. You drive to the datacenter or rack, plug in a monitor and keyboard, and you're staring at a kernel panic that says 'VFS: Unable to mount root fs on unknown-block'. That's the moment your Linux server has a broken root filesystem. Luna: That's pretty much the worst boot-time error you can get. The kernel can't even find the partition with /sbin/init. Lucas: Exactly. And the default response from a lot of sysadmins — especially if they're stressed and under time pressure — is to immediately reach for the live USB, boot a full desktop environment, and start clicking around. But you don't need a GUI. You can rescue the system from the initramfs shell that the kernel drops you into, if you know how. Luna: That's a good point. Most modern distributions give you a rescue shell before they give up entirely. It's just hidden behind a timeout or an option in the bootloader. Lucas: Right. On RHEL or CentOS, if the boot fails, the dracut initramfs will drop you into a shell automatically after a few retries. On Ubuntu, if you hold Shift or press Esc during boot, you can get to the GRUB menu, then select the recovery mode entry. That gives you a root shell — but it's a limited, read-only environment. The filesystem is mounted read-only because it failed to mount read-write. Luna: So the first thing you want to do is figure out whether the damage is on the filesystem metadata or the actual disk block layer. Lucas: Yeah. You can start by checking dmesg — that kernel ring buffer — because it usually has a clear error message. Something like 'EXT4-fs error: ext4_find_entry: reading directory #xxxxx offset 0'. That tells you it's a filesystem-level corruption, not a dead disk. So you can attempt a repair with fsck. Luna: But fsck on a mounted filesystem is dangerous. You need to unmount it first, which is tricky when it's your root partition. Lucas: Exactly. In the initramfs shell, you can remount root read-only or try to fsck it directly. The trick is that the initramfs already has the root mounted — but usually read-only. You can check with 'mount | grep root' and see if it's on /sysroot or /root. In Ubuntu's recovery mode, it's usually on /host or /root. You then remount it read-only if it isn't already, and run fsck on the block device. Luna: And you need to know what the block device is. That's not always obvious if you're running LVM or encryption. Lucas: True. You can find it by looking at /proc/mounts or by running 'blkid' — that lists all block devices with their UUIDs and filesystem types. For LVM, you'll see something like /dev/mapper/vg-root. For encrypted LUKS, you need to unlock the device first with cryptsetup luksOpen. Luna: So once you have the device path, you run 'fsck -f /dev/sda2' or whatever it is. The -f forces a check even if the filesystem appears clean. Lucas: Right. And fsck will ask you yes or no for each repair — you can pass -y to auto-yes, but I prefer to answer manually the first time to see what it's doing. After fsck completes, you try to remount the root read-write. If that works, you exit the initramfs shell and the system should continue booting. Luna: But what if fsck can't fix it — or you need to do something more invasive, like reinstall GRUB or rebuild the initramfs? Lucas: Then you need a full chroot. You mount the root partition — say on /mnt — and you bind-mount /proc, /sys, /dev into that chroot. So the sequence is: mount /dev/sda2 /mnt, mount --bind /proc /mnt/proc, mount --bind /sys /mnt/sys, mount --bind /dev /mnt/dev. Then chroot /mnt. Now you're inside your broken system with full access to its binaries. Luna: And from there you can reinstall GRUB, rebuild the initramfs with dracut or mkinitramfs, or even fix a broken fstab. Lucas: Exactly. For GRUB, you run 'grub2-install /dev/sda' on RHEL, or 'grub-install /dev/sda' on Debian, then 'grub2-mkconfig -o /boot/grub2/grub.cfg' or 'update-grub'. For the initramfs, on RHEL it's 'dracut --force', on Ubuntu it's 'update-initramfs -u'. Both will regenerate the initial ramdisk based on the current kernel and modules. Luna: One thing I've seen trip people up: after chroot, the shell environment is minimal — you might not even have $PATH set correctly. So you may need to run '/sbin/grub2-install' or source /etc/profile. Lucas: Great point. I always do 'export PATH=/usr/sbin:/usr/bin:/sbin:/bin' right after chrooting. Also, if your system uses SELinux, you might need to either setenforce 0 or create an autorelabel file — 'touch /.autorelabel' — so SELinux relabels the filesystem on the next boot. Otherwise, after chroot, SELinux contexts get messed up and services might fail to start. Luna: That's a subtle one. So the full rescue process is: identify the root device, fsck it, mount it, chroot in, fix the bootloader and initramfs, maybe handle SELinux, and then exit and reboot. Lucas: Right. And if the root filesystem is so damaged that even fsck can't repair it? Then you need a backup. You boot from a live USB, mount a healthy partition or external drive, and restore from your last known good backup. That's the hard truth — no amount of clever chrooting can fix a physically corrupted superblock beyond what fsck can handle. Luna: That segues nicely into something we touch on a lot in this show. Honestly, if today's episode gave you a concrete rescue procedure you didn't know before, and it's worth the price of a coffee to you, that's the link — buy me a coffee dot com slash fexingo. It helps keep the show ad-free and focused on real sysadmin skills. Lucas: Yeah, listener support makes a difference. And we appreciate it. So, back to the rescue scenario — what if you don't have a live USB handy and you're at a remote site with no physical access? Luna: Then you're relying on out-of-band management — IPMI, iDRAC, or iLO — to mount a remote ISO. Most enterprise servers have that. You boot the live ISO over the network, and then you follow the same steps we just described. Lucas: Exactly. And if the server is virtual? You can attach the virtual disk to another VM and fix it from there. The principles are the same — it's just a different way to get access to the block device. Luna: I want to emphasize the backup part. I've seen sysadmins spend hours trying to fsck a completely wrecked ext4 partition when they could have booted from a live CD and restored from a ZFS snapshot in ten minutes. Lucas: Absolutely. The rescue technique we covered today is for when the filesystem is repairable. But you need to have a clear threshold — if fsck starts asking 'Multiply-claimed block?' and 'Duplicate blocks?' more than a handful of times, you're probably better off restoring from backup. The time you spend deciding that is time you could spend restoring. Luna: And test your backups regularly. There's nothing worse than discovering your backup script silently failed for six months when you need it most. Lucas: Exactly. So to recap: if you see VFS unable to mount root, don't panic. Check dmesg, identify the device, run fsck, chroot if needed, rebuild bootloader and initramfs, fix SELinux, and reboot. And if that doesn't work, restore from backup. It's a skill every sysadmin should have in their back pocket. Luna: And if you want to see the exact commands we used, we'll put a quick reference on the show notes page. But the key is to practice this in a test environment before you need it for real. Lucas: Good idea. Next episode, we'll talk about what to do when the disk itself is failing — not just the filesystem. SMART errors, reallocated sectors, and knowing when to replace the drive before it's too late.