Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Rescue Mode for Recovery
Transcript
- Lucas: You update a kernel on a production Linux server, reboot, and the machine goes dark. No SSH, no ping, no console output beyond a blinking cursor. This is the moment every sysadmin dreads, and today we're going to walk through exactly how to get that server back without reinstalling. Luna: I've definitely been there. That moment when you realize the kernel panic message scrolled by too fast to read on the IPMI console. What's the first move? Lucas: First, you need to interrupt the boot process. When the BIOS or UEFI hands off to GRUB, you'll see the GRUB menu — usually you have to hold Shift on older BIOS systems or press Esc on UEFI. If you miss it, just force a hard reset and try again. Once you're in the GRUB menu, highlight your default kernel entry and press 'e' to edit the boot parameters. Luna: So you're editing the GRUB command line before the kernel even loads. What are you looking for? Lucas: You're looking for the line that starts with 'linux' — that's the kernel command line. At the end of that line, add the word 'single' or '1' to boot into single-user mode, or 'init=/bin/bash' to drop straight into a root shell. I usually go with 'init=/bin/bash' because it bypasses systemd entirely and gives you a bare shell without any services starting. Luna: That makes sense — if the kernel itself is failing, you don't want systemd potentially crashing during boot. But once you're in that shell, what's the filesystem situation? Lucas: The root filesystem is mounted read-only at that point. So the very first thing you do is remount it read-write: 'mount -o remount,rw /'. Without that, you can't change anything — can't remove a bad kernel package, can't rebuild the initramfs. Then you've got a live root filesystem to work with. Luna: Okay, so now you're in a shell with a writable root. If it's a kernel panic from a new kernel, what's the fix? Lucas: You want to remove the bad kernel package and install the previous known-good kernel. On a Debian-based system, you'd run 'apt list --installed | grep linux-image' to see what kernels you have. Then 'apt remove linux-image-X.X.X-X-generic' for the bad one. On rhel based systems, it's 'rpm -qa | grep kernel' and then 'yum remove kernel-X.X.X-X'. After removal, run 'update-grub' on Debian or 'grub2-mkconfig -o /boot/grub2/grub.cfg' on RHEL. Luna: But what if the problem isn't the kernel itself — what if it's a missing driver in the initramfs? I've had a server refuse to boot because the NVMe driver wasn't included after a kernel update. Lucas: That's a great point. So sometimes the kernel boots fine, but it can't find the root filesystem because the initramfs doesn't have the right storage driver. In that case, you chroot into the broken system from a live CD or rescue environment. Boot from a USB stick or the vendor's rescue image, mount your root partition to /mnt, mount /boot and maybe /boot/efi, then chroot into that mounted root: 'chroot /mnt /bin/bash'. Luna: Chrooting — that's the classic way to fix a system from the outside. And once you're in the chroot, you can rebuild the initramfs. Lucas: Exactly. On Debian/Ubuntu, you'd run 'update-initramfs -u -k all' to rebuild for all installed kernels. On RHEL/Fedora, use 'dracut --force --regenerate-all'. On Arch, it's 'mkinitcpio -P'. You want to make sure the initramfs includes the modules your root device needs. If you know the driver name, you can explicitly add it in /etc/mkinitcpio.conf or /etc/dracut.conf.d/. Luna: I've also had cases where the system booted but the network wasn't coming up because the NIC driver was missing from the initramfs. Same fix — rebuild the initramfs with the right modules. Now, what about servers you can't physically touch? A lot of our listeners manage headless machines. Lucas: Right, and that's where serial console becomes your lifeline. If you configure GRUB to output to the serial port, you can interact with the boot menu over a serial connection. You need to add 'console=ttyS0,115200' to the kernel command line in /etc/default/grub, and also set GRUB_TERMINAL=serial and GRUB_SERIAL_COMMAND='serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1'. Then run update-grub. Next time the server boots, you can connect via a serial console — either through IPMI SOL or a direct serial cable — and you'll see the GRUB menu. Luna: That means even if the server is headless and the network stack isn't up yet, you can still interrupt the boot and drop into rescue mode. That's huge for remote sites. Any gotchas with serial console rescue? Lucas: One thing: when you boot with 'init=/bin/bash' over serial, the shell might not have a proper terminal — no job control, no Ctrl+C. You can work around it by using 'setsid' or just getting comfortable with the bare environment. Also, if you're using UEFI, serial console setup is a bit trickier because GRUB's UEFI serial support isn't as mature. Some UEFI firmware doesn't expose a serial device until the OS boots, so you might need to fall back to IPMI virtual media or a recovery ISO. Luna: Speaking of recovery ISOs — do you recommend keeping a custom rescue ISO on hand? Something with your specific drivers and tools? Lucas: Absolutely. I have a USB stick with SystemRescue and a custom config that automatically mounts my luks encrypted root and drops me into a chroot. That saves me ten minutes of typing every time I need it. You can build one with dracut's 'live' feature or just use a standard live distribution and add a script. The key is test it before you need it. Luna: That's the real takeaway: practice your rescue procedure. Because when the server goes dark at 2 AM, you don't want to be reading man pages. Lucas: Honestly, if today's tech conversation gave you something usable, that's the link — buy me a coffee dot com slash fexingo. The smallest gesture keeps this ad-free and focused on practical content. Luna: Yeah, it really does make a difference. We keep the show independent because of listeners who chip in. Appreciate every single one. Lucas: So back to the nitty-gritty. Another common failure mode: the root filesystem itself is corrupted, and the kernel panics when it can't mount it. In that case, you need to fsck from a rescue environment. Boot with init=/bin/bash, remount root read-only if it's mounted, then run 'fsck -y /dev/sda1' or whatever your root partition is. If the filesystem is so broken that fsck can't fix it, you might need to restore from backup — but that's a different episode. Luna: Right, that's a whole other can of worms. But for the kernel update scenario, the process is pretty straightforward: interrupt GRUB, add init=/bin/bash, remount rw, remove the bad kernel, and rebuild initramfs. What about the case where you can't even get to GRUB — like a corrupted bootloader? Lucas: That's the worst case. If GRUB itself is corrupted, you need a live USB to chroot in and reinstall GRUB. Boot the live environment, mount your root partition to /mnt, mount /boot and /boot/efi if separate, then use 'grub-install --boot-directory=/mnt/boot /dev/sda' on BIOS, or on UEFI use 'grub-install --target=x86_64-efi --efi-directory=/mnt/boot/efi --bootloader-id=GRUB --recheck'. Then update the config. Luna: And that's why you should always have a live USB — or at least access to a recovery ISO via IPMI virtual media — before you start messing with bootloaders. Lucas: Exactly. And one last tip: if you're managing a fleet of servers, automate the rescue procedure. Write an Ansible playbook or a script that mounts root from a rescue environment, chroots, and rolls back the kernel. You can even bake it into your PXE boot image. That way, when a kernel update goes sideways, you can fix all the affected machines in parallel. Luna: That's smart — turn a manual panic into a one-command recovery. Any final advice for someone who's never done this before? Lucas: Set up a test VM, install a kernel, then intentionally break it — delete the kernel image or corrupt the initramfs — and walk through the recovery steps. Do it during office hours with a fresh cup of coffee. That way, when it happens for real at 2 AM, it's muscle memory.