Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose Linux Server Boot Failures with Initramfs
Transcript
- Lucas: You reboot a production server on a patch Tuesday, and instead of the familiar login prompt, you get a busybox shell. If that's ever happened to you, you know the feeling — the entire machine is sitting there, network down, no services, and you've got a pound sign and a very limited set of commands. Luna: That busybox prompt is the initramfs emergency shell. It's basically the kernel saying 'I couldn't mount the real root filesystem, so here's a tiny rescue environment.' The question is: why did it fail? Lucas: Exactly. And the answer usually falls into one of three buckets: a missing block device driver, a damaged filesystem, or a corrupted initramfs image itself. Today I want to walk through how you diagnose each scenario, because the fix is very different depending on which one you're dealing with. Luna: Let's start with the most common one I've seen — the missing driver. You upgrade your kernel, or you move a disk to a different controller, and suddenly the storage isn't there. Lucas: Right. So you're in the busybox shell. First thing: run 'ls /dev' and look for your root device. If you're expecting something like '/dev/sda1' or '/dev/nvme0n1p2' and it's just not there, the kernel didn't load the driver for that controller. Luna: And the fix isn't to manually load the module in the initramfs — because you're already in the rescue shell. The real fix is to rebuild the initramfs with the correct driver included. Lucas: Exactly. On most modern distributions, you use 'dracut' or 'mkinitcpio' depending on your distro. But if you can't boot into your installed system, you need a recovery image — either a live USB, or if you're in a data center, the out-of-band console might let you boot an alternative kernel. Luna: I've had good luck booting into a rescue mode from GRUB. You can edit the kernel command line right there — add 'single' or 'emergency' to get a shell without the initramfs, then rebuild. Lucas: That's a great point. If you can get to GRUB, highlight your kernel entry, press 'e' to edit, and append 'rd.break' — that drops you into the initramfs shell before it tries to mount root. You can then check what's available. But let's say you don't have that luxury; you're at the busybox prompt with no networking. Luna: The next thing I'd look at is the kernel messages. In the busybox shell, 'dmesg' might be available. If not, check 'cat /proc/kmsg' or look at the console output you saw during boot. Lucas: Yeah, the kernel often prints a clear error: 'VFS: Cannot open root device' or 'request_module: runaway loop modprobe'. That tells you exactly which module it tried and failed to load. Write that down, because it's the clue you need for the dracut rebuild. Luna: Now, the second bucket: damaged filesystem. That's a different symptom. The device shows up in /dev, but the mount fails with an I/O error or a superblock issue. Lucas: Right. So you see '/dev/sda1' in the device list. You try to mount it manually — 'mount /dev/sda1 /mnt' — and you get 'mount: /dev/sda1: can't read superblock'. At that point, you need to run fsck on that device. But careful: if the filesystem is badly corrupted, you want to run fsck from the rescue shell with the '-n' flag first to see what it would do without making changes. Luna: And if fsck can fix it, great. But sometimes the superblock is so damaged that fsck can't even find the backup. You might need to restore from a backup superblock using 'mke2fs -n' to list the backup locations. Lucas: That's an advanced move, but it's saved me a few times. The third bucket is a corrupted initramfs image itself. This can happen if the initramfs file on disk got truncated during an update, or if there's a bad sector on the boot partition. Luna: How do you tell the difference between a missing driver and a corrupted image? In both cases, you might end up in busybox. But with a corrupted image, the boot will often fail earlier — you'll see an error like 'gzip: stdin: unexpected end of file' or 'cpio: premature end of archive'. Lucas: Exactly. If you see that, the image is corrupted. You'll need to boot from a recovery kernel — again, GRUB rescue or a live USB — and then rebuild the initramfs. On Red Hat based systems, that's 'dracut -f' to force rebuild. On Debian, 'update-initramfs -u'. Luna: And if you don't have a live USB handy? I've actually used the kernel from the GRUB command line to chain-load a known good initramfs from a backup copy on a different partition. Lucas: That's clever. Let's talk about prevention, because nobody wants to be in this situation at 2 AM. The biggest thing is: always test kernel and initramfs updates in a non-production environment first. And keep a backup of your working initramfs. On most systems, there's a fallback initramfs image — like 'initramfs-5.x.x-fallback.img' — that the bootloader can use if the primary one fails. Luna: Right, but that fallback only helps if the bootloader entry is configured to try it. You should verify that your GRUB config actually has the fallback line. I've seen setups where the fallback is present on disk but nobody set up the bootloader to use it. Lucas: Absolutely. And another tip: if you're managing servers remotely, make sure your out-of-band management — iDRAC, iLO, IPMI — is configured and tested. Because the worst case is you have a boot failure and no physical access. You need that virtual console to even see the busybox prompt. Luna: All this is good, but I want to circle back to something practical. A listener might be thinking, 'I've never seen this, but I want to be ready.' Is there a safe way to simulate a boot failure to practice? Lucas: Great question. You can corrupt your initramfs safely — make a copy of it first, then use 'dd' to overwrite the first few bytes with zeros. That will force a checksum error on next boot, and you can practice recovery. Or, even safer, set up a VM, snapshot it, and then remove the storage driver from the initramfs manually. That way you can step through the recovery process without risking a real server. Luna: I've done that exact lab. It's eye-opening to work through the dracut rebuild from a rescue environment, especially if you've never done it under pressure. Lucas: And if you're looking for more structured practice, the Red Hat Certified Engineer exam — the RHCE — has a whole section on boot troubleshooting. The techniques we talked about today cover most of what you'd see on the exam and in the real world. Luna: This is one of those topics where a little practice pays off disproportionally. When a server won't boot, you don't have time to Google — you need muscle memory. Lucas: If today's tech conversation gave you something usable, and you want to support the show staying ad-free and focused on this kind of deep-dive content, we're at buy me a coffee dot com slash fexingo. A couple of dollars a month is genuinely what keeps these going. Luna: Yeah, it's a small way to say this kind of content matters. No pressure, but if you've gotten value from the episode, it helps us keep producing it. Lucas: Alright, let's wrap with a quick checklist. If you hit a boot failure: one, check kernel messages for missing driver or corruption. Two, check if the root device appears in /dev. Three, if device is missing, add the driver and rebuild initramfs. Four, if device is there but won't mount, run fsck. Five, if initramfs is corrupt, rebuild from a recovery environment. Luna: And practice it once in a VM. That's my takeaway. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.