Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Build a Custom Linux Server Initramfs
Transcript
- Lucas: If you've ever watched a Linux server boot and wondered what happens between the kernel loading and the root filesystem appearing, you're looking at the initramfs — the initial RAM filesystem. Today, we're going to talk about why you might want to build your own instead of relying on the distribution's default. Luna: I think most sysadmins treat initramfs like magic — it's just there. When does it make sense to roll your own? Lucas: Fair question. The default initramfs that ships with Ubuntu or RHEL is a general-purpose image. It includes a ton of kernel modules for every imaginable storage controller, filesystem, and network driver. That's great for compatibility, but it can be huge — sometimes 50 megabytes or more — and it takes time to decompress and load. Lucas: Speed and size are the big ones. I've seen a deployment of about 200 servers where swapping the default initramfs for a custom one shaved 45 seconds off the boot time per server. Over the fleet, that's significant. But there's also security — a smaller initramfs means a smaller attack surface. And sometimes you need to include a specific driver that the distribution doesn't ship, like a proprietary RAID controller module. Luna: Forty-five seconds? That's huge. How do you even start building one? Lucas: The core idea is simple: an initramfs is just a compressed cpio archive that the kernel unpacks into a tmpfs at boot. Inside it, you need a minimal root filesystem with an init script, BusyBox for basic utilities, and the kernel modules your hardware requires. You can assemble it manually or use tools like mkinitramfs or dracut with a custom configuration. Luna: Let's talk about that manual process first. What does the init script actually do? Lucas: The init script is the first userspace process. It typically loads the necessary kernel modules, mounts the real root filesystem, and then pivots to it using pivot_root. If you're using a modern kernel, you might also use switch_root. The script needs to handle things like detecting the root device, maybe decrypting LUKS, or assembling RAID arrays. It's basically a miniature boot environment. Luna: And BusyBox provides the commands — like mount, ls, modprobe — in a single stripped-down binary. Lucas: Exactly. BusyBox is the standard Swiss Army knife for embedded Linux and initramfs. You can also use toybox, but BusyBox is more common. Once you have your init script and BusyBox, you create the directory structure: /bin, /sbin, /etc, /proc, /sys, /dev, and so on. Copy BusyBox to /bin/busybox, then create symlinks for each command, or just let BusyBox handle it with --install. Luna: What about kernel modules? How do you decide which ones to include? Lucas: That's the tricky part. You need to know your hardware. On the target server, run lsmod to see which modules are loaded. But also check that they're not already built into the kernel — built-in modules don't need to be in the initramfs. Then you copy the.ko files from /lib/modules/$/ into the initramfs under /lib/modules. You'll also need modules.dep and modules.alias, which you can generate with depmod. Luna: And then you package it up with cpio and gzip? Lucas: Right. Once the directory is ready, you do something like: find. | cpio -H newc -o | gzip > /boot/initramfs-custom.img. Then you update your bootloader configuration to point to that image. On a modern system with GRUB, you'd run update-grub after placing the image in /boot. Luna: Let me guess — it almost never works on the first try. Lucas: Almost never. The first time I did this, I forgot to include the storage controller module, and the kernel panicked because it couldn't find the root device. The trick is to test with a console output — append console=ttyS0 or console=tty0 to the kernel command line so you can see where it's failing. You can also boot with the 'break' parameter in some initramfs generators to drop into a shell before pivot_root. Luna: So debugging is a lot like kernel development — lots of rebooting. Lucas: Exactly. But once it works, the payoff is real. I had a case where a server with a rare RAID controller needed a driver that wasn't in the default initramfs. Building a custom one with just that driver and BusyBox made the boot process reliable and fast. Luna: What about using tools like dracut? That's supposed to make it easier, right? Lucas: Dracut is a huge step up from manual assembly. It's modular and scriptable. You can create a custom dracut configuration file in /etc/dracut.conf.d/ that lists only the modules you need. For example, you can add 'add_drivers+=" my_raid_mod "' and 'omit_drivers+=" everything_else "'. Then run dracut -f /boot/initramfs-custom.img and it generates a minimal image. Luna: Does it handle things like LVM or LUKS automatically? Lucas: It has modules for that. You can include or exclude them. One thing to watch out for: dracut by default includes a lot of stuff. If you really want a minimal image, you need to explicitly list what you want and use the --hostonly option, which tails the image to the current hardware. But be careful — if you move that image to a different server, it might not boot. Luna: That's a good point. So custom initramfs is best for homogeneous fleets or single-purpose servers. Lucas: Exactly. For a fleet of identical servers, a custom initramfs makes sense. For a heterogeneous environment, the generic one is safer. But even then, you can create a few variants — one for storage-heavy nodes, one for compute nodes, et cetera. Luna: Let's talk about security. You mentioned reducing attack surface. How does a smaller initramfs help there? Lucas: The initramfs runs as root before the real root filesystem is mounted. If an attacker can modify the initramfs image on disk, they can inject malicious code that runs at boot. A smaller image with fewer binaries means fewer potential vulnerabilities. Also, by removing unnecessary kernel modules, you prevent kernel exploits that rely on those modules being loaded. It's defense in depth. Luna: So it's like stripping down a container image — only ship what you need. Lucas: Exactly the same philosophy. In fact, some people even use BusyBox-based initramfs for embedded systems or as the root filesystem itself in very small environments. But for servers, it's mostly about boot speed and reliability. Luna: I want to circle back to the 45-seconds example. How big was that initramfs before and after? Lucas: The default was about 65 megabytes. After stripping it down to just the required storage and network modules plus BusyBox, it was under 8 megabytes. That's roughly an 88 percent reduction. And because it's smaller, it decompresses faster, and the kernel spends less time loading modules. The 45-second savings came from both the smaller size and the fact that the init script didn't have to probe for hardware that wasn't there. Luna: That's a huge improvement. I'm surprised more sysadmins don't do this. Lucas: I think the barrier is the fear of breaking boot. But with a good testing process — start with a VM, use serial console, keep a fallback initramfs in GRUB — it's manageable. And honestly, once you understand the initramfs, you have a much deeper understanding of how Linux boots. Luna: Speaking of that fallback — do you recommend keeping the default initramfs as a backup entry in GRUB? Lucas: Absolutely. When you update your bootloader, add a menu entry that points to the original initramfs. That way, if your custom one fails, you can boot the old one and fix the issue. It's a safety net that costs nothing. Luna: That seems like a no-brainer. So what's the most common mistake people make when building a custom initramfs? Lucas: Forgetting the root delay. Some storage devices — especially on virtual machines or SANs — take a few seconds to become available. If your init script tries to mount the root device immediately, it fails. You need a loop that waits for the device to appear, maybe with a timeout. Another common mistake is not including the kernel module for the root filesystem type itself — for instance, if you use ext4 but only include ext3, you'll get a panic. Luna: So you need to know not just your hardware but also your filesystem driver. Lucas: Right. And if you use LVM or mdadm RAID, you need those tools in the initramfs too. That's where dracut's modules really shine — they handle those complexities. But if you're doing it manually, you need to include lvm or mdadm and their dependencies. Luna: Let's talk about the init script structure for a second. Do you have a template you usually start from? Lucas: Yeah, I keep a simple one. It starts by mounting proc, sys, and devtmpfs. Then it loads the storage module — modprobe my_raid_mod. Then it waits for the root device to appear, mount it to /newroot, and runs exec switch_root /newroot /sbin/init. That's the core. Some people also add a shell fallback if something fails, which is great for debugging. Luna: And you can test that script by booting with 'rd.break' in some distributions, right? Lucas: Yes, if you're using dracut, you can add 'rd.break' to the kernel command line, and it drops you into a shell before pivot_root. That lets you manually run your script and see where it fails. For manual initramfs, you can add a simple 'exec /bin/sh' at the end of the init script to get a shell if something goes wrong. Luna: That's a nice debugging trick. I also like the idea of using a VM to test — it's a lot faster than rebooting a physical server. Lucas: Exactly. I always test new initramfs images in a VM first with the same hardware configuration. Tools like libvirt or VirtualBox make it easy to attach a custom kernel and initramfs. Once it boots cleanly in the VM, I move to a test server, then to production. It's a standard change-management pipeline. Luna: One thing that comes to mind is that not all distributions use the same init system. Some use systemd in the initramfs. How does that change things? Lucas: Modern distributions like Fedora and Ubuntu use systemd in the initramfs — it's called systemd-boot. That means the init script is replaced by systemd units. You can still customize it, but it's more complex. In those cases, dracut is the way to go because it generates the proper systemd-based initramfs automatically. If you try to hand-roll a systemd initramfs, you're in for a world of hurt. Luna: So for most sysadmins, the practical takeaway is: use dracut with --hostonly and a custom config file. Lucas: For speed and sanity, yes. But it's worth understanding the manual process at least once, because it demystifies the boot sequence. And if you ever need to debug a boot failure on a system where dracut isn't available — like an embedded device — you'll be glad you know how it works. Luna: That makes sense. Let's pivot a bit — you mentioned security earlier. Are there any specific security hardening steps you'd recommend for an initramfs? Lucas: A few. First, sign your initramfs image if your kernel supports it. On UEFI systems, you can use the kernel's CONFIG_EFI_STUB and sign the kernel and initramfs together. That prevents tampering at boot. Second, strip all binaries to reduce size and remove debugging symbols. Third, ensure the initramfs doesn't include any sensitive data like passwords or keys — if you need decryption, use a keyfile stored on a hardware token or prompt interactively. Luna: What about compressing the initramfs with something like LZ4 instead of gzip for faster decompression? Lucas: Great idea. LZ4 decompresses much faster than gzip, at the cost of a slightly larger compressed size. On modern servers with fast CPUs, the decompression time matters more than the image size. You can specify the compression method in dracut with --compress=lz4, or when manually running gzip, just replace it with lz4. Just make sure your kernel has LZ4 support compiled in. Luna: And you can also use zstd, which is a good middle ground. Lucas: Exactly. The kernel has supported zstd since version 5.9. It's becoming the new standard because it offers compression ratios close to gzip with speeds close to LZ4. On Ubuntu 22.04 and newer, the default initramfs uses zstd. Luna: So the landscape is shifting toward better compression algorithms. But the core concept remains the same: a small, fast, secure boot environment. Lucas: Right. And no matter what algorithm you use, the process of building your own initramfs gives you a level of control that you just don't get with the distribution default. It's one of those sysadmin skills that separates the people who understand the boot process from those who just accept it. Luna: That's a good place to land. If today's conversation gave you something practical — maybe you're thinking about slashing boot times on your own fleet — that's the whole point of this show. We deliberately don't run ads on these episodes. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. No pressure, just a way to keep it ad-free. Lucas: Exactly. And it's the kind of listener support that lets us dive deep into topics like initramfs without worrying about sponsors. So if it's useful, that's the place. Luna: Alright, back to the tech. One more topic I want to touch on: how do you handle initramfs updates when the kernel gets updated? Lucas: That's a critical point. If you're using a custom initramfs, you need to rebuild it whenever the kernel updates, because the modules are tied to the kernel version. You can automate this with a hook script in /etc/kernel/postinst.d/ that runs dracut or your manual build script. On Debian, you can use kernel image postinst hooks. Otherwise, you'll boot into a new kernel with an old initramfs that might not have the right modules. Luna: So it's not a set-and-forget kind of thing. You need to have a process. Lucas: Exactly. And it's another reason to use dracut — it integrates with the kernel package manager. On Fedora, if you install a new kernel, dracut automatically rebuilds the initramfs. On Ubuntu with mkinitramfs, you can configure it similarly. But if you're doing it manually, you have to remember. Luna: Alright, so to sum up: custom initramfs can save boot time, improve security, and handle special hardware. But it requires careful planning, testing, and automation. Lucas: That's it. And once you've done it a few times, it's not as daunting as it sounds. The key is to start small — maybe just strip down the default initramfs by removing a few modules, then test. You'll quickly see the benefits. Luna: And keep a fallback. Always keep a fallback. Lucas: Always. Thanks for joining us on Linux Server Admin. We'll see you next time.