Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your Linux Server Needs Immutable Root
Transcript
- Lucas: So you SSH into a server you haven't touched in six months. You run a quick 'df -h' and your root partition is at 87 percent. You check what's in /usr/local and find a Python script from a contractor who left two years ago, some random.deb file half-unpacked, and a cron job that emails a report to an address that bounces. Luna: The server is running fine. But it's a mess. Lucas: Right. It works, but you have no idea what's actually supposed to be there. That drift—the slow accumulation of unmanaged changes—is one of the biggest sources of production incidents. And there's a way to kill it at the architectural level: make your root filesystem immutable. Luna: Immutable root—so the entire OS partition is read-only at runtime? That sounds extreme. Lucas: It sounds aggressive, but it's become a practical mainstream approach in the last few years. Think of it this way: instead of treating your server like a pet you hand-edit, you treat it like an appliance. The OS image is fixed. You deploy updates as a whole new image, atomically. If something tries to write to /usr or /etc at runtime, it fails. Luna: So no more accidental 'sudo rm -rf /var/log' or an installer script that splats files all over the place. Lucas: Exactly. Two major Linux distributions have been pushing this model: Fedora Silverblue—which is technically aimed at workstations but works fine on servers—and openSUSE MicroOS, which is built specifically for container hosts and edge devices. Both use a read-only root by default. Luna: But how do you install packages? If the root is read-only, you can't just 'apt install' or 'dnf install'. Lucas: That's the key design decision. Silverblue uses something called rpm-ostree. Instead of a live package manager that modifies /usr directly, you layer packages on top of a base image. When you run 'rpm-ostree install htop', it downloads the package, creates a new filesystem tree combining the base image plus that layer, and then on next boot, you switch into that new deployment. It's atomic—if the update fails, you just reboot into the previous deployment. Luna: So no partial upgrades. No half-installed packages. That would have saved me from a broken libssl dependency last year. Lucas: Right. MicroOS takes a slightly different approach—it uses transactional-update, which applies package changes as a snapshot. You run 'transactional-update pkg install htop', it creates a new snapshot with the package, and you reboot into it. Both models guarantee that the running system is never mutated mid-operation. Luna: And for applications that need to write data? Like databases, logs, user uploads? Lucas: Those go on separate, writable partitions—usually /var and /srv. The immutability applies to the OS itself: /usr, /etc, /bin, /lib. Your data lives elsewhere. MicroOS even has a built-in mechanism for /etc called 'etc-overlay' that merges config changes persistently, but the core OS stays sealed. Luna: This feels like the logical endpoint of the container movement. You run your apps in containers, which are already immutable images. Why not make the host itself behave the same way? Lucas: That's exactly the thinking. In fact, both Silverblue and MicroOS were designed with container workloads in mind. You install Docker or Podman as a layered package, and then every container runs on an immutable base that you know hasn't drifted. It creates a clean boundary: the host is fixed, the data is persistent, and the containers are ephemeral and reproducible. Luna: Let's talk security. If root is read-only, can malware still persist? Say something gets in via a web app vulnerability. Lucas: It's much harder. Traditional Linux malware often writes to /usr/bin or /lib to establish persistence. On an immutable system, those paths are read-only. It can't install a cron job if /etc/cron.d is immutable. It can't drop a kernel module into /lib/modules. The attacker would need to find a writable location—like /var—which is a much smaller and more obvious footprint. Plus, since updates replace the entire root, any hidden modifications are wiped on the next update. Luna: I've heard the argument that immutability makes troubleshooting harder—you can't just edit a config file and restart a service. You have to rebuild and redeploy. Lucas: That's a real trade-off. If you're debugging a live issue, you can't SSH in and change nginx.conf on the fly. You have to either overlay a temporary config or use a tool like 'rpm-ostree usroverlay' which creates a temporary writable overlay for debugging—but that's not persistent. The discipline is: every change goes through version control and gets deployed as a new image. That's painful if you're used to cowboy sysadmin, but it's exactly the discipline that prevents drift in the first place. Luna: So it shifts the sysadmin role from 'fix it live' to 'update the deployment pipeline'. That's a different skillset. Lucas: Exactly. And for organizations that already use infrastructure as code with Ansible or Terraform, this is a natural fit. Your playbooks define the entire OS state, the immutable image bakes that state in, and you never have to worry about one server being different from another because someone ran a manual command three years ago. Luna: Quick honest thing—this kind of deep-dive content is exactly why a handful of listeners chip in monthly through Buy Me A Coffee dot com slash fexingo. That's literally what funds making this many shows. We don't run ads, and it stays listener-supported. Lucas: Yeah, and it's a small group already keeping it going. If today's tech conversation gave you something usable, that link is there. No pressure—just part of how we keep the server lights on. Luna: Alright—back to immutable root. So let's talk about real-world gotchas. One I've hit: what about kernel modules? If you need a proprietary driver, like NVIDIA on a GPU server, can you even layer that on Silverblue? Lucas: Yes, but it's not as seamless as a traditional distro. Silverblue allows layering kernel modules via rpm-ostree, but you have to rebuild the deployment each time the kernel updates. Some people prefer to use the container model for GPU workloads—run the driver inside the container, not on the host. MicroOS handles this slightly better with its transactional-update system, which can trigger a rebuild automatically. Luna: So there's still rough edges. What about monitoring agents? A lot of them want to drop scripts into /etc or /usr/lib. Lucas: That's a real pain point. Many monitoring agents assume a mutable filesystem. You either have to install them as layered packages, which requires rebuilding the image, or run them inside a container with access to the host via a socket or bind mount. The ecosystem is still catching up. But the trend is clear—more vendors are starting to support immutable hosts. Datadog, for example, now offers an agent container image that runs on Podman or Docker. Luna: So for a sysadmin evaluating this today, what's the best use case? Production web servers? Database hosts? Lucas: I'd say the strongest case is for container orchestration nodes—Kubernetes worker nodes, Docker Swarm hosts, or dedicated CI/CD runners. Those machines don't need a lot of os level customization. They just need Docker, kubelet, and maybe a few tools. Immutable root guarantees that the node is identical to the image you tested. For database servers, it's more complex because you often need kernel tuning, custom sysctl settings, and sometimes specific filesystem features. But even there, you can bake those into the image and accept that changes require a reboot. Luna: What about the dreaded 'but we've always done it this way' resistance from the ops team? Lucas: It's real. The biggest cultural shift is moving from a mutable, forgiving environment to one that enforces discipline. But I've seen teams adopt it gradually: start with one stateless service, like a load balancer or a reverse proxy. Get comfortable with the workflow. Then expand. The key is to have a good rollback story—and immutable systems have the best rollback story. You boot into the previous deployment, and you're back to a known good state in under a minute. Luna: That's the killer feature, honestly. No more 'I accidentally ran apt upgrade and now the app is broken, let me spend an hour restoring from backup.' Just reboot and you're back. Lucas: Right. And that reliability extends to updates. When you push a new image to a fleet, you can do canary deployments: update one node, verify it works, then roll out to the rest. If something goes wrong, you roll back the image—not by reverting a package, but by flipping a bootloader entry. It's much cleaner. Luna: Let's talk about the tooling ecosystem. Is there a standard way to build these images, or are we back to 'each distro does its own thing'? Lucas: It's fragmented. For Silverblue, you use 'rpm-ostree compose tree' with a JSON manifest, or you can use tools like 'Image Builder' from Red Hat. For MicroOS, you use Kiwi or the Open Build Service. But there's also a growing trend of using generic image builders like Packer or mkosi to create immutable images for any distro, not just the ones that ship read-only by default. You can take a standard Debian, make root read-only, and use overlay filesystems for /etc. Luna: So you could roll your own immutable Ubuntu or Debian? That sounds like a project. Lucas: It is a project, but it's doable. There's even a tool called 'immudeb' that makes Debian immutable with a few commands. The core idea is: mount /usr read-only, use tmpfs for /run, and overlay a writable layer on /etc for config changes that you commit back to a git repo. It's not as polished as Silverblue, but it gives you the same guarantees. Luna: What about boot speed? Atomic updates often involve more layers—does that slow down startup? Lucas: Surprisingly no. Silverblue boots a bit slower than a standard Fedora because it uses an initramfs with ostree, but we're talking a few seconds. MicroOS is actually very fast because it uses a minimal image. For server use, boot time is rarely the bottleneck—uptime is measured in months or years. What matters is that when you do reboot, it's reliable. Luna: I want to circle back to a point you made earlier: the security angle. If root is immutable, does that also protect against ransomware that encrypts the OS? Lucas: Yes and no. Ransomware that targets the OS can't encrypt /usr because it's read-only. But it can still encrypt /var, /home, and any mounted storage. The benefit is that the OS itself remains bootable, and you can still run recovery tools. And since the OS image is immutable, you can simply reboot into a clean snapshot and restore your data from backup. It doesn't prevent the attack, but it dramatically reduces recovery time. Luna: That's a good note for disaster recovery planning. Have you seen any production outages caused by immutability itself? Like an application that needed to write to /usr and failed silently? Lucas: Yes, that's a common pitfall. Developers sometimes hardcode log paths to /usr/local/var or something similar. On an immutable system, that write fails silently unless you've configured the app to crash on write errors. The fix is to enforce that all writable paths are under /var or /srv, and to test your app on an immutable environment in staging. But it's a one-time pain—once you fix the paths, you never see the issue again. Luna: So the message is: immutability forces good behavior. If your app can't run on an immutable host, that's a bug in your app, not in the OS. Lucas: Exactly. And that mindset—treating the OS as a fixed layer—is becoming the default in cloud-native environments. Container images are immutable. infrastructure as code is immutable. It makes sense that the host OS follows the same pattern. I wouldn't be surprised if by 2030, the majority of production Linux servers are running some form of immutable root. Luna: It's one of those ideas that sounds radical until you try it, and then you wonder why you didn't do it sooner. I'm definitely going to spin up a MicroOS VM this weekend. Lucas: Do it. And if you hit a snag, you know where to find us. That's it for this episode—go make your servers a little more boring.