Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Disk Encryption with LUKS and Clevis
Transcript
- Lucas: Alright, let's talk about something that keeps a lot of sysadmins up at night — disk encryption on a Linux server. Specifically, how do you encrypt a disk so that it's secure at rest, but also unlocks automatically after a reboot without someone typing a password? Luna: Yeah, that tension is real. You want the security of full-disk encryption, but if your server reboots in the middle of the night, you don't want to have to drive to the data center. Lucas: Exactly. And the solution that's been around for years is LUKS — Linux Unified Key Setup. It's the standard for block-device encryption on Linux. You set up a LUKS partition, provide a passphrase at boot, and the system decrypts the disk. But that manual step is a problem for headless servers. Luna: Right, so what's the modern way to automate that? I've heard about Clevis and Tang — network-bound disk encryption. Lucas: That's exactly the combo I want to drill into. Clevis is a framework for automated decryption. Tang is a server that provides cryptographic keys over the network. Together, they let you bind a LUKS volume to a network service — so when the server boots, it reaches out to the Tang server, gets the key, and unlocks the disk automatically. Luna: So it's like a network-based key escrow, but without storing the key on the client. How secure is that? Lucas: Well, it's a trade-off. The key is never stored on the encrypted machine — it's served by the Tang server on demand. That means if someone steals the physical server, they can't decrypt it without access to the Tang server. But if the Tang server is compromised or unreachable, the server won't boot. Luna: That's the classic availability vs. security trade-off. For a data center with a reliable network, it's probably fine. But if you have a remote server with flaky connectivity, that could be a problem. Lucas: Exactly. And you can layer fallbacks. You can set up a LUKS slot with a manual passphrase as a backup, so if the Tang server is down, you can still SSH in — if you have an out-of-band management console — and type the password. That's actually the recommended setup. Luna: So let's get practical. If I have a fresh server running a recent Ubuntu or RHEL, what's the workflow to set this up? Lucas: Sure. First, you need a Tang server. That can be a separate machine or even a container. You install the tang package, which exposes a simple HTTP endpoint. Then on the client, you install clevis and clevis-luks. Let's say you have a new disk at /dev/sdb. You create a LUKS partition with cryptsetup luksFormat, set a passphrase. Then you use clevis luks bind -d /dev/sdb -s 1 tang '{"url":"http://tang-server:7500"}' — that binds LUKS slot 1 to the Tang server. Luna: And the Tang server needs to be reachable at boot time. So the network has to be up before the encryption is unlocked. That usually means configuring network in the initramfs. Lucas: Right. On most distros, systemd handles that automatically if you have network configuration in the initramfs. You might need to rebuild the initramfs after installing clevis. But once that's done, at boot, the initramfs runs clevis, which reaches out to the Tang server, gets the key, and unlocks the disk. It's seamless. Luna: What about the LUKS header itself? If that gets corrupted, you lose everything. Is there a way to back that up? Lucas: Great question. Yes. The LUKS header contains the encrypted key slots and metadata. You should back it up immediately after setup. Use cryptsetup luksHeaderBackup /dev/sdb --header backup file luks header backup.img. Store that file securely — it's sensitive because it contains the encrypted keys. If the header is damaged, you can restore it with the inverse command. Luna: How does this compare to using tpm based unlocking? Like with systemd-cryptenroll on a system that has a TPM chip? Lucas: That's another good option. tpm based unlocking ties the key to the specific hardware. That's more secure if you're worried about network-based attacks, because the key never leaves the machine. But it doesn't help if you need to move a disk to another server — the TPM is tied to the motherboard. Clevis and Tang are more flexible in a clustered environment because the key is network-based. Luna: So Tang is almost like a distributed key management service. You could have multiple Tang servers for redundancy, right? Lucas: Exactly. You can bind multiple Tang servers to different LUKS slots, or you can use a technique called Shamir secret sharing with Clevis. Clevis supports a 'sss' pin that lets you split the key across multiple Tang servers — so you need, say, 3 out of 5 to unlock. That's advanced, but it gives you high availability without a single point of failure. Luna: That's clever. But let's talk about the security model. If someone compromises the Tang server, they can decrypt any disk bound to it. So you'd want to lock down that Tang server tightly — no unnecessary services, maybe even run it in a container with minimal access. Lucas: Absolutely. The Tang server should be isolated. It doesn't need to be on the same network as the clients — it just needs to be reachable. You can put it behind a firewall, restrict access by IP, and use HTTPS with certificates. Tang also supports key rotation, which is important for long-term security. Luna: Key rotation — how does that work with LUKS? If you rotate the Tang key, do you have to rebind all the disks? Lucas: You don't have to rebind if you use a feature called 'advertising key rotation'. Tang servers can advertise multiple keys. When a client binds, it stores the current key identifier. If the server rotates keys, the old key is still available for decryption until you rebind the client to the new key. So you can rotate without downtime. Then you rebind at your convenience. Luna: That's well thought out. For listeners who are thinking of implementing this, what's the biggest gotcha you've seen? Lucas: The biggest gotcha is the initramfs. If your network isn't configured in the initramfs, the Clevis plugin can't reach the Tang server. You'll end up at a busybox shell asking for a password. So test your initramfs — boot once with a console attached and make sure the network comes up before the unlock prompt. Also, make sure your Tang server uses a predictable hostname or IP — DNS might not be available at that stage. Luna: So static IP or a local hosts file in the initramfs. Good tip. Now, one more thing — what about performance overhead? Does LUKS add noticeable latency? Lucas: Modern CPUs have aes ni instructions, so encryption is hardware-accelerated. In practice, the overhead is typically under 5 percent for sequential reads and writes. For most server workloads, it's negligible. The bigger performance consideration is that LUKS operates at the block layer, so it's transparent to the filesystem — you can use any filesystem on top. Luna: Alright, that's a solid overview. Let's take a quick step back. You know, it's impressive how much tooling has evolved. A few years ago, automated LUKS unlocking was a custom script hack. Now we have Clevis and Tang as standard packages. Lucas: Yeah, and it's part of a larger trend — the Linux ecosystem getting serious about security without sacrificing automation. And speaking of that, I want to mention something briefly. This show is possible because a small number of listeners chip in monthly on Buy Me a Coffee at buy me a coffee dot com slash fexingo. That support keeps us ad-free and focused on deep dives like this. Luna: Yeah, it's a simple model — no sponsors, no tracking. Just people who find value in the content and want to keep it going. We're grateful for that. Lucas: Exactly. So, back to encryption — I want to cover one more scenario: what if you need to encrypt the root filesystem on an already-running server? That's a trickier migration. Luna: Right, because you can't encrypt a live root filesystem. You'd need to boot from a rescue disk, shrink the partition, create a LUKS container, copy the data, and reconfigure the bootloader. Lucas: And that's a whole episode in itself. But for new installations, it's straightforward. Most installers — like the Ubuntu Server installer or RHEL's Anaconda — have an option to set up LUKS with Clevis during installation. You just point it at your Tang server, and it configures everything automatically. Luna: That's the ideal path. But for existing servers, you might want to consider using a separate encrypted data disk first — encrypt /var or /home — before tackling root. Lucas: Good advice. Start small. Encrypt a test disk, get comfortable with the tools, then plan the root encryption migration during a maintenance window. And always have a backup of your LUKS header. Luna: And test the automatic unlock in a staging environment. Nothing worse than a production server that won't boot because the Tang server is down. Lucas: Absolutely. So to wrap up: LUKS gives you strong encryption, Clevis and Tang give you automation, and a backup passphrase gives you a safety net. It's a setup that balances security and convenience for most server environments. Luna: And if you want to dive deeper into initramfs or key rotation, we can cover that in a future episode. For now, go encrypt something. Lucas: I like that. Until next time.