Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Automated Patching with Unattended Upgrades
Transcript
- Lucas: Let's talk about something every sysadmin does — or should do — but rarely gets right: automated security patching. Not the quarterly big-bang upgrades, but the steady stream of point fixes for OpenSSL, systemd, the kernel. You know, the reason your SSH session says 'reboot required' every three weeks. Luna: Right. And most people I know either set-and-forget unattended-upgrades and hope for the best, or they disable it entirely because of one bad experience with a broken package. Lucas: Exactly. So today we're going to drill into unattended-upgrades on Debian and Ubuntu — the standard tool — and cover a setup that's actually production-safe. Not the default 'apply everything as soon as it lands' config, but something you can run on a web server or a database host without waking up to a pager. Luna: Good. So where do we start? Lucas: Installation is trivial — just 'apt install unattended-upgrades'. But the real work is in the config file at /etc/apt/apt.conf.d/50unattended-upgrades. That file controls which repos you pull from, what packages you blacklist, and how aggressive the update schedule is. Luna: And the default includes 'stable' updates and security updates. But I've seen people accidentally include 'proposed' or 'backports' and suddenly they're running a bleeding-edge kernel with no testing. Lucas: Exactly. So first rule: explicitly whitelist only the origins you want. In that config, you uncomment the lines for 'origin=Debian,archive=stable' and 'origin=Debian,archive=stable-updates' and 'origin=Debian,archive=stable-security'. For Ubuntu it's 'origin=Ubuntu,archive=focal-security' and so on. Leave everything else commented. Luna: What about the kernel? A lot of people are nervous about auto-updating the kernel. Lucas: Smart nervous. Kernel updates can change behavior, break out-of-tree modules, or require a reboot. I strongly recommend blacklisting kernel packages. Add a line like 'Unattended-Upgrade::Package-Blacklist { "linux-image-."; }'. Then you handle kernel upgrades manually, on your own schedule, with staged rollouts. Luna: That feels like a good middle ground. Security fixes for user-space, manual control for the kernel. Lucas: Right. And you can extend that blacklist to any package you want to pin. Database engines, container runtimes, anything where a minor version change could cause a compatibility issue. The syntax is regex, so you can be as broad or as narrow as you need. Luna: Okay, so we've got the config. How do we trigger it? Lucas: The unattended-upgrades package installs a systemd timer by default: apt-daily.timer and apt daily upgrade.timer. The first updates package lists, the second actually applies upgrades. They run daily — the default is staggered around 6 AM with a random delay to avoid hammering mirrors. Luna: And you can adjust those timers if your maintenance window is different? Say, 2 AM on a Tuesday? Lucas: Absolutely. Use 'systemctl edit apt daily upgrade.timer' and override the OnCalendar value. I usually set mine to 3 AM Sunday. That way any reboot it triggers happens during low-traffic hours — assuming you've configured automatic reboots, which we'll get to. Luna: Right, automatic reboot. That's the part that scares most people. Lucas: It should scare you, because the default unattended-upgrades config does NOT reboot. And that's correct — if a security fix doesn't require a reboot, you don't want one. But sometimes a kernel or libc update requires a reboot to take effect. The package leaves a flag file at /var/run/reboot-required. Luna: So you can use that flag to trigger a reboot through a separate mechanism, not from unattended-upgrades itself. Lucas: Exactly. I use a small script that runs via a systemd timer an hour after the upgrade timer. The script checks for that flag file. If it exists, it logs a message and then runs 'shutdown -r +5' — a five-minute delay so you can cancel if you're watching. You can also use 'needrestart' to check which services need a restart, and restart only those instead of the whole box. Luna: I like that. More surgical than a full reboot. Lucas: Yeah. And for services like nginx or PostgreSQL, a restart is quick and doesn't affect other workloads. But if the kernel needs updating, you still need a reboot. The trick is making that reboot predictable. I set a window — say, Sunday 4 AM — and only allow the reboot script to act during that window. If the flag appears on Tuesday, it waits until Sunday. Luna: That avoids the dreaded '3 AM Wednesday reboot that kills the ETL job' scenario. Lucas: Precisely. Now, the other big piece is monitoring. Unattended-upgrades can mail you a summary, but that's old-school. I prefer to log to syslog and then forward those logs to a central aggregator. You can also use the 'Unattended-Upgrade::Mail' directive to send a single email per server per run. Luna: And what should you check in those logs? Just 'success' or 'failure'? Lucas: More than that. Look for the 'Packages that were upgraded' line. That tells you what changed. Also check for 'Packages with held back' — those are packages that apt chose not to upgrade, often because of phasing or dependency issues. If you see a pattern of held backs, you might need to investigate manually. Luna: Phased updates — that's a Ubuntu thing, right? Lucas: Yes. Starting with Ubuntu 20.04, apt uses phased updates by default. It rolls out updates to a gradually increasing percentage of users over 7-14 days. So your server might see an update on day 5 when another server on day 7 hasn't yet. Unattended-upgrades respects phasing, so you don't need to worry about getting a bad update early. Luna: That's actually a great safety net. But if you want to force all servers to get the update immediately — say for a critical zero-day — you can disable phasing in apt.conf. Lucas: You can, but I'd recommend against it unless you absolutely need to. Phasing is there for a reason. Instead, you can manually run 'apt upgrade' on one or two canary servers first, then let the phased rollout reach the rest. Luna: Good practice. So let's summarize the sane unattended-upgrades config: whitelist only stable and security repos, blacklist kernel packages, set a reasonable schedule, and use a controlled reboot mechanism. Anything else? Lucas: One more thing: configure 'Unattended-Upgrade::remove unused dependencies' to true. That cleans up old dependencies that are no longer needed. Prevents your boot partition from filling up with old kernel headers and such. Luna: That's a nice touch. I've definitely had a server run out of /boot because of accumulated kernels. Lucas: Yeah, that's a classic. And if you're using a staging repository — like a local apt mirror that only gets updates after you've tested them — you can point unattended-upgrades to that mirror. It's trivial to override the 'APT::Sources' line in the config. Luna: That's the gold standard for enterprise: a staging repo that delays patches by a week. Lucas: Absolutely. But even without that, a well-tuned unattended-upgrades config is leagues better than the all-or-nothing approach most people use. Luna: You know, I find that even a small amount of listener support — a couple of dollars a month — genuinely makes a difference in keeping shows like this ad-free and focused on real content. If these episodes have saved you an hour of debugging or helped you build a better config, buy me a coffee dot com slash fexingo is where you can chip in. Lucas: It's true. And it means we can keep diving into these specific, practical topics without any sponsor breaks. So if you've gotten value, toss something in. We appreciate it. Luna: Back to the topic: one last thing I want to touch on is testing. How do you verify unattended-upgrades actually works without waiting for a real security update? Lucas: Great question. You can simulate a run with 'sudo unattended-upgrades --dry-run --debug'. That shows you exactly what packages would be upgraded, without touching the system. Do that after every config change. Also, you can force a check by running 'sudo apt update' and then 'sudo unattended-upgrades'. Luna: And check the log at /var/log/unattended-upgrades/unattended-upgrades.log to confirm everything ran cleanly. Lucas: Exactly. So the takeaway: unattended-upgrades is a powerful tool, but only if you configure it deliberately. Blacklist what you need, set a controlled reboot window, monitor the logs, and test your config. Do that, and your servers stay patched without manual toil. Luna: And without waking you up at 3 AM. Lucas: That's the goal.