Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your Linux Server Needs a Dedicated Patch Automation Schedule
Transcript
- Lucas: So a friend of mine — runs operations at a mid-sized e-commerce shop, maybe fifty Linux servers — calls me last month. Friday, 3 PM. They patched the kernel on their main database server manually, and the system wouldn't come back after reboot. Luna: Friday afternoon. Oh, I can already guess where this is going. Lucas: Exactly. They ended up with a twelve-hour outage over the weekend because the patch introduced a conflict with their custom storage driver. No rollback plan, no pre-patch snapshot, and — get this — it was the same senior engineer who'd patched manually for years. Luna: So the lesson is: stop patching by hand? Or at least stop doing it on Friday? Lucas: Both, really. But the deeper issue is that most sysadmins still treat patching as an ad-hoc task. You SSH in, run apt upgrade or yum update, and hope it works. And when it breaks, you learn the hard way that you should've automated the whole cycle — scheduling, staging, testing, rollback. Luna: Let's talk about what a proper patch automation schedule looks like. What's the first component? Lucas: The first thing is defining your patch window. For production servers, that's typically a fixed time — say, 2 AM on a Tuesday or Wednesday — when user traffic is lowest. But you don't patch everything at once. You split your fleet into groups: canary, then staggered batches. Luna: Canary being one or two servers that get the update first, right? If they survive, you roll to the rest. Lucas: Exactly. And the canary should be representative of your workload — not just a random idle box. My friend's team now runs a canary that serves real traffic for five minutes post-patch before they proceed. If latency spikes or errors appear, they halt the rollout. Luna: What about the actual tooling? A lot of people use unattended-upgrades on Debian or yum-cron on RHEL. Those are automated — but they apply updates as soon as they're available. Lucas: Right, and that's the problem. Unattended-upgrades is great for security patches on non-critical systems, but for production, you want control over timing. You don't want a kernel update hitting your web tier at 3 PM on a Tuesday because the repo just synced. So the schedule should be cron-driven: a script that checks for updates, downloads them, but only installs during your window. Luna: And you need to handle the reboot. If it's a kernel update, most servers require a reboot. Do you automate that too? Lucas: You can, but you need a way to verify that the server comes back cleanly. I've seen setups where after reboot, the server pings a health endpoint. If it doesn't respond within five minutes, the automation rolls back to the previous kernel using GRUB's saved entry. That kind of safety net is non-negotiable. Luna: Rolling back automatically — that's smart. How common is that in practice? Lucas: Less common than it should be. Most teams I talk to either don't have rollback automation, or they rely on full system images. But with modern package managers, you can keep the last three kernels and boot into the previous one via GRUB_DEFAULT=saved. It takes maybe thirty minutes to script. Luna: So the schedule itself — what does it look like day-to-day? Walk me through an example. Lucas: Sure. Let's say you have thirty servers. On patch day — let's call it Tuesday — your cron job at 2 AM runs on the canary node. It applies all updates, triggers a reboot if needed, then runs a smoke test script that checks HTTP 200, database connectivity, and a synthetic transaction. If all green, it moves to batch one — say, ten servers — then waits ten minutes, then batch two, and so on. Luna: And if batch one fails? Lucas: Then the whole rollout stops. The automation sends an alert to the on-call engineer, and the remaining batches are held. The engineer can manually inspect the failure, decide whether to roll back batch one, and adjust. That's way better than discovering the issue at 9 AM when users start complaining. Luna: I like the staged approach. But what about dependencies — like when a package update requires a newer library that other applications depend on? That's where silent breakage happens. Lucas: That's a big one. Package managers resolve dependencies at install time, but they don't test your application against the new library. So a critical part of the smoke test is actually running your app through its paces. For a web app, that might mean hitting an API endpoint that exercises a specific feature. For a database, running a query that uses a stored procedure. Luna: So the smoke test has to be tailored to your stack. It's not just 'server is up'. Lucas: Right. 'Server is up' is table stakes. You need to verify functionality. My friend's team now has a test that places a test order, checks it appears in the database, and confirms the confirmation email was sent. If that works, they're confident the patch didn't break anything critical. Luna: What about non-kernel updates? Like OpenSSL or libcurl — those can have security implications but don't need a reboot. Lucas: Those are actually trickier in some ways because there's no reboot to signal a change. You need to restart the services that use those libraries — web server, database, whatever. So your automation should include a service restart step after library updates. And again, verify the service comes back healthy. Luna: I'm thinking about the human side too. If you automate all this, do sysadmins become complacent? Like, they stop thinking about what's in the updates? Lucas: That's a real concern. Automation shouldn't replace awareness. The best setups still generate a summary report after each patch window — what was updated, what was skipped, any errors. The sysadmin reviews it the next morning. That keeps them in the loop without the drudgery of manual patching. Luna: So the goal is to reduce toil, not eliminate thinking. Lucas: Exactly. And there's another angle: compliance. If you're audited for pci dss or SOC2, you need to prove that patches are applied within a certain window. Automated logs with timestamps make that audit trail trivial. Manual patching often leaves gaps — 'I think I updated that server last week'. Luna: Alright, let's talk about the actual tools. You mentioned unattended-upgrades and yum-cron, but what about Ansible or Puppet for patch orchestration? Lucas: Ansible is my go-to for patch automation because it's agentless and idempotent. You can write a playbook that applies updates, reboots if needed, waits for the server to come back, and runs post-patch checks. And you can target groups — canary, web, db — with different schedules. Luna: But you still need a cron job or a scheduler to trigger that playbook, right? Lucas: Right. You can use AWX or Rundeck for a web UI, or just a cron job on a control node that runs ansible-playbook on a schedule. The key is that the playbook itself contains all the logic: update packages, handle reboot, verify health. That way, the schedule is just the trigger. Luna: What about servers that can't tolerate any downtime? Like a load balancer or a primary database. Lucas: For those, you need to patch in a maintenance window even with automation. Or use rolling updates — for a database cluster, you patch replicas first, then failover, then patch the former primary. That's more complex, but still automatable with tools like Ansible or a database-specific tool like pg_auto_failover for PostgreSQL. Luna: Let's zoom out. What's the single biggest mistake teams make when they first try to automate patching? Lucas: They don't test the automation itself. They write a script, run it once on a test server, it works, then they roll it out to production. But they haven't tested the failure modes — what happens if the repo is unreachable, or if the package list is empty, or if the reboot hangs. You need to simulate those scenarios. Luna: So chaos engineering for patching. Lucas: Exactly. Force a failure in staging. See if your rollback works. See if the alert fires. Because the worst time to discover your automation has a bug is during an actual patch window. Luna: Speaking of the real world, a quick honest thing — we keep this show ad-free and listener-supported. A handful of folks chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making episodes like this one possible. If you've gotten something useful out of today's conversation, that's where you can help keep it going. Lucas: Yeah, it's a small group but it makes a big difference. No pressure — just if the show is part of your week, that link is there. Anyway, back to patch schedules: one more thing I want to hit is the importance of a patch policy document. Luna: A document? Sounds boring, but I bet it matters. Lucas: It does. Write down your patch window, your escalation path, your rollback procedure. When something goes wrong, you don't want to improvise. You want to follow a playbook. That document is what you hand to a new hire or to an auditor. And it forces you to think through edge cases. Luna: Like what happens if patching runs long and overlaps with business hours? Lucas: Exactly. Or if the canary fails, do you investigate immediately or hold until the next window? Having those decisions pre-made saves time and reduces stress. My friend's team now has a runbook that says: if canary fails, page the lead engineer within 15 minutes. No waiting until Monday. Luna: Let me ask the listener's question: I have ten servers, not fifty. Do I still need all this? Lucas: Yes, but scale it down. You can still have a canary — pick one server. You can still automate with a simple bash script and cron. The principles are the same: test before rolling out, have a rollback plan, and never patch on a Friday. The cost of a failed manual patch on a small fleet is just as painful, maybe more so because you don't have redundancy. Luna: That's a good point. A single server outage can take down your whole business if you're small. Lucas: Right. So the takeaway: stop treating patching as a manual to-do. Build a schedule, automate the routine, and test the automation. It's one of those things that feels like overhead until the day it saves your weekend. Luna: Or your job. I've seen people fired over patching incidents. Lucas: True. And the irony is, the fix is mostly process and a few lines of code. Alright, that's the episode. If you want to dive deeper into Ansible playbooks for patching, we can do a follow-up. Let us know.