Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Tame Linux Server Timeouts With Systemd Drop-In Units
Transcript
- Lucas: You're running a database migration on a PostgreSQL server, it's been going for about three minutes, and you need to restart the service to pick up a configuration change. You type systemctl restart postgresql, and then you wait. And wait. And after ninety seconds, systemd kills the process. Luna: That ninety-second wall. I've been there. The migration fails, the service goes down hard, and you're left with a corrupted state and a pager. Lucas: Exactly. The default TimeoutStopSec in systemd is ninety seconds. For most services, that's plenty. But for long-running operations — database migrations, backup scripts, batch jobs — ninety seconds is a recipe for disaster. Today I want to talk about drop-in units, which are systemd's cleanest way to override those defaults without touching the vendor's unit file. Luna: Why not just edit the unit file directly? I've done that. It works. Lucas: It works until the next package update. When postgresql gets upgraded, the package manager overwrites /usr/lib/systemd/system/postgresql.service. Your changes are gone. A drop-in unit lives in /etc/systemd/system/<service>.d/ — a directory that the package manager never touches. It layers on top of the vendor unit. You can override specific directives, like TimeoutStopSec, without copying the entire file. Luna: So it's like a config drop-in for nginx or sshd. A.conf file that only changes what you specify. Lucas: Exactly the same principle. You create a directory named after the service with a.d suffix, then put a file ending in.conf in it. Systemd reads all of them in lexicographical order, and the last one wins. For example: /etc/systemd/system/postgresql.service.d/override.conf. Luna: And inside that file you just write the section and key you want to change? No copying the whole unit? Lucas: Right. You write and then TimeoutStopSec=300. That's it. Then run systemctl daemon-reload and systemctl restart postgresql. Systemd merges the drop-in with the vendor unit, and your custom timeout takes effect. The original unit file stays pristine. Luna: Okay, but what about TimeoutStartSec? I've had services that take a long time to initialize — like a Java app server warming up its cache. Lucas: Same mechanism. TimeoutStartSec defaults to ninety seconds too. You can set it to infinity with TimeoutStartSec=infinity, but I don't recommend that — if the service truly hangs, you want it to fail eventually. Better to set a reasonable upper bound based on your observability data. If your app server consistently starts in forty seconds, set it to sixty. You get a buffer without losing the guardrail. Luna: Makes sense. And this is all documented in systemd.service man page, but I think a lot of people don't know about the drop-in pattern. They go straight to editing the unit file or they just accept the default timeout. Lucas: Right. And the default timeout isn't just for stop and start. There's also TimeoutSec, which is a shorthand that sets both TimeoutStartSec and TimeoutStopSec to the same value. And there's RuntimeMaxSec — which is less common but useful for services that should only run for a limited time, like a one-shot backup script. Luna: I've never used RuntimeMaxSec. Could you give a concrete example? Lucas: Sure. Imagine you have a systemd timer that kicks off a nightly database dump. That dump should never take more than two hours. You set RuntimeMaxSec=7200 in the service's drop-in. If the dump hangs — maybe the disk is slow or the database is locked — systemd will kill it and mark the unit as failed. That's better than having a zombie process that never completes, silently blocking the next day's backup. Luna: So you're using systemd as a watchdog. That's clever. But what if you need different timeouts for different operations? Like a long timeout for stop but a short one for start? Lucas: You can set them independently. TimeoutStartSec and TimeoutStopSec are separate directives. So for a database service, you might want a generous TimeoutStopSec because you want to let in-flight transactions finish, but a tight TimeoutStartSec because if the database doesn't come up quickly, something is wrong. I usually set TimeoutStopSec to 300 for PostgreSQL, and keep TimeoutStartSec at the default ninety. Luna: And you do all this in one drop-in file? Or do you split them into multiple files? Lucas: One file is fine for most services. But you could split them: 10-timeouts.conf for timeout settings, 20-resource.conf for memory limits, 30-environment.conf for env vars. Systemd reads them alphabetically, so numbering is a common convention. That way you can have different drop-ins managed by different teams or different configuration management tools, and they don't step on each other as long as they don't override the same key. Luna: That's a nice pattern for Ansible or Puppet. You can drop a file per setting, and each role manages its own concern. Lucas: Exactly. And it makes auditing easy. You can list all drop-ins for a service with systemctl cat postgresql, which shows the merged unit, or systemctl show postgresql to see the effective property values. If something goes wrong, you can check which drop-in is causing the issue by looking at the file names. Luna: I want to go back to something you said earlier — the package manager doesn't touch /etc/systemd/system. But what about if the vendor ships a drop-in of their own? Some packages put files in /usr/lib/systemd/system/<service>.d/. Do your drop-ins in /etc override those? Lucas: Yes. The order of precedence is: /etc overrides /run overrides /usr. So any drop-in in /etc systemd/system/<service>.d/ will override one in /usr/lib/systemd/system/<service>.d/. And within the same directory, lexicographic order applies. So if you want to ensure your override wins over both the vendor unit and any vendor drop-ins, put it in /etc with a high alphabetical prefix, like 99-override.conf. Luna: Good. I had a case where a vendor shipped a drop-in that set a very low timeout, and I couldn't figure out why my override wasn't taking effect. Turned out my drop-in was named 00-override.conf and theirs was 99-vendor.conf. I was alphabetically earlier. Lucas: That's the classic gotcha. Always check the merged unit with systemctl cat to see what's actually taking effect. Don't assume your file wins just because you put it in /etc. Luna: Speaking of checks, let's talk about production migrations. I've seen people increase TimeoutStopSec to an hour because they don't want to tune the actual operation. That feels like a crutch. Lucas: It is a crutch, but sometimes it's the right stopgap. If you have a migration that genuinely takes forty-five minutes on a good day, setting TimeoutStopSec to 3600 isn't unreasonable — as long as you also have monitoring that alerts if it goes over that. The real fix is to make the migration faster or break it into smaller chunks, but in the meantime, you'd rather have the service shut down gracefully than be killed mid-transaction. Luna: And with drop-ins, you can change that timeout easily as you improve the migration. No need to touch the main unit file. Lucas: Right. And that brings up another point: drop-ins aren't just for timeouts. You can override any unit directive. Environment variables, resource limits with LimitNOFILE, even the ExecStart command itself — though overriding ExecStart is risky because you're replacing the entire command, not appending to it. Luna: So if you want to add an extra flag to a service, you can't just add it in a drop-in. You have to redefine the whole ExecStart line. Lucas: Correct. That's one of the limitations. For something like adding --verbose to a daemon, you either have to copy the entire ExecStart line into your drop-in — which then breaks if the vendor changes the original command — or you use an environment variable, if the service supports it. Some services allow you to pass extra arguments via an environment file. That's cleaner. Luna: Let's pivot to a practical workflow. Say I'm setting up a new service on a server. What's your recommended approach to timeouts from day one? Lucas: First, don't rely on defaults. After you install the service, run systemctl cat <service> to see the full unit. Check TimeoutStartSec and TimeoutStopSec. If they're not explicitly set, systemd uses the built-in defaults. Then, based on your understanding of the service's startup and shutdown profile, create a drop-in with explicit values. Even if you set them to the same defaults, you're documenting your intent. Later, when you have real metrics, adjust. Luna: So you're saying every service should have a drop-in, even if it just duplicates the defaults? Lucas: I wouldn't say every service, but certainly any service that's critical or has non-trivial startup or shutdown behavior. For a simple nginx static file server, the ninety-second default is fine. For a database, a message queue, a custom application — yes, create a drop-in. It's a few lines in a file, and it saves you a late-night debugging session when a migration or a cache warmup hits the timeout. Luna: And if you're using configuration management, you can template those drop-ins. Ansible can loop over a list of services and drop a file for each one. Lucas: Exactly. One thing I'll add: don't forget to run systemctl daemon-reload after creating or modifying a drop-in. Otherwise systemd doesn't know about the new configuration. I've forgotten that more times than I'd like to admit. Luna: And reload doesn't restart the service. So it's safe to run during production. Lucas: Safe, yes. But you still need a restart or a reload of the service itself to pick up the new timeout values, because daemon-reload only tells systemd to re-read the unit definitions. The running service continues with its old settings until it's restarted. Luna: Good distinction. So the sequence is: create drop-in, daemon-reload, then restart the service during a maintenance window. Lucas: Perfect. One more thing — directory permissions. The.d directory and its contents should be owned by root, with 755 on the directory and 644 on the files. If permissions are wrong, systemd silently ignores the drop-in. I've debugged a case where a colleague created the directory as root but the files had 600, and systemd just skipped them. No error, no warning. The service ran with defaults. Luna: That's nasty. Silent failure. So always verify with systemctl show <service> | grep Timeout after deploying a drop-in. Lucas: Absolutely. And if you want to be extra safe, you can use systemctl verify <service> — it checks the unit file and drop-ins for syntax errors and missing dependencies. Luna: That's a command I don't use enough. Okay, I'm convinced. Drop-ins are the way to go for timeout overrides. And I'll start adding them to my standard server setup. Lucas: Glad to hear it. And look — if you found this conversation useful, it's exactly the kind of practical, hands-on content we try to bring you every week on this show. We keep it ad-free and focused on what helps you in your day job. A handful of listeners already support us through buy me a coffee dot com slash fexingo, and that support is what keeps the lights on and lets us spend time on episodes like this one. Luna: Yeah, no sponsors, no tracking, just server advice from people who actually run servers. If you've gotten value from these episodes — maybe a tip that saved you an hour of debugging — and you want to help keep the show going, that's where you can chip in. Lucas: No pressure, really. But if it's something you believe in, it makes a difference. And now back to timeouts — one last tip. If you're ever in a situation where a service won't stop and you need to force it, you can use systemctl kill --signal=SIGKILL <service>. But that's a last resort. With proper TimeoutStopSec settings and drop-ins, you shouldn't need it often. Luna: Good final note. So to recap: drop-ins are clean, package-upgrade safe, and the right tool for timeout tuning. Start with a directory, a.conf file, and a daemon-reload. Lucas: Exactly. And as always, test in staging first. Happy timeout tuning.