Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your Linux Server Needs a Dedicated Journald Configuration
Transcript
- Lucas: So you're chasing a production issue — maybe a web server that's intermittently returning five-oh-x errors — and you go to pull the logs. You run journalctl — list-boots, and you see a gap. Boot two shows logs for two hours, then silence for forty minutes, then boot three picks up. And you think, okay, maybe it rebooted. But it didn't. Luna: You're saying the logs just stopped mid-session? That's unsettling. Lucas: Exactly. And the culprit is almost always journald's default rate-limiting configuration. Systemd's journal daemon is designed to prevent a chatty process from filling up your log partition, which is a noble goal. But the default limits are surprisingly low — ten thousand messages per thirty seconds by default. On a busy server running something like Nginx or PostgreSQL with debugging enabled, that threshold gets smashed in seconds. Luna: Wait, so it's silently dropping logs? That seems like a design choice that makes debugging harder at the exact moment you need logs most. Lucas: That's the trade-off. The default is meant for workstations or low-traffic servers. In production, you need to tune it. And the fix is straightforward: create a drop-in configuration file in /etc/systemd/journald.conf.d/. You don't touch the main journald.conf, because package updates might overwrite it. You drop a.conf file with just the overrides you need. Luna: What would you put in there? I'm guessing RateLimitIntervalSec and RateLimitBurst are the main ones. Lucas: Exactly. I typically set RateLimitIntervalSec to one second and RateLimitBurst to fifty thousand — that's fifty thousand messages per second allowed. For most web servers that's plenty. And then you want to set SystemMaxUse to control total disk usage, say ten gigabytes. And MaxRetentionSec to eight weeks if you're worried about old logs piling up. The key is to test it with journalctl — verify that the new limits took effect by checking journalctl — header. Luna: And that header output shows the current limits? I've never looked at that. Lucas: It shows you the runtime configuration — the rate limit intervals and bursts, the max disk usage, whether storage is persistent or volatile. It's a good sanity check. But there's another piece: by default, journald stores logs in /run/log/journal, which is a tmpfs — volatile, lost on reboot. If you want logs to survive crashes, you need to enable persistent storage by creating /var/log/journal. The daemon detects that directory automatically and starts writing there. Luna: Right, but then you need to think about disk space and rotation. If you're logging at fifty thousand messages per second, ten gigabytes might fill up fast. Lucas: That's where SystemMaxUse and MaxRetentionSec become critical. And you can also use journalctl's built-in vacuum commands: journalctl — vacuum-time=30d or — vacuum-size=5G to clean up manually. But ideally you set the limits in the config and let systemd handle rotation. Luna: So what about forwarding logs to a central aggregator? If you're tuning journald to keep more logs locally, you might also want to ship them off-box. Lucas: Absolutely. Journald can forward to syslog, to kmsg, or to a socket that tools like rsyslog or Vector or Logstash can consume. The ForwardToSyslog option is there, but I usually prefer to use journald's own native transport — journald's journal-remote or systemd journal upload for central collection. It's binary, but it preserves structured metadata like code locations and kernel fields. Luna: And if you're using something like the Elastic stack, you'd run Filebeat with the systemd input module, right? Lucas: Yes. Filebeat's systemd module reads from journald natively. It's efficient. But all of that depends on journald not dropping your logs at the source. So the first step is always making sure your local rate limits match your actual traffic. Luna: This is genuinely useful — I think a lot of people assume journald just works out of the box in production. Lucas: If today's talk gave you something you can use, that's exactly the point. These episodes stay ad-free because of listener support — you can find us at buy me a coffee dot com slash fexingo. Every bit helps us keep digging into these topics. Luna: Yeah, it's a small way to make sure we can keep going deep on the practical stuff. Lucas: Anyway, back to journald — one common mistake is forgetting that rate limiting applies per-service by default. Each unit gets its own rate limit counter. So if you have a dozen services all logging heavily at the same time, each one can hit its limit independently. That can lead to partial log loss. Luna: Oh, that's a nuance I hadn't considered. So the ten thousand messages per thirty seconds is per unit, not global? Lucas: Exactly. The setting journald.conf has a global default, but each service's own rate limit — set via LogRateLimitIntervalSec and LogRateLimitBurst in the service unit — overrides that. Many services don't set them, so they inherit the global default. But some, like systemd-resolved, have their own internal limits. You can check with systemctl show some-service — property LogRateLimitBurst. Luna: So you could increase rate limits globally, but then a single buggy service could flood the journal. That's the trade-off you mentioned. Lucas: Right. That's why I prefer to increase the global limit modestly and then raise the limit per service on the ones that need it, like your database or web server. You do that with a drop-in for the service unit itself — systemctl edit some-service, and add LogRateLimitIntervalSec and LogRateLimitBurst. That way, noisy services get room, but a runaway script still gets throttled. Luna: That's a clean pattern. And speaking of drop-ins, what about the journald.conf.d directory — any gotchas with file naming? Lucas: The files have to end in.conf, and they're read in lexicographic order. So if you have multiple files with conflicting settings, the last one sorted wins. I usually name mine something like 99-local.conf to ensure it's loaded last and takes precedence. Place it in /etc/systemd/journald.conf.d/. Luna: And you need to restart systemd-journald to pick up the changes? Lucas: Yes, systemctl restart systemd-journald. But be aware — restarting drops the volatile journal if you're not using persistent storage. So it's safer to enable persistent storage first, then restart. And always test the config with journalctl — header to confirm the new limits. Luna: I've seen cases where people set SystemMaxUse too low and then wonder why old logs are missing. A ten-gigabyte limit on a system with heavy logging might only hold a few hours of history. Lucas: Exactly. You need to estimate your log volume. A quick way: take the number of messages per second from journalctl — header, multiply by the average message size — maybe five hundred bytes — and then figure out how much disk you need for your desired retention. For a busy web cluster, a hundred gigabytes or more isn't unreasonable if you're keeping weeks of logs. Luna: And don't forget about rotation. Journald handles rotation automatically based on size and time, but if you're using rsyslog or another syslog daemon alongside it, you might get double rotation issues. Lucas: Right. If you're forwarding to syslog, make sure rsyslog isn't also doing its own rotation on the same files. I've seen cases where rsyslog's logrotate cron job deletes journald's persistent files because they're in the same directory. Not ideal. Luna: So best practice is: either use journald alone with its built-in rotation, or forward to a dedicated aggregator and let journald keep a smaller local buffer. Lucas: Yes. For most production setups, I recommend persistent journald with a generous size limit, and then forward everything to a central system like Elasticsearch or Loki. That way, even if a server crashes and the disk fails, you have the logs off-box. Luna: And when you're debugging a crash, having those persistent logs from the previous boot is invaluable. journalctl — boot=-1 saves you so many headaches. Lucas: Absolutely. That one flag — — boot=-1 — is why persistent journald is worth the disk space. Without it, you're blind to what happened before the reboot. And if your journald was rate-limiting at the moment of the crash, you might not even have the crash log itself. Luna: Yeah, that's the worst: the logs you need most are the ones that got dropped. Lucas: So the takeaway is: don't assume the defaults are right for your workload. Check your log volume, set appropriate rate limits, enable persistent storage, and verify with journalctl — header. It's a few minutes of config work that can save hours of frustration during an incident. Luna: And if you're using containers or orchestration, remember that each container's journald is separate — you might need to tune inside the container or use a logging driver that bypasses journald entirely. Lucas: Good point. Docker's default logging driver is json-file, not journald. But if you use journald as the driver, you get the same rate-limiting issues. So the same principles apply: set limits in the container's journald.conf or switch to a driver that doesn't throttle. Luna: Alright, so the action items are clear: check your current journald config, create a local override, and test. Anything else? Lucas: One last thing: if you're on an SSD, consider setting SystemMaxUse to avoid excessive writes. Journald can be write-heavy, and SSDs have limited write endurance. Spreading the logs across a separate disk or using a RAM buffer can help. But that's maybe a topic for another episode. Luna: Yeah, we could do a whole show on disk I/O and logging. For now, I think this is a solid upgrade for anyone running Linux servers. Lucas: I hope so. Go check your journalctl — header. If you see a gap in your logs, you know what to look at.