Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux systemd Journal for Centralized Logging
Transcript
- Lucas: So today I want to talk about something that every sysadmin eventually has to deal with: log management. Specifically, systemd's journal — journald — and why you might want to use it instead of traditional syslog for centralised logging. Luna: I've been meaning to dig into journald more. My default is still tailing /var/log/syslog. What's the real advantage? Lucas: The big one is structure. Traditional syslog gives you a flat text line with a timestamp, hostname, and a message. That's fine for human reading, but it's terrible for machines. Journald stores logs as binary entries with structured metadata — things like the unit name, PID, priority, and even extra fields like JSON payloads if the application writes them. Luna: Right, so you can search on metadata instead of grepping for strings. That makes sense. But is it persistent by default? Lucas: That's the gotcha. On most distributions, journald runs in volatile mode — logs live in memory under /run/log/journal. That's great for performance, but if your server crashes or reboots unexpectedly, you lose all the logs. For a production server, you almost always want persistent storage. Luna: Okay, so how do you enable that? Lucas: You create the directory /var/log/journal. Then systemd-journald will automatically start writing there. You can also set Storage=persistent in /etc/systemd/journald.conf. But you need to be careful about disk usage. Journald can eat up a lot of space if you don't cap it. Luna: Yeah, I've seen journal files grow to gigabytes on busy servers. What's a good limit? Lucas: I usually set SystemMaxUse=500M and MaxRetentionSec=1week. That keeps the last week of logs at around 500 megabytes. You can also use RuntimeMaxUse for the volatile journal. The key is to test — if you have a high-volume application like a web server, you might need more. Luna: Let's talk querying. journalctl is the tool, right? I've used it for basic things like journalctl -u nginx, but I know there's more. Lucas: journalctl is incredibly powerful. The -u flag filters by unit, but you can combine filters. For example, to see all SSH failures in the last 24 hours, you'd do: journalctl _SYSTEMD_UNIT=sshd.service --since '2026-06-20' PRIORITY=4. That gives you only warning-level and above messages from the SSH daemon. Luna: PRIORITY=4 — that's warning, right? So 0 is emergency, 3 is error. Good for cutting through noise. Lucas: Exactly. And you can output in JSON with -o json-pretty, which is fantastic for feeding into a log aggregator like Loki or Elasticsearch. You get all the structured fields ready to parse. Luna: That's way cleaner than trying to parse syslog with regex. But what about forwarding to a central server? Journald can do that natively, right? Lucas: It can, but the native protocol is binary and not widely supported by traditional log collectors. What I recommend is using a socket to forward to rsyslog or syslog-ng. You set up ForwardToSyslog=yes in journald.conf, then configure rsyslog to forward to your central server over TCP or UDP. Luna: So journald is the local collector, and rsyslog handles the network transport. That's a common pattern I've seen. Lucas: Yeah, and it works well. But if you want a modern approach, you can also use journald's native gateway — systemd journal gatewayd — which serves the journal over HTTP. That's more niche, but useful for containerised environments where you don't want a full syslog stack. Luna: Let's talk about rate limiting. I've had journald drop messages under high load. How do you handle that? Lucas: By default, journald limits the rate of messages from a single unit to 10,000 messages per 30 seconds. If you exceed that, it starts dropping. You can tweak RateLimitIntervalSec and RateLimitBurst in journald.conf. For a noisy application like a debug-level logger, you might need to increase those values. Luna: But if you increase them too much, you risk disk I/O spikes. There's always a trade-off. Lucas: Right. In practice, I usually leave defaults and only adjust if I'm actively debugging a problem. The bigger issue is making sure you're not logging sensitive data — journald stores everything the application writes, including passwords or tokens if the app is badly coded. Luna: That's a good point. So for compliance, you might want to filter out certain fields before they hit the journal. Lucas: Exactly. You can use journald's LogNamespace feature to isolate logs from different services, or you can use a proxy like systemd journal upload to filter before forwarding. But that's getting advanced. Luna: I think a lot of our listeners are moving to journald from syslog. The biggest pain point I hear about is learning journalctl syntax. Got any go-to examples? Lucas: Sure. My most used command is: journalctl -f -u sshd.service — that follows logs from SSH in real time. For troubleshooting boot issues: journalctl -b -1 shows logs from the previous boot. And for a dump of all critical errors: journalctl -p err -b. Luna: The -p flag for priority is great. And you can combine it with --since. I use journalctl -p err --since yesterday all the time. Lucas: If you're ever stuck, journalctl --help is surprisingly readable. But I'll also note: the --no-pager option is essential if you're piping output into another command. Luna: One more thing — how do you handle log rotation with journald? Does it do it automatically? Lucas: Journald handles rotation internally. When it reaches the size limit or time limit, it archives the current file and starts a new one. You can see the list with journalctl --list-boots. Old journals are automatically deleted based on MaxRetentionSec. No cron job needed. Luna: That's cleaner than syslog rotation. I still find it weird that journald doesn't write plain text by default. Some teams still want grep-able files. Lucas: You can export to text with journalctl -o short-iso, or convert an entire journal with journalctl --output=export > logs.txt. But honestly, once you get used to the structured querying, you won't want to go back. Luna: Alright, I'm convinced. I'm going to switch my test server to persistent journald today. Thanks for the walkthrough. Lucas: Glad it was useful. And hey — if today's tech conversation gave you something you can actually use on your servers, the way we keep this show ad-free is listener support. You can buy us a coffee at buy me a coffee dot com slash fexingo. Luna: Yeah, it really helps. Every bit goes straight into production costs and keeps the content focused on what matters — no sponsors, no fluff. Lucas: Exactly. So back to journald — one more tip: if you're running containers, consider using journald as the log driver for Docker or Podman. It gives you structured logs per container without needing a separate logging agent. Luna: That's a great point. I've set that up with Podman and it integrates seamlessly with systemd. Each container gets its own unit, so journalctl -u container-name works out of the box. Lucas: And you can use the same journalctl filters to find errors across all containers. That's a huge win for troubleshooting in a microservices environment. Luna: Final thought: never leave journald in volatile mode on a production server. I've seen too many post-mortems where the logs vanished after a crash. Lucas: Couldn't agree more. Make the directory, set the limits, and forward to a central server. Your future self will thank you.