Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Systemd Journal for Centralized Logging
Transcript
- Lucas: Every Linux server produces logs. But when you have a dozen, a hundred, or a thousand servers, logging into each one and tailing journalctl becomes a non-starter. Luna: Right — you need a central place to see everything. And most people reach for something like the ELK stack or Graylog. Lucas: And those are great tools, but they also bring a lot of weight: Java runtimes, Elasticsearch clusters, index management. Sometimes you just want a lightweight, built-in solution that ships your systemd journal to a central server without any extra infrastructure. Luna: And systemd actually includes that capability out of the box, doesn't it? Lucas: Exactly. There's a pair of tools: systemd journal upload and systemd journal remote. They've been part of systemd for years, but I think a lot of admins either don't know about them or assume they're too basic. The thing is, for a lot of use cases they're exactly what you need. Luna: So let's break it down. What do we need on the sending side? Lucas: On each client server you install systemd journal upload — on Debian-based systems it's in the systemd journal remote package, oddly enough. You configure it with a simple config file at /etc/systemd/journal-upload.conf. You set the URL of your central server, optionally a trust store for TLS, and that's basically it. Luna: And the central server runs systemd journal remote, which listens for incoming logs and writes them into its own journal. Lucas: Right. It listens on a TCP port — default is 19532 — and it can accept plain connections or tls encrypted. Obviously for production you want TLS. You can configure it to store logs per-host or in a single combined journal. Luna: And the great part is that you query all those remote logs with the same journalctl commands you already know — just with the --merge flag or by pointing to the directory where remote journals live. Lucas: Let's walk through a concrete setup. On the central server, install systemd journal remote. Then create a directory for the remote journals — something like /var/log/journal/remote. Edit /etc/systemd/journal-remote.conf and set ListenStream=0.0.0.0:19532, point Store to that directory, and enable TLS by pointing to your certificate and key. Luna: For TLS, you'd ideally have a proper CA. You could use Let's Encrypt or an internal CA. Lucas: Let's Encrypt works for servers with public DNS. For internal hosts, you can stand up a simple CA with easy-rsa or just use self-signed certs and distribute the CA cert to all clients. The important thing is that journal-remote and journal-upload both support TLS natively. Luna: So once the central server is listening, you configure each client. In /etc/systemd/journal-upload.conf you set URL=https://your central server:19532, point to the CA cert, and maybe set ServerKeyFile and ServerCertificateFile if the server requires client certificates. Lucas: Then you enable and start systemd journal upload.service. By default it'll start sending new journal entries immediately. It also catches up on old entries — you can control the backlog with options like --save-state or the built-in state file. Luna: One thing that trips people up: the journal on the client can get pretty big. By default, systemd journals are persistent and can grow to 10% of the filesystem. That's a lot of data to ship over the wire initially. Lucas: Good point. You can set limits in /etc/systemd/journald.conf — MaxUse, KeepFree, MaxFileSize, MaxRetentionSec. For central logging, you might want to reduce the local retention because you're already shipping logs off-box. Or you could use SystemMaxUse to cap it at, say, 500 megabytes. Luna: And for the initial catch-up, you might want to throttle. There's no built-in rate limiting in journal-upload itself, but you can use systemd resource control — set CPUQuota or IOWeight on the service unit. Lucas: Exactly. Or you could simply delete old journals on clients before enabling upload, so only new logs get sent. It's a trade-off. Luna: If today's conversation gave you something usable, the way these episodes stay ad-free is listener support. You can buy me a coffee dot com slash fexingo — it's a small way to keep the server lights on, as it were. Lucas: Yeah, appreciate that. It genuinely helps. Alright — back to the setup. So after the clients are configured, on the central server you should see journals appearing in the remote directory. You can test with journalctl --directory=/var/log/journal/remote — or better, use journalctl --merge to see local and remote logs together. Luna: But there's a catch: --merge might mix timestamps from different time zones. You want all your servers to use UTC if possible. Lucas: Absolutely. Consistent time zones across the fleet is table stakes for any centralized logging. Also make sure NTP is solid everywhere — chrony, not the old ntpd. Luna: What about filtering? Can you route certain logs from certain hosts to different storage locations? Lucas: That's where it gets interesting. systemd journal remote can filter by fields like _HOSTNAME or _SYSTEMD_UNIT using the SplitMode=host option. If you set SplitMode=host in journal-remote.conf, it creates a separate journal file per sending host. You can also use journal-upload's --key and --cert for per-client identities. Luna: And then on the central server you could run separate instances of journal-remote on different ports, each pointing to a different storage directory. Lucas: That's a valid approach for organizational separation. Or you could use journalctl's built-in filtering to query per host — journalctl -p err --directory=/var/log/journal/remote works fine. It's not as fancy as a web UI, but for many ops teams it's enough. Luna: Let's talk about reliability. What happens if the network goes down or the central server is unreachable? Lucas: journal-upload handles that gracefully. It keeps track of the last successfully sent cursor in its state file. When the connection comes back, it resumes from that point. No gaps, no duplicate floods. Luna: But the state file is per-client, right? So if you restart the service or lose the state file, it might re-send everything. Lucas: That's a real risk. You should back up the state file or at least ensure it's on persistent storage. If you're running in a container, you'd want to volume-mount that file. Also, you can set --trust=all on the server side to accept all connections, but for production you'll want certificate verification. Luna: And you can use journalctl --verify on the remote journals to check integrity. Lucas: Exactly. The journals themselves are checksummed. If a log entry gets corrupted in transit or on disk, journalctl will report it. That's a nice peace of mind feature you don't get with plain syslog. Luna: So the use case is clear: if you have a handful to a few dozen servers and you want centralized logging without spinning up Elasticsearch, systemd's native tools are a solid choice. Lucas: I'd say even a few hundred servers could work, especially if you're only sending priority logs — say, err and above — and you keep the retention reasonable. The overhead is minimal: just a few megabytes per client for the service, and the network traffic is compressed by default. Luna: What about searching? journalctl has --grep now, but it's not as fast as a dedicated index. Lucas: It uses the journal's built-in indices, so it's reasonably fast for field-based searches. Full-text grep can be slower, but for targeted queries like 'all errors from nginx on host web-01 in the last hour' it's nearly instant. Luna: And you can combine it with tools like lnav for a more interactive experience. Lucas: Sure. But the beauty is you don't have to — journalctl alone is a powerful query language once you learn the field filters. _SYSTEMD_UNIT, _PID, PRIORITY, _HOSTNAME, _COMM — these are all indexed. Luna: One last thing: rotation. The central server's journals need to be rotated, or they'll fill up the disk. Lucas: Right. systemd journal remote doesn't rotate automatically — it relies on journald's own rotation settings on the central server. You should configure MaxRetentionSec or MaxUse in /etc/systemd/journald.conf on the central host. And if you split by host, each host's journal file is rotated independently. Luna: So you could set MaxUse=10G and SystemKeepFree=5G, and systemd handles the rest. Lucas: Yes. And you can use journalctl --vacuum-size=5G to manually trim, or schedule a daily vacuum job with a systemd timer. Luna: Alright — I think we've got enough to get someone started. Any final pro tip? Lucas: Test with a single host on a LAN before rolling out. Use journalctl -f on the central server to watch logs arrive in real time. And don't forget to open the firewall on port 19532. Luna: Good advice. And if you want to go deeper, the systemd journal documentation is actually quite readable. Lucas: Next time we'll talk about forwarding those central logs into a SIEM or long-term archive. But for now, if you've got a handful of servers, you've got all you need.