Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Logrotate for Log Management
Transcript
- Lucas: Alright, let's talk about logrotate. Not the sexiest topic in server engineering, but if you've ever had a server grind to a halt because /var/log ate the entire disk, you know it's one of the most important. Luna: Oh, I've definitely been there. A single misbehaving application writing debug logs at full speed — bye-bye free space. Lucas: Exactly. And logrotate is the standard tool on nearly every Linux distribution for preventing that. It's a cron job that compresses, archives, and eventually deletes old log files based on rules you define. But here's the thing — a lot of people just accept the defaults and never think about it again. Luna: And the defaults are fine for a basic setup, but they're probably not right for your actual workload. So what should we actually configure? Lucas: Before we dive in — if you've gotten something usable out of these episodes, consider supporting the show. A couple of dollars a month on buy me a coffee dot com slash fexingo genuinely keeps these going ad-free. It's a small thing that makes a real difference. Luna: Yeah, totally. We don't have sponsors, so it's just listeners like you keeping the lights on. Lucas: Alright, back to logrotate. So the configuration lives in /etc/logrotate.conf and /etc/logrotate.d/. The main file sets global defaults — like weekly rotation, keeping four weeks of logs, and compressing them with gzip. Then per-service configs in logrotate.d override those. Luna: So if I have Nginx running, I'd create something like /etc/logrotate.d/nginx. What does a good config look like? Lucas: Let me give you a concrete example. Say you have Nginx serving about 50,000 requests per day. That generates maybe 100 to 200 megabytes of access logs daily. You don't want to rotate daily because you'd end up with too many small files. Weekly is better. So you write: /var/log/nginx/*.log { weekly rotate 8 compress missingok notifempty postrotate invoke-rc.d nginx reload 2>/dev/null endscript } Luna: Wait — why 'notifempty'? I've seen that in a lot of configs. Lucas: It means skip rotation if the log file is empty. Saves you from creating empty gz files. And 'missingok' means don't error out if a log file doesn't exist — useful if you have multiple virtual hosts and some logs are missing. Luna: Got it. And the postrotate script reloads Nginx so it starts writing to the newly created log file. That's important, because Nginx doesn't reopen logs on its own. Lucas: Right. Some services like rsyslog handle reopening automatically with SIGHUP. But for Nginx and Apache, you need that postrotate. Now, the key decision is rotation frequency. You have daily, weekly, monthly. For high-traffic servers, you might even rotate hourly with a custom cron job. But weekly is a solid default for most production workloads. Luna: What about retention? 'rotate 8' means keep eight weeks of logs. That's two months. Is that enough for compliance or troubleshooting? Lucas: It depends. For pci dss or SOC2, you might need six months or a year. You can increase 'rotate' to 52 for weekly — that gives you a year. But watch your disk space. A 200-megabyte weekly log, compressed to maybe 40 megabytes, times 52 files is about two gigabytes. That's fine. But if you have verbose debug logs, it adds up fast. Luna: And you can also set 'maxage' to delete logs older than a certain number of days. That's more predictable than relying on 'rotate' count. Lucas: Exactly. 'maxage 30' deletes any rotated log older than 30 days, regardless of how many files there are. That's useful when log sizes vary wildly. Now, one feature I love is compression. By default, logrotate uses gzip, which gives good compression ratios. But you can change to bzip2 or xz for better compression at the cost of more CPU time. Luna: And you can also use 'delaycompress' — that skips compression on the most recent rotated log. So if you need to tail the old log immediately after rotation, it's still uncompressed. Lucas: Good point. Let's talk about 'copytruncate'. This is for services that don't close their log files — like Docker containers or some Java apps. Instead of moving the file and creating a new one, logrotate copies the content to a new file and truncates the original to zero bytes. The app keeps writing to the same file descriptor. Luna: But there's a risk of losing a few log lines between the copy and the truncate. For most use cases that's acceptable, but for audit logs you might not want that. Lucas: Right. If you need zero data loss, you're better off using 'create' and a proper postrotate script that makes the service reopen its log file. But for Docker containers, copytruncate is often the only option. Luna: So how do you test a logrotate config without waiting for the cron job to run? Lucas: Use 'logrotate -d /etc/logrotate.d/nginx'. The -d flag runs in debug mode — it shows you what would happen without actually rotating anything. Then you can run 'logrotate -f' to force an immediate rotation and verify your config. Always test on a non-production server first. Luna: What about the global config? I usually set 'rotate 4', 'weekly', 'compress', and 'delaycompress' globally, then override per service. Lucas: That's a smart baseline. I'd also add 'create 0640 root adm' to the global config so new log files have the right permissions. And set 'dateext' to append the date to rotated filenames instead of a simple number. That makes it easier to find logs from a specific day. Luna: Yeah, 'dateext' is great. Instead of naming files 'syslog.1.gz', 'syslog.2.gz', you get 'syslog-20260701.gz'. Much clearer. Lucas: One more thing: if you use systemd's journald, logrotate doesn't touch those logs. Journald has its own rotation built-in, controlled by SystemMaxUse and MaxRetentionSec in /etc/systemd/journald.conf. But for traditional text logs, logrotate is still the standard. Luna: That's a good clarification. So for a modern server, you might have journald for systemd services and logrotate for application logs. Two systems, but both need attention. Lucas: Exactly. Now, let's talk about a common mistake: forgetting that logrotate runs from cron daily. If you set 'daily' rotation but your cron runs logrotate once a day, it works. But if you set 'hourly', you need to add a cron job yourself, or use systemd timers. Logrotate itself doesn't enforce the schedule — it just checks if enough time has passed since last rotation. Luna: And you can check the last rotation time in /var/lib/logrotate/status. That file shows when each log was last rotated. Very handy for debugging. Lucas: So to wrap up: know your log volume, choose a rotation frequency that matches it, set retention based on compliance needs, test with debug mode, and always verify postrotate scripts. Logrotate is simple but powerful — and ignoring it is a classic way to crash a server. Luna: And if you're building a new server, add logrotate configs early. It's much easier than recovering from a full disk at 2 AM. Lucas: That's the takeaway. Next time, we'll look at centralized logging with rsyslog and how to forward logs from multiple servers. Until then, keep your logs rotated.