Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Tame Linux Server Logs with Logrotate
Transcript
- Lucas: Let me tell you about the time a production web server went down at 3 AM because nobody checked the logs. And by checked the logs, I mean nobody checked the logrotate configuration. Luna: Oh, that classic. The disk just filled up silently with old Apache access logs, right? Lucas: Exactly. Apache had been running for months, generating maybe 500 megabytes of logs per day. Logrotate was supposedly handling rotation, but it was set to rotate only once a week and keep four weeks' worth — uncompressed. That's about fourteen gigabytes, which on a forty-gig root partition with databases and application code is a ticking time bomb. Luna: So the disk hits 100 percent, Apache crashes, and suddenly you're ssh ing in at 3 AM with a frantic call from the NOC. Lucas: Yep. And the fix? Delete a few old log files, fix the logrotate config, restart Apache. But the lesson is that logrotate is one of those tools that runs quietly in cron — you forget it's there until it breaks. So today, let's walk through a proper logrotate configuration, the gotchas, and how to audit your existing setup before it bites you. Luna: I think a lot of people install logrotate and assume the defaults are fine. But the defaults are conservative — they're designed to not break anything, not to be optimal for your production load. Lucas: Right. The default logrotate.conf on Ubuntu typically rotates weekly, keeps four weeks, and compresses with gzip. That's a reasonable starting point, but it doesn't account for log volume. If you have a busy web server or a verbose application, four weeks of uncompressed logs could be twenty or thirty gigabytes. Luna: And compression helps a ton. Apache text logs compress at a ratio of about ten to one. So a 100 megabyte uncompressed log becomes ten megabytes after gzip. Lucas: Exactly. So the first thing I do when I set up logrotate for a service is specify daily rotation, compression, and a retention period based on business needs — usually 30 or 90 days. But you also want to use the 'delaycompress' directive. That means the most recent rotated log is kept uncompressed, giving running services time to finish writing before the old file is compressed. Luna: And the 'copytruncate' option is another one to be careful with. Some people use it for logs that are written to by a process that doesn't reopen log files on SIGHUP. Lucas: Right. Copytruncate copies the log file to a new name, then truncates the original. But there's a small window between the copy and truncate where data is lost. Perfectly fine for low-volume logs, but on a high-traffic web server, that window could mean losing hundreds of lines. Better to use the standard create + postrotate reload method if the service supports it. Luna: Let's talk about postrotate scripts. That's where most mistakes happen. Lucas: Massively. The postrotate script runs after the log files are rotated. For Apache, you typically send a USR1 signal or run 'reload'. The gotcha is that if you're rotating multiple log files in one config block — say, all the virtual host logs — you need the 'sharedscripts' directive. Without it, the postrotate script runs once for every log file that's rotated. So if you have fifty virtual hosts, Apache gets reloaded fifty times in a row. Luna: And that can cause a reload storm, right? Each reload might take a second, so you're looking at nearly a minute of reloading, during which requests might get dropped. Lucas: Exactly. With 'sharedscripts', the postrotate runs once after all logs in that block have been rotated. So the golden rule: if you have more than one log file in a block, use 'sharedscripts' and handle the reload in the postrotate just once. Luna: Another thing: permissions. Log files are often owned by root or by the service user. If you set 'create' to a mode that doesn't match the service's permissions, the service might not be able to write to the newly created log file. Lucas: Good point. You can specify the owner and group in the create directive. For example, 'create 0640 www-data www-data' makes sure the new log is writeable by Apache. Otherwise, you end up with a root-owned log that Apache can't write to, and you get silent failures — Apache writes to stderr, but you don't see it until you check error logs. Luna: And don't forget maxage. If you want to delete logs older than a certain number of days, you set maxage — but it's absolute, not based on the rotation count. So if you rotate daily and set maxage 30, it will delete any log older than 30 days, even if you have fewer than 30 rotated files. Lucas: Right. I personally prefer 'rotate' plus 'maxage' together. Rotate keeps a fixed number of files, maxage ensures old ones are purged even if they haven't been rotated enough times. But you have to be careful — if your rotation frequency is too low, maxage might delete a file before the rotation cycle intends. Luna: Let's look at a concrete example. Say you have an Nginx server with logs in /var/log/nginx/. What does a good config look like? Lucas: I'd write something like this: /var/log/nginx/*.log { daily, rotate 30, compress, delaycompress, missingok, notifempty, create 0640 nginx adm, sharedscripts, postrotate if; then kill -USR1 $; fi; endscript }. Luna: That covers daily rotation, 30 days retention, compression with delay, and a safe postrotate that only reloads if the PID file exists. Nice. Lucas: One trap I see often: people put the postrotate script on one line and forget the semicolons between commands. The postrotate block is essentially a shell script segment. So if you have multiple commands, separate them with semicolons or newlines, and always end with 'endscript' on its own line. Luna: And test your config. You can run 'logrotate -d /etc/logrotate.d/nginx' to do a dry run. That shows you exactly what would happen without actually rotating anything. Lucas: Absolutely. The -d flag is your best friend. It prints debug output: which log files match, whether rotation is needed, and what commands would be executed. If something looks wrong, fix it before you run it for real. Luna: Now, there's also the matter of system logs — syslog, auth.log, kernel logs. Those are usually handled by the distro's default logrotate config. But you might want to customize them. For instance, if you're running a security audit, you might want to keep auth.log for 90 days instead of 4 weeks. Lucas: Right. The default configs in /etc/logrotate.d/ are fine for general use, but they're not tuned for your environment. My rule: after you install any service that writes logs, add a logrotate config for it. Don't rely on the package manager's default — it might be too aggressive or too lenient. Luna: And one more thing: if you're running in a containerized environment, logrotate inside a container can be tricky. Most container images don't include logrotate by default. You either need to mount your logs to a host volume and rotate them on the host, or use a sidecar container that handles log rotation. Lucas: Good point. In Docker, you can also use the built-in log driver with max-size and max-file options. That avoids logrotate altogether. But for traditional servers, logrotate is the standard and it's not going anywhere. Luna: So, if someone is listening and wants to audit their logrotate setup today, what's the one thing they should check first? Lucas: Run 'logrotate -d /etc/logrotate.conf' and look at the output for any config that would rotate a log file that's currently gigabytes in size. If you see a log that hasn't been rotated in weeks and is huge, that's your smoking gun. Also check that the postrotate scripts are correct with 'sharedscripts' where needed. And verify permissions — make sure the 'create' mode matches the service user. Luna: That's a solid checklist. And if you find a misconfiguration, you can force a rotation with 'logrotate -f /etc/logrotate.d/whatever', but be careful — that will rotate immediately regardless of frequency. Lucas: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these. It means we keep the show ad-free and focused on practical tips like this. So if today's logrotate walkthrough saved you an hour of debugging, that's exactly the kind of thing that keeps us going. Luna: Yeah, it's a small group that already keeps it running. No pressure, just if it's useful, you know where to find it. Lucas: Alright, back to auditing. So the second thing I'd check is the 'missingok' directive. If a log file is missing — say you removed a virtual host but didn't update logrotate — without 'missingok', logrotate will throw an error and potentially stop processing other logs in that block. So always include 'missingok' for safety. Luna: And 'notifempty' prevents rotating an empty log file, which saves unnecessary I/O and compression. Usually a good idea. Lucas: Exactly. So to wrap up: logrotate is a simple tool but the defaults can lead to disk-full disasters. Customize rotation frequency, retention, compression, and postrotate scripts for each service. Use dry runs. Check permissions and sharedscripts. And monitor your disk usage — even with perfect logrotate, logs can still grow unexpectedly if an application goes into a verbose debugging mode. Luna: Good point. Logrotate handles normal rotation, but you still need monitoring. Set an alert when disk usage hits 80 percent, and you'll catch anomalies before 3 AM. Lucas: That's the takeaway. A little upfront config saves a lot of late-night panic. Thanks for listening, and we'll be back with another server deep dive soon.