Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Logwatch for Daily Reporting
Transcript
- Lucas: If you manage more than a handful of Linux servers, you've probably had that moment where you log in and think — did anything happen last night while I was asleep? You could grep through /var/log/messages, but that's like looking for a needle in a haystack of kernel timestamps. Luna: Yeah, and on a busy web server, that messages file can be tens of thousands of lines by morning. There's got to be a better way. Lucas: There is. It's called Logwatch — a lightweight, battle-tested tool that's been part of the Linux ecosystem since the late 90s. It parses your system logs every night and compiles them into a concise daily report, then either emails it to you or drops it in a file. You get the highlights without the noise. Luna: And it's in pretty much every distro's repos, right? apt install logwatch, yum install logwatch — that simple. Lucas: Exactly. And if this kind of practical sysadmin utility is useful to you — the way we keep this show ad-free is through listener support. If you got something out of today, you can buy me a coffee dot com slash fexingo. No pressure, just helps us keep delivering episodes like this one. Luna: Seconded. Now — let's get into how Logwatch actually works. Lucas, I've installed it before, but I've never tweaked the defaults. What happens out of the box? Lucas: Good question. Out of the box, Logwatch runs as a cron job — typically daily — and it sends a report to root@localhost. So if you're not checking mail on the server, you might never see it. First thing I do is redirect to a real email address or set it to output to a file. Luna: Right, because it uses sendmail or similar. On a modern server, you might not even have a mail transfer agent installed. Lucas: Exactly. So for a quick test, you can run it manually: logwatch --detail high --output stdout. That prints the full report to your terminal. It shows you everything from SSH login failures to disk usage warnings to service restarts. Luna: And the default detail level is probably medium. What does that look like in practice? Lucas: Medium gives you a summary per service. For SSH, you'll see something like: 'Failed logins from 10.0.0.23: 47 attempts, from 185.220.101.42: 12 attempts.' It groups by IP, gives you counts, and tells you if any accounts were involved. It doesn't show the full log lines, just the aggregate. Luna: That's actually super useful for a morning scan. You can spot a brute-force spike without digging through auth.log line by line. Lucas: Right. And the configuration lives in /etc/logwatch/conf/logwatch.conf. That's where you set MailTo, Detail, and Range — which lets you say 'yesterday' or 'today' or 'all' for historical reports. Luna: There's also service-specific overrides in /etc/logwatch/conf/services/, right? I remember you can create custom.conf files for particular daemons. Lucas: Yes. The Logwatch scripts themselves live in /usr/share/logwatch/scripts/services/. Each major service has its own filter — sshd, httpd, fail2ban, sudo, disk space, kernel. But you can override the default Detail level per service. So if you want SSH at 'high' but Apache at 'low', you can do that. Luna: And it's all Perl under the hood. The filters are Perl scripts that parse log files with regex. That's why it's been around so long — it's portable and fast. Lucas: Exactly. One thing I want to highlight — Logwatch can integrate with fail2ban. If you have fail2ban running, Logwatch will report how many IPs were banned and for which services. It's a nice closed loop: fail2ban blocks the attackers, Logwatch tells you about it in the morning. Luna: I had a situation on a staging server last month where Logwatch caught a brute-force that fail2ban hadn't been configured for yet. The report showed 2,000 SSH attempts from a single IP overnight. That gave me the kick to install fail2ban properly. Lucas: That's a perfect example. The report gave you signal. Without Logwatch, you might not have noticed for days. And the server wasn't critical, so you weren't checking logs manually. Luna: So Logwatch is essentially a last line of defense for awareness. But what about its weaknesses? When would you recommend something else? Lucas: Logwatch doesn't do real-time alerting. It's a daily digest. If you need immediate notifications for critical events — like a service crash or a disk filling up — you want something like Nagios, Prometheus, or even a simple script with systemd timers. Luna: And it doesn't index logs or support searching across days easily. It's more of a push model than a pull model. Lucas: Exactly. For pulling, you have tools like lnav — the log file navigator — which gives you an interactive terminal interface with sql like queries. Or you go full stack with the ELK stack or Graylog. But for a simple daily email, Logwatch is hard to beat. Luna: And it's zero infrastructure. No database, no daemon, no web interface. It's a cron job that runs in a few seconds. Lucas: Let's walk through a concrete setup. I'm assuming Ubuntu 24.04, but it's similar on RHEL. sudo apt install logwatch, then edit /etc/logwatch/conf/logwatch.conf. Set MailTo = your@email.com, Detail = Med, and Range = yesterday. Then run logwatch --detail med --mailto your@email.com to test. Luna: And if you don't have a mail server, you can output to a file: logwatch --detail high --output file --filename /var/log/logwatch-report.txt. Then have another script push that somewhere. Lucas: Right. And you can set the cron job to run whenever you want. The default is /etc/cron.daily/0logwatch, which runs at 6:25 AM on most Debian systems. You can move it to hourly if you want more frequent reports, but that's usually overkill. Luna: One thing I've done is create separate Logwatch configs for different servers. For a web server, I want Apache and MySQL details. For a database server, I want system logs and MySQL only. You can do that with the --service flag. Lucas: Absolutely. logwatch --service sshd --service httpd --detail high will only report on those two services. You can chain as many as you want. And if you want to exclude something, there's no direct exclude flag, but you can set the service Detail to No in the conf. Luna: I've also seen people use Logwatch to monitor custom logs by writing their own script in /usr/share/logwatch/scripts/services/. It's not trivial, but it's doable if you know Perl. Lucas: That's a deeper dive for another episode. For today, the key takeaway is: install Logwatch, point the report to your inbox, and start your day with a 10-second scan. It will catch things you miss. Luna: And it's free, open source, and actively maintained. The latest version is 7.11, and it still works on the newest kernels. Lucas: One last tip: if you're on a server with very high log volume — like a busy reverse proxy — you might want to increase the Detail to 'high' and pipe the report through less. The default medium report can still be long, but high gives you the actual log lines for the top offenders. Luna: Good point. I think the next step for many listeners would be to pair Logwatch with a simple script that sends alerts if certain thresholds are hit — like more than 100 SSH failures. But that's a different tool. Lucas: Actually, Logwatch can do that via its own filtering. The scripts have a concept of 'threshold' — you can set a minimum count for an entry to appear. For example, in the sshd script, you can set $threshold = 5 to only show IPs with more than 5 attempts. Luna: I didn't know that. That's actually really useful for cutting noise. Where's that configured? Lucas: It's inside the individual service scripts, not in the main conf. So you'd edit /usr/share/logwatch/scripts/services/sshd and look for the line that says 'my $threshold' — usually near the top. Change it from the default 1 to something like 5 or 10. Luna: But you'd lose that on package upgrade, right? You'd need to copy the script to a custom location or use a conf override. Lucas: Exactly. The better approach is to create a custom service config in /etc/logwatch/conf/services/sshd.conf and set the threshold there using the Logwatch config syntax. That survives upgrades. The format is: $service_config{'threshold'} = 5; inside a Perl block. Luna: That's the kind of detail I love — real-world upgrade-safe config. Alright, I think we've given listeners a solid foundation. Any final thoughts? Lucas: Just this: if you're not getting daily log summaries, you're flying blind. Logwatch is the cheapest, easiest way to get a window into your server's overnight activity. Set it up today, and check the report tomorrow morning. You'll be surprised what you find.