Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Auditd for Server Security Monitoring
Transcript
- Lucas: So you've locked down SSH keys, hardened sudoers, and even set up SELinux — but if someone does get in, do you actually know what they touched? Luna: That's the million-dollar question. And honestly, most people don't, because they've never turned on auditd. Lucas: Exactly. Today we're talking about the Linux Audit subsystem — auditd, auditctl, ausearch, aureport. It's been in the kernel forever, but it's one of those tools that a lot of sysadmins know exists but have never actually configured. Luna: I think part of the problem is that it seems intimidating. There is a lot of documentation, and people are scared they're going to fill up their disk with logs. Lucas: That fear is legitimate. An unfiltered auditd can generate gigabytes of noise in minutes. But with a few targeted rules, you can keep it lean and actually useful. Let's start with the basics: auditd logs events that the kernel reports, based on rules you define. It's not a general-purpose logger like syslog — it's specifically for security-relevant stuff. File access, system calls, user changes. Luna: And it's separate from the kernel's normal logging, which is a good thing. You don't want your security audit mixed in with debug messages from the graphics driver. Lucas: Right. The key components are: auditd, the daemon that writes events to disk; auditctl, the command to add rules; ausearch, to query the logs; and aureport, to generate summary reports. For this episode, I want to drill into a specific use case: detecting unauthorized access to sensitive files like /etc/shadow or SSH private keys. Luna: That's a good one. Because if an attacker gets read access to /etc/shadow, they can try to crack password hashes offline. You want to know that happened. Lucas: Exactly. So let's build a rule. First, you need to make sure auditd is installed and running. On Debian-based systems, it's 'apt install auditd'. On RHEL, it's 'dnf install audit'. Then enable and start the service. Luna: And I'd add: make sure the audit log file is on a separate partition or at least not in /var/log if you have space concerns. The default is /var/log/audit/audit.log, but you can change it in /etc/audit/auditd.conf. Lucas: Good point. Now let's add a rule to watch /etc/shadow for any read access. You use auditctl with the -w flag for a file watch, -p for permissions, and -k for a key phrase you can search on later. Like this: 'auditctl -w /etc/shadow -p rwa -k shadow-watch'. Luna: The -p rwa means watch for read, write, and attribute change. You could also use 'a' for any access. I usually stick to rwa because execute on /etc/shadow doesn't make sense. Lucas: Right. Then if any process reads that file, auditd logs an event. You can see it with 'ausearch -k shadow-watch'. The output is quite verbose at first, but you learn to read it. You'll see the event type, timestamp, syscall, uid, and the path. Luna: One thing to note: by default, auditd logs the syscall and the path, but not the contents. It's an audit trail, not a data-loss prevention tool. It tells you that something happened, but not what exactly was read. Lucas: That's important. And if you want to monitor a whole directory, you can use '-w /etc/ssh -p rwa -k ssh config watch'. But be careful — if you watch a directory with too many files, the log volume can spike. I learned that the hard way when I watched /tmp and filled up /var in about ten minutes. Luna: Yeah, /tmp is a classic trap. So rule of thumb: only watch specific files, not broad directories unless you have a really good reason. And always test your rules on a non-production system first. Lucas: Now, to make rules persistent, you add them to /etc/audit/rules.d/ — typically a file like audit.rules. The syntax is the same as auditctl, but without the 'auditctl' command. For example: '-w /etc/shadow -p rwa -k shadow-watch'. Then restart auditd or run 'augenrules --load'. Luna: And you should verify that the rules are loaded with 'auditctl -l'. That lists all active rules. If you see your rule there, you're golden. Lucas: Now, let's talk about reading the logs. Ausearch is powerful but its output is not the friendliest. For example, running 'ausearch -k shadow-watch' gives you a series of records. Each record has fields like type, msg, uid, pid, etc. I typically use 'ausearch -i' to interpret numeric values like uid into human-readable names. Luna: The -i flag is a lifesaver. Because without it, you're looking at uid 1000 and wondering who that is. With -i, it shows 'luna' instead. Lucas: And if you want a quick summary, use aureport. 'aureport -x' shows a summary of all executable commands. 'aureport -f' summarizes file access. But my favorite is 'aureport -u' for user-related events. You can combine it with --start and --end to look at a specific time window. Luna: There's also the --summary flag that gives you a count of events by type. It's a good way to spot anomalies, like a huge spike in file access events in the middle of the night. Lucas: Now, one thing that trips people up: the audit log can rotate automatically. In /etc/audit/auditd.conf, you control max_log_file, num_logs, and max_log_file_action. The default is usually max_log_file = 8 and num_logs = 5, so you get about 40 MB of logs before old ones are deleted. For a production server, you might want to increase that. Luna: And if you don't rotate at all, you can set max_log_file_action to IGNORE, but then you risk filling the disk. I prefer to use a dedicated partition and set it to ROTATE. That way you always have the last N rotations. Lucas: Good. So let's pivot to a real scenario. Say you're a sysadmin and you get an alert that someone accessed /etc/shadow. You run 'ausearch -k shadow-watch -i' and see that the process was /usr/bin/sshd, uid 0. That's normal — SSH daemon reads shadow when someone logs in. But if you see uid 1001, which is a regular user, running a custom script? That's suspicious. Luna: And you can also see the timestamp and the session. If it's an interactive session, you can cross-reference with 'last' or 'who' to see who was logged in at the time. Lucas: Auditd also logs the parent process ID, so you can trace back to see how that user's session spawned the script. That kind of forensic trail is invaluable during incident response. Luna: And it's not just for breaches. Auditd is also useful for compliance. If you need to prove that no one accessed PCI or HIPAA data, you can generate reports from audit logs. Lucas: Absolutely. Some organizations even ship audit logs to a central log server using auditd's built-in remote logging feature or via audisp-remote plugin. That way, even if the server is compromised, the logs are off-box. Luna: But that adds complexity. For most small to mid-size deployments, local log rotation and occasional manual checks are fine. You just need to know what to look for. Lucas: Exactly. And speaking of being useful — if today's tech conversation gave you something usable, these episodes stay ad-free because of listener support. If that matters to you, you can toss a coffee our way at buy me a coffee dot com slash fexingo. Luna: Yeah, it genuinely helps keep the server lights on, no pun intended. We appreciate every bit of support. Lucas: Anyway, back to auditd. One more advanced tip: you can use auditctl to monitor syscalls directly instead of files. For example, '-S openat -S open -S creat' to watch file openings. That gives you even finer granularity, but it requires more syscall knowledge. Luna: Syscall auditing is powerful, but it generates a ton of events. I usually stick to file watches for 90% of use cases. But if you need to monitor a specific behavior like privilege escalation attempts, syscall rules are the way to go. Lucas: Right. And there's also the immutable flag. Once you set '-e 2' in your rules, auditd rules become immutable — they cannot be changed or removed until the system reboots. This is great for security because it prevents an attacker from disabling your audit rules. Luna: But the downside is that you can't add rules on the fly anymore. So you need to plan ahead. I'd only enable immutable mode in a hardened production environment after thorough testing. Lucas: Good advice. So to wrap up: auditd gives you eyes on your file system. It's not a replacement for intrusion detection, but it's a critical layer. Start small — pick three sensitive files, add rules, generate some test events, and learn to read the logs. You'll be surprised how much visibility you gain. Luna: And over time, you can expand to cover more files and maybe even syscalls. Just remember: auditd is a scalpel, not a chainsaw. Lucas: That's the perfect way to put it. Thanks, Luna. That's all for today. If you want to try this out, set up a test VM and play with auditctl. The documentation is good, but nothing beats hands-on experience.