Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Auditd for Server Security Monitoring
Transcript
- Lucas: I want to talk about a tool that often flies under the radar for a lot of sysadmins, but it is baked right into the Linux kernel — auditd. Luna: Auditd — the Linux Audit framework. I feel like I know it exists, but I rarely configure it from scratch. Where does it fit in the security stack? Lucas: Right — it is not something you touch every day, but when you need to know who accessed a sensitive file or what command ran at a precise moment, auditd is the answer. It is a system-wide event logging system that can track system calls, file accesses, user logins, and more. Luna: So it is different from something like syslog, which is more about application-level messages, right? Lucas: Exactly. Syslog is great for application logs, but auditd lives at the kernel level. It catches low-level events that applications might never report. For compliance frameworks like PCI DSS, it is often required to have something like auditd in place. Luna: Okay, so how do you get started? Is it a service you install, or is it already there? Lucas: On most modern distributions — Ubuntu, RHEL, Debian — auditd is either pre-installed or one apt or yum away. The package is called auditd. Once installed, the service runs as a daemon and loads rules from /etc/audit/audit.rules. Luna: And those rules define what events to watch? Lucas: Exactly. You write rules using auditctl, or you edit the rules file directly. For example, if you want to monitor changes to /etc/passwd, you would add a rule like: -w /etc/passwd -p wa -k passwd_changes. That means watch the file, log on write and attribute change, and tag it with the key passwd_changes. Luna: And the -k key is for grouping events in logs, so you can search later? Lucas: Yes. The key is critical when you have dozens of rules. Later you can search with ausearch -k passwd_changes. Without keys, you are digging through thousands of raw events. Luna: What about tracking command execution? Can you log every command a user runs? Lucas: You can, but carefully. There is a rule that watches execve system calls. Something like: -a always,exit -S execve -k command_execution. That logs every command executed by any user. But on a busy server, that can generate gigabytes of logs per day. Luna: So you need to be selective. What is a more practical approach? Lucas: I would start with critical files: /etc/passwd, /etc/shadow, /etc/ssh/sshd_config, maybe your web server configuration. Then add rules for specific users if needed — like monitoring root's activities. You can filter by user ID with -F uid=0. Luna: That makes sense. And once the logs start flowing, how do you actually read them? The raw log format is not the friendliest. Lucas: That is where ausearch and aureport come in. Ausearch lets you query logs by time, key, user, system call, and more. For example, ausearch -ts today -k passwd_changes shows all events from today tagged with that key. Aureport gives you summary reports: like a list of all commands run by a user in the last hour. Luna: So you can do forensic analysis after a breach or during an audit. Lucas: Exactly. And that is where auditd shines — it gives you a reliable, kernel-verified record. Unlike bash history, which a savvy attacker can clear, auditd events go straight to the kernel and cannot be easily tampered with if you secure the logs. Luna: Where should you store the logs? Locally on the server? Lucas: Best practice is to ship them off-box. Auditd can forward events to a remote syslog server using audispd — the audit dispatcher plugin. You configure /etc/audisp/plugins.d/syslog.conf to enable forwarding. Then you can centralize logs from all your servers into something like Elasticsearch or Splunk. Luna: And that helps with compliance too — PCI DSS requirement 10 requires log reviews and central storage. Lucas: Exactly. But you have to be careful about log size. Audit logs can grow fast. I have seen servers fill up /var/log because audit.log hit 20 gigabytes overnight from an overly broad rule. Luna: So you need to tune rules and set log rotation. Is that handled automatically? Lucas: Auditd comes with its own rotation — the logrotate.conf for audit is in /etc/audit/auditd.conf. You can set max_log_file, max_log_file_action, and num_logs. But many people also use system logrotate on /var/log/audit/audit.log. Luna: I want to circle back to something you mentioned earlier — the difference between auditd and SELinux. They both deal with security, but they are not the same, right? Lucas: Right. SELinux is about mandatory access control — it can block an action before it happens. Auditd is purely a logging mechanism. It does not block anything. But they work together: SELinux denials are logged via auditd. So when you see an AVC denial in audit.log, that is SELinux talking through auditd. Luna: So auditd is the eyes, SELinux is the gatekeeper. Lucas: That is a great way to put it. And because auditd does not block, it is safe to enable on production systems without worrying about breaking things. You just have to manage the log volume. Luna: What about performance impact? Does monitoring every system call slow the server down? Lucas: It depends on the rule. Watching execve on a busy application server can add overhead because every command triggers a kernel event. But watching file writes on a few config files is negligible. I would benchmark on a staging server before rolling out broad rules. Luna: Good advice. So if someone wants to start using auditd today, what is the minimum viable configuration? Lucas: Install auditd, make sure the service is running, then add a few watch rules for critical files. Use -k keys to tag them. Then verify with ausearch. That is your starting point. Gradually add more rules based on what you need to monitor. Luna: And check the logs regularly? Or set up alerts? Lucas: Automate alerts. Use something like auditd's built-in plugin or a custom script that tails audit.log and triggers on specific keys. For example, if someone modifies /etc/shadow, you might want an email or a Slack notification. Luna: That is a solid plan. And it keeps you audit-ready for compliance without the overhead of heavy SIEM tools. Lucas: Exactly. Auditd is one of those tools that, once you have it set up, gives you peace of mind. You have a record of what happened, and you can prove it to auditors or investigators. Luna: That is exactly the kind of practical insight that makes this show valuable. And if listeners find these deep-dive episodes useful, supporting the show is a great way to keep them ad-free and coming. Lucas: Yeah, we hear that a lot. If today's conversation gave you something you can apply on your servers, and you want to help us keep producing these episodes without ads, you can support us at buy me a coffee dot com slash fexingo. Luna: It really helps us cover our hosting costs and research time, and it keeps the show independent. No pressure, but if it is something you have been thinking about, that is the link. Lucas: And back to auditd — one more thing I want to mention is the auditd daemon's ability to send logs to a remote server using the audisp-remote plugin. That is critical if you want to centralize logs and protect them from local tampering. Luna: So you configure /etc/audisp/audisp-remote.conf with the remote server address and port. Lucas: Yes. And you need to ensure the remote syslog server is configured to accept logs. It is common to use rsyslog with TCP input. That way, even if the local server is compromised, the logs survive elsewhere. Luna: That is a key part of a defense-in-depth strategy. Lucas: Absolutely. And finally, if you ever need to test your audit rules without waiting for real events, you can use the auditctl -l command to list loaded rules, and trigger events manually — like touching a watched file — then run ausearch to see if they were captured. Luna: That is a good troubleshooting tip. So the workflow is: define rules, load them, test with a trigger, verify with ausearch, and then automate log review. Lucas: Exactly. And once you get comfortable, you can even write rules that monitor network connections using socket system calls, or track changes to binary files. Auditd is incredibly flexible. Luna: It sounds like a must-have for any production environment that takes security seriously. Lucas: I think so. It does not replace other tools, but it fills a gap that nothing else does — kernel-level accountability. And it is free, open source, and already on your system. Luna: Great episode. Next time we can talk about integrating auditd with modern SIEMs like Wazuh or Elastic. Lucas: That would be a good follow-up. For now, try adding one audit rule this week and see what you discover.