Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Auditing with Auditd for Security Compliance
Transcript
- Lucas: So you're managing a Linux server, and one day you notice that a user's privilege level changed — but you're not sure when or how. That's exactly the kind of gap auditd is built to close. Luna: And by auditd, you mean the Linux Audit Daemon — the built-in framework for logging system calls, file accesses, and user actions, right? Lucas: Exactly. It's part of the kernel and has been around for years. Unlike something like Falco, which we covered in episode 82 and is more about runtime threat detection, auditd is the go-to for forensic-grade logging. It's what you'd rely on for a compliance audit. Luna: So let's say I want to know who modified /etc/shadow last Tuesday at 3 PM — auditd can tell me that? Lucas: Yes, if you set a rule to watch that file. And the great thing is, auditd doesn't just log the event — it captures the exact command, the user ID, the timestamp, and even the syscall used. It's incredibly granular. Luna: Alright, let's walk through the setup. First, I assume you need to install auditd? Lucas: On most distributions it's straightforward. On Ubuntu or Debian, it's 'apt install auditd'. On RHEL or Rocky Linux, it's 'dnf install audit'. Once installed, the daemon starts automatically and you can check its status with 'systemctl status auditd'. Luna: And then you define rules? Where do those live? Lucas: The main rule file is /etc/audit/rules.d/audit.rules. You write rules using the auditctl command syntax, and they persist across reboots. But during testing, you can also add rules live with 'auditctl -w /path/to/file -p rwxa -k key_name'. The -w flag means watch, -p specifies permissions, and -k lets you assign a key for easier filtering later. Luna: So if I want to monitor /etc/shadow for any write or attribute change, I'd run something like 'auditctl -w /etc/shadow -p wa -k shadow_watch'? Lucas: Exactly. And you can test it by modifying the file — even just touching it — and then checking the logs with 'ausearch -k shadow_watch'. That command will pull up every event tagged with that key. Luna: Nice. But what about monitoring commands themselves? Say I want to log every time someone runs sudo? Lucas: That's a classic use case. You can use a syscall rule. For example, 'auditctl -a always,exit -F arch=b64 -S execve -F uid=0 -k sudo_exec' will log all execve syscalls made by root. But be careful — that can generate a lot of noise. I'd recommend narrowing it down with filters like '-F exe=/usr/bin/sudo' to only log when sudo itself is executed. Luna: Makes sense. And once you have those rules, you can generate reports with aureport, right? Lucas: Right. Aureport is the reporting tool. For instance, 'aureport -x' gives you a summary of executable runs. 'aureport -f' shows file access summaries. And 'aureport -u' gives you user login summaries. It's great for a daily compliance check. Luna: I've also used 'ausearch -i' to get interpretable output — that converts numeric UIDs to usernames and timestamps to human-readable dates. It's a lifesaver when you're digging through logs. Lucas: Absolutely. And you can combine ausearch with grep or jq if you're piping to JSON. But honestly, most sysadmins just need to know three commands: auditctl to add rules, ausearch to query events, and aureport to summarize. That covers 90 percent of auditing needs. Luna: I want to talk about a real scenario. A colleague of mine once caught a developer who was accessing the production database directly using a shared service account. They set up a watch on the MySQL binary and on the /etc/mysql directory, and auditd logged every single access. That's powerful. Lucas: It is. And that kind of audit trail is exactly what SOC 2 or pci dss auditors want to see. They want evidence that you're monitoring changes to critical files and that you can produce a report on demand. Auditd can do that without needing an expensive SIEM. Luna: Speaking of compliance, let's talk about immutable audit logs. Is there a way to prevent even root from tampering with the audit logs? Lucas: Yes, there is. You can set the audit subsystem to immutable mode by adding '-e 2' to your rules file. Once that's set, no one — not even root — can change audit rules or stop the daemon until the system is rebooted. And if you combine that with sending logs to a remote server, you have a tamper-proof audit trail. Luna: That's a solid practice. But you need to be careful — once you set immutable mode, you can't add or remove rules without a reboot. So make sure your rules are correct before enabling it. Lucas: Exactly. And that's why you test with '-e 1' first, which is locked mode but allows a reboot to clear it. Then '-e 2' is permanent until reboot. I usually set up all my rules, test them with ausearch, then add the immutable flag to the rules file and reboot to confirm. Luna: One thing I've always wondered: how does auditd compare to something like syslog or systemd-journald for security auditing? Lucas: The key difference is that auditd is syscall-level. It catches things that syslog never sees, like a process opening a file and then immediately closing it. Syslog is text-based and application-driven — it only logs what the application chooses to log. Auditd logs what the kernel does. That's why it's the standard for forensics. Luna: Right, and you can also feed auditd logs into systemd-journald via the audit socket, so you have a unified view. I've done that with 'journalctl _TRANSPORT=audit' to see audit events alongside other logs. Lucas: That's a smart setup. And if you're running a larger environment, you can forward auditd logs to a central log server using audisp, the audit dispatcher plugin. There's a built-in plugin for syslog, or you can write your own. Luna: Alright, so let's say I want to get started today. What's a minimal set of rules you'd recommend for a typical web server? Lucas: I'd start with a few key files: /etc/passwd, /etc/shadow, /etc/group, /etc/sudoers, and /etc/ssh/sshd_config. Watch them for write and attribute changes. Then add a rule to log all sudo executions. And for compliance, also watch /var/log itself — because if someone deletes logs, that's a red flag. Luna: That's a solid baseline. And you'd also want to set a rule to monitor changes to auditd's own configuration, right? Lucas: Great point. Always watch /etc/audit/ and /etc/audit/rules.d/. If someone tries to disable auditing, you'll know. And I'd add a rule to log any use of the auditctl command itself — that way you catch anyone trying to add or remove rules live. Luna: So with maybe ten rules, you have a decent auditing layer that can help you sleep better at night. Lucas: Exactly. And honestly, if today's conversation gave you something usable, that's the link — buy me a coffee dot com slash fexingo. It's a small thing that keeps this show ad-free and focused on practical content. Luna: Yeah, I think that's fair. We put a lot of work into making sure every episode has at least one concrete takeaway, and listener support makes that sustainable. Lucas: Alright, back to the nuts and bolts. One more thing I want to mention — auditd's default log file is /var/log/audit/audit.log. It's a text file, but it's not meant to be read directly. Always use ausearch or aureport to query it. Parsing raw audit logs is a pain. Luna: And it's important to manage log rotation. Auditd has its own rotation mechanism via the 'max_log_file' and 'num_logs' parameters in /etc/audit/auditd.conf. You can set a max size, say 50 megabytes, and keep five rotated logs. Lucas: Right. And if you're worried about disk space, you can also configure 'space_left_action' to email an admin or even shut down the system if logs fill up. That's a bit extreme, but for high-security environments, it's common. Luna: Let's talk about a gotcha I've hit: auditd can cause performance overhead if you log too many events. For example, logging every execve syscall on a busy server can slow things down noticeably. Lucas: That's true. The key is to be surgical with your rules. Watch specific files instead of entire directories. Use keys to organize events. And monitor the audit backlog with 'auditctl -s' — that shows you how many events are queued. If the backlog grows, you're logging too much or your disk can't keep up. Luna: And you can also increase the backlog limit with '-b' in auditctl. The default is 64, but you can set it to 8192 or higher if needed. Lucas: Good advice. Alright, let's also touch on how to use auditd for incident response. Suppose you suspect a breach. You can immediately add a rule to watch the attacker's suspected working directory or any suspicious binaries. Then use ausearch to see if they were executed. Luna: And if you have immutable mode enabled, you can't add rules on the fly — so you'd need to plan ahead. That's why I keep a set of 'incident response' rules in a separate file that I can load after a reboot if needed. Lucas: That's clever. You could even have a script that boots the system into a special audit mode with extra rules. But for most cases, having a solid permanent rule set is sufficient. Luna: One last thought: if you're in a containerized environment, auditd inside a container can be tricky because the kernel audit subsystem is shared. You might need to use capabilities or run the audit daemon on the host and monitor container processes via cgroups. Lucas: Right. Auditd is primarily a host-level tool. For containers, tools like Falco are more appropriate. But for the host itself, auditd is irreplaceable. It's lean, it's built in, and it's trusted by auditors worldwide. Luna: And it's free. No licensing costs, no agent to install. Just a few lines of configuration and you have enterprise-grade auditing. Lucas: Exactly. So if you haven't tried auditd yet, start with one file — maybe /etc/shadow — and see what you learn. You might be surprised at what's happening on your own servers.