Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Audit Logs for Incident Response
Transcript
- Lucas: So you get an alert from your intrusion detection system — something about an SSH key being added to the authorized_keys file on one of your production servers. Your pulse goes up, you SSH in, and you see a new entry. Now what? Luna: That's the moment where audit logs go from 'nice to have' to 'I really hope I configured this right.' Lucas: Exactly. And most sysadmins I know have a love-hate relationship with auditd. It's powerful, but the default config drowns you in noise. Today I want to walk through how you actually use audit logs for incident response — not just collecting them, but querying them under pressure. Luna: Alright, so let's ground this. We're on a standard Linux server — say Ubuntu 24.04 or RHEL 9. The attacker added an SSH public key to /home/deploy/.ssh/authorized_keys. How do you set up auditd to catch that? Lucas: First, you write a rule. The key here is to watch that file with the 'w' permission — write access. You do that with auditctl or by dropping a file in /etc/audit/rules.d. Something like: '-w /home/deploy/.ssh/authorized_keys -p wa -k ssh key change'. The -k flag gives it a key name so you can filter later. Luna: Right, the key name is your friend. Without it, you're grep-ing through thousands of events hoping to spot the needle. Lucas: Exactly. So the rule is live, and the attacker adds their key. Auditd logs an event with type=SYSCALL, a syscall number, the UID, the PID, and the file path. But here's the thing — the raw log is cryptic. You need ausearch and aureport to make sense of it. Luna: So ausearch is your search tool. Give me the command you'd run first. Lucas: I'd run: 'ausearch -k ssh key change -ts today -i'. The -i flag interprets numeric values like UIDs into names. That gives you a list of all events with that key from today. But in an incident, you want the timeline, not just the raw events. That's where aureport shines. Luna: What does aureport give you that ausearch doesn't? Lucas: Aureport can summarize. You run 'aureport -f -i -k ssh key change' and it tells you which file was accessed most, what operations, and the time range. For incident response, I usually start with 'aureport -x --summary' to see all executed commands, then narrow down. Luna: Okay, so you've identified the event. Now you need to figure out who did it. The log gives you a UID — but that could be a compromised account. What else do you look for? Lucas: The PID and the parent PID. If the attacker came in via SSH, the parent process is sshd. You can trace back: ausearch -p <PID> gives you everything that process did. In our scenario, the attacker added the key and maybe ran a few commands. You reconstruct the session. Luna: That's the theory. But in practice, I've seen logs where PIDs get reused — especially on busy servers. You could get false positives. Lucas: Great point. PID reuse is real. That's why you also record the session ID or use audit's 'loginuid' — that's a special field that tracks the login UID across process changes. The loginuid doesn't change even if the user switches accounts with su or sudo. It's the golden thread. Luna: So you'd filter by loginuid rather than just UID. That's a pro tip. What about the actual key content? Auditd doesn't log file contents by default, right? Lucas: Right. Auditd logs the fact that the file was modified, not the content. To capture the content, you'd need something like ima — Integrity Measurement Architecture — or a file integrity checker like AIDE. But for incident response, knowing when and by whom is often enough. You can then check the authorized_keys file itself for the new key. Luna: So you've identified the UID, the loginuid, the time, the parent process. What's next — do you pivot to other logs? Lucas: Absolutely. Audit logs are one piece. You correlate with auth.log for SSH login times, with syslog for any commands the attacker ran, and maybe with netstat or connection tracking for outbound connections. But audit gives you the file-level proof. In a post-incident report, that audit trail is gold. Luna: Let me ask about log noise. A lot of sysadmins avoid auditd because they tried it once and their logs filled up with cron jobs touching files. How do you manage that? Lucas: Two things. First, be specific with your rules. Don't watch entire directories if you can watch specific files. Second, use 'exclude' rules. For example, you can exclude events from certain UIDs — like the root user's cron jobs — if they're not relevant. The syntax is '-a exclude,always -F uid=0' but that's too broad. Better to exclude specific syscalls for known-good processes. Luna: Or you can increase the log buffer size and rotate logs faster. But the real fix is tuning rules over time. Lucas: Exactly. You start broad and then carve out exceptions as you see the noise. After a few weeks, you have a tailored set. And remember, you can have multiple rule files in /etc/audit/rules.d/ — one for security-critical files, one for application files, one for system calls. Luna: That modular approach makes it easier to troubleshoot. If you see unexpected events, you can disable one file at a time. Lucas: Right. Now let's talk about a common mistake: not auditing the audit logs themselves. If an attacker gains root, they can clear the audit log — 'auditctl -e 0' disables auditing. So you want to send logs to a remote syslog server or use audit's built-in remote logging with audisp. Luna: You need a separate log collector, otherwise you lose the evidence. That's basic but often overlooked. Lucas: The audisp plugin 'audisp-remote' can forward events over TCP or UDP to a central log host. But that's a whole setup on its own. For smaller setups, even just a cron job that runs 'ausearch --start yesterday --end today -r' and emails the output can work. Luna: And if you're on a tight budget — like a homelab or a small business — that cron approach is better than nothing. Lucas: That actually brings up something I want to mention. Shows like this one, where we get into the nitty-gritty of server administration, take a fair bit of research and scripting to put together. A couple of dollars a month from listeners who find these practical walkthroughs useful genuinely makes a difference — it's what keeps the podcast ad-free and focused on real-world scenarios. If you've gotten value out of today's episode or any of the past 69, you can support the show at buy me a coffee dot com slash fexingo. No pressure, just if it's been helpful. Luna: Yeah, and that support lets us keep experimenting with topics like auditd deep dives instead of repeating the same basics. Every contribution helps. Lucas: So back to incident response. Once you've collected the audit events, how do you present them to a manager or a compliance auditor? They don't want raw logs. Luna: That's where aureport's summary mode comes in, but also formatting. You can use 'aureport -f -i --summary' to get a csv like output, then import into a spreadsheet. Lucas: And you can customize aureport with '--node' to show hostname, or '--success' to filter. For a timeline, I run: 'aureport -t -i' which shows each event with a timestamp. Then I sort by time and look for clusters around the incident window. Luna: One tool I've started using is 'auditbeat' from Elastic — it ships audit logs to Elasticsearch and gives you a nice dashboard. But that's a heavier stack. Lucas: Right, and if you're in a rush, command-line tools are faster. No waiting for Kibana to load. Let's walk through a concrete example. Say the attacker added a key at 10:03:15 AM. You run 'ausearch -k ssh key change -ts 10:00 -te 10:10 -i'. You see one event with UID 1005, loginuid 1002, and PID 3042. Luna: Then you check who was logged in at that time — last, who, or 'ausearch -ua 1002' for all events by that loginuid. Lucas: Exactly. And you might find that loginuid 1002 is actually a service account that should never have interactive SSH access. That's your red flag. You then check if the SSH session is still active and kill it. Luna: So audit logs told you the file, the time, the user, and the process. That's 90% of the investigation right there. Lucas: The other 10% is understanding the attacker's actions after the key was added. Did they run sudo? Did they access other files? You can follow the PID chain. Use 'ausearch -p 3042 -i' to see all syscalls from that process. If the key addition was from an SSH session, the parent PID — say 3010 — is sshd. You can then check what that sshd process did: 'ausearch -pp 3010 -i'. Luna: And you can go up the chain all the way to the initial login. That's powerful. Lucas: One caveat: on a heavily loaded server, the audit log can get huge quickly. You need to set max log file size and number of rotated files in /etc/audit/auditd.conf. I typically set max_log_file = 100 and num_logs = 5, then use 'space_left_action = email' and 'admin_space_left_action = halt' to prevent the disk from filling up. Luna: Halt is drastic, but if audit logs are full, you might miss critical events. Some people prefer 'rotate' instead. Lucas: True. For production, I'd recommend 'rotate' and monitor disk space separately. The halt option is for compliance-heavy environments where missing even one event is worse than a server going down. Luna: Fair. So we've covered setting rules, querying with ausearch and aureport, following the PID chain, and remote logging. What's one thing you wish more sysadmins knew about auditd? Lucas: That you can test rules without committing them. Use 'auditctl -l' to list current rules, and 'auditctl -R <file>' to reload. But if you're testing, use 'auditctl -a' to add a rule temporarily — it won't survive a reboot. That way you can verify it catches the right events before making it permanent. Luna: Good tip. And the -D flag deletes all rules — handy for resetting during testing. Lucas: Right. So next time you get an alert about an unauthorized SSH key, you know exactly where to start. Set up those watches today, before you need them. Future you will be grateful. Luna: And maybe test a simulated attack in a lab environment first. That's how you build confidence. Lucas: Absolutely. Thanks for listening, and we'll be back next week with another deep dive.