Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server File Integrity Monitoring with AIDE
Transcript
- Lucas: You've patched your server, locked down SSH, firewalled everything — but how do you know if someone actually changed a critical system binary? Luna: That's the blind spot, right? You can't watch every file every second. Lucas: Exactly. And that's where file integrity monitoring — specifically a tool called AIDE — comes in. AIDE stands for Advanced Intrusion Detection Environment, and it's been around for decades. It takes a snapshot of your files — checksums, permissions, ownership, size — and then later compares the current state against that baseline. Luna: So it's like a tamper-detection system for your filesystem. Lucas: Precisely. Let's say someone compromises a web server and replaces /bin/ls with a trojan that hides their processes. AIDE will catch that because the checksum changed. Or if they modify /etc/passwd to add a backdoor user — AIDE will flag it. The key is you have to initialize the database when the system is in a known good state. Luna: Which brings us to the classic chicken and egg problem — how do you know your baseline is clean? Lucas: Right. Ideally you set up AIDE right after a fresh OS install or after you've hardened the server. You can also cross-check the database against package manager checksums — Debian's dpkg --verify, for example. But the practical workflow is: install AIDE, configure what to monitor, initialize the database, then run periodic checks. Luna: Let's walk through that. How do you install it? Lucas: On Debian or Ubuntu, it's apt-get install aide. On RHEL or CentOS, yum install aide. Once installed, the main config file is /etc/aide/aide.conf. You define rules — which directories and files to include, which attributes to check. The default config is pretty sensible, but you'll almost certainly want to customize it. Luna: What do you usually change? Lucas: First, you want to exclude noisy directories like /tmp, /var/cache, /proc, /sys — those change constantly and would generate false positives. Second, you might want to add specific critical files like /etc/shadow or /etc/ssh/sshd_config. Third, you can define different check levels — for binaries you might check sha256 and permissions, for log files maybe just size and modification time. Luna: So you can tune the sensitivity per path. Lucas: Exactly. AIDE has six default rule groups: Normal, Log, Data, All, etc. You can also define custom rules with attributes like p+i+n+u+g+s+m+c+md5+sha256 — that's permissions, inode, number of links, user, group, size, mtime, ctime, md5 hash, sha256 hash. For most critical binaries, you want the full treatment. Luna: After the config, you initialize the database. What's the command? Lucas: aideinit. That scans all the files you've defined and creates a database — by default at /var/lib/aide/aide.db.new. Then you copy that to aide.db so it becomes the active baseline. The initial scan can take a while on large filesystems — I've seen it run for hours on a big NAS server — but on a typical web server it's usually under a minute. Luna: And then you run checks with aide --check. Lucas: Right. AIDE compares the current filesystem against the database and outputs any changes. If nothing changed, you get a clean report. If something changed — say you updated a package — it'll list the file, the attribute that changed, and the old and new values. That's where you have to decide: is this a legitimate change or a potential intrusion? Luna: How do you handle legitimate updates? Because package upgrades change binaries all the time. Lucas: Good question. The standard approach is to run a check after any planned maintenance, verify that only expected files changed — ideally by cross-referencing with the package manager — and then update the baseline. You do that by running aideinit again and replacing the database. Some teams script this into their deployment pipeline: after an update, run aide --check, if everything's clean, copy the new database. Luna: What about automating the daily check? You don't want to run it manually every morning. Lucas: Absolutely. The most common method is a systemd timer or a cron job. On modern systems, I prefer systemd — you create a service unit and a timer unit. The service runs aide --check and outputs to a log file or sends an email. The timer triggers it daily at, say, 2 AM. Here's a minimal example: /etc/systemd/system/aidecheck.service with ExecStart=/usr/bin/aide --check, and a timer that says OnCalendar=daily. Luna: And if a change is detected, you want to know about it immediately, not just when you check logs. Lucas: Right. You can pipe the output to mail — AIDE has a built-in mail command, or you can use a wrapper script. I usually set it up to send the report to a dedicated security mailbox. Some folks integrate with Slack or PagerDuty via a script that parses the output for added or removed files. The key is that you need an alert that someone actually reads. Luna: What about performance impact? Running checksums on every file every day could be heavy. Lucas: It depends on the filesystem size and the number of files. For a typical LAMP server with maybe 50,000 files, a daily check takes maybe 30 seconds to a minute. For a file server with millions of files, it could be hours. In that case, you might split the database — monitor only critical directories daily, and the rest weekly. AIDE allows multiple databases and config files. Luna: Let's talk about a real-world scenario. Say you have a compromised web server — what does AIDE do for you? Lucas: Great example. Let's say an attacker exploits a vulnerable WordPress plugin, gets a shell, and drops a backdoor in /usr/lib/apache2/modules. If you have AIDE monitoring that directory with sha256 checksums, the next check will flag the new file or the modified checksum. You get a report showing /usr/lib/apache2/modules/evil.so — added, with a hash you've never seen. That's your cue to investigate, remove the file, and figure out how they got in. Luna: So it's not just about detection — it's about giving you a timeline. You know when the change happened. Lucas: Exactly. AIDE reports the timestamp of the last check and the current check. If you run it daily, you know the compromise happened within that 24-hour window. That narrows down logs and helps incident response. Luna: How does AIDE compare to Tripwire? That's the other big name in file integrity. Lucas: Tripwire is more feature-rich — it has a policy language, better reporting, and commercial support. But it's also more complex and requires a license for the enterprise version. AIDE is open source, simpler, and free. For most small to medium deployments, AIDE is perfectly adequate. Tripwire might be overkill unless you need pci dss compliance reporting or very granular policies. Luna: Speaking of compliance, does AIDE help with standards like PCI or HIPAA? Lucas: Absolutely. Both require file integrity monitoring for critical systems. AIDE can satisfy that — as long as you have documented procedures for baseline updates and alert response. Auditors will want to see that you're actually reviewing reports and taking action. AIDE gives you the raw data; the process around it is what makes it compliance-ready. Luna: One gotcha: if the attacker gains root access, they could modify the AIDE binary or the database itself. Lucas: That is the classic attack — you need to protect the database. Best practice is to store the database on a read-only filesystem, like a mounted cd rom or a remote syslog server. Or at least set permissions so only root can write to it. Some people sign the database with GPG and verify the signature before each check. If the attacker reinitializes the database, the signature would mismatch. Luna: So AIDE isn't a silver bullet — it's one layer. Lucas: Exactly. Defense in depth. AIDE should be part of a larger security stack including intrusion detection, log monitoring, and regular updates. But for a lightweight, open-source tool that can give you early warning of unauthorized changes, it's hard to beat. Luna: Let's talk about practical tuning. What's the best config for a busy web server? Lucas: For a typical LAMP stack, I'd monitor: /etc, /usr/bin, /usr/sbin, /usr/lib, /usr/local/bin, /var/www, /boot, and root's dotfiles. I'd exclude /tmp, /var/tmp, /var/cache, /proc, /sys, /run, and /dev. I'd use a rule with sha256 and all metadata for binaries, and a lighter rule with just md5 and permissions for config files. Here's a snippet: /etc Normal, /usr/bin FullCheck, where FullCheck is defined as p+i+n+u+g+s+m+c+sha256. Luna: What about database directories like MySQL or PostgreSQL? Those change constantly. Lucas: Right — you generally don't want to monitor database data files because they change on every write. Instead, monitor the binary and config directories. If someone replaces mysqld, you want to know. If they alter my.cnf, you want to know. The actual data changes are normal. Luna: Alright, I'm convinced. Let's set up AIDE on a test server. Lucas: Let's do it. Install AIDE, edit /etc/aide/aide.conf, run aideinit, then set up a systemd timer. We'll go through it step by step. Luna: This kind of practical, no-nonsense guidance is exactly why I value shows like this. It's ad-free and focused on real sysadmin skills. If you find these episodes useful, consider supporting the show at buy me a coffee dot com slash fexingo. It helps keep the content independent. Lucas: Absolutely. Listener support keeps us going — and lets us dive into topics like file integrity monitoring without any sponsors dictating the agenda. Thanks to everyone who chips in. Luna: So back to AIDE: first command is apt-get install aide, then we configure. Lucas: Right. Once installed, the default config is at /etc/aide/aide.conf. Open it and you'll see lines like 'All = p+i+n+u+g+s+m+c+md5+sha256'. You can define custom groups. For a start, I recommend uncommenting the lines that include key directories and adding exclusions. Then run 'sudo aideinit'. That creates /var/lib/aide/aide.db.new. Copy it to /var/lib/aide/aide.db. Then test with 'sudo aide --check'. Luna: And if everything looks good, set up the timer. Lucas: Create /etc/systemd/system/aidecheck.service with ExecStart=/usr/bin/aide --check and maybe a mail command. Then /etc/systemd/system/aidecheck.timer with OnCalendar=daily. Enable and start the timer. Now you have automated file integrity monitoring — a solid security layer that runs itself. Luna: One final question: how often should you update the baseline? After every package update? Weekly? Lucas: It depends on your change frequency. On a stable production server, I'd update the baseline after each planned maintenance window. On a dev server that changes daily, maybe weekly. The important thing is to verify the changes before accepting them. AIDE gives you the power to know exactly what changed — use that power. Luna: Good advice. Thanks Lucas, this was a practical deep dive. Lucas: Thanks Luna. AIDE is one of those tools that once you set it up, you wonder why you didn't sooner. It's peace of mind for your servers.