Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Audit SSH Keys Before They Become a Liability
Transcript
- Lucas: Alright, let's talk about something that keeps sysadmins up at night — or at least should: SSH keys. Luna: Because every server you've ever set up still has that one key from a contractor who left three years ago? Lucas: Exactly. And it's not just one key. It's hundreds. I was helping a startup last month that had sixty production servers. They had over four hundred authorized keys spread across them. Some of those keys were tied to people who hadn't touched a terminal in two years. Luna: Four hundred. And they didn't even know? Lucas: No. Because SSH keys are invisible until someone checks. There's no expiry date on a public key file. The authorized_keys file doesn't send you a reminder. And most teams only think about keys when something goes wrong — a breach, an audit, or a termination that someone forgot to propagate. Luna: So where do you even start? Do you just grep through every home directory? Lucas: Pretty much. The first step is inventory. You need to know what exists before you can decide what to keep. The simplest way is a Bash one-liner that iterates over every user on a server with a home directory and cats their ~/.ssh/authorized_keys file. Lucas: Something like: for user in $; do echo "User: $user"; cat /home/$user/.ssh/authorized_keys 2>/dev/null; done. You pipe that to a file, and suddenly you have a full key inventory per server. Luna: But that only gives you the keys, not who they belong to. How do you map a key to a person? Lucas: That's the hard part. Most SSH key comments are useless — people leave the default or something like 'mykey'. So you have to cross-reference. One approach is to look at the last login time for each user. If a user hasn't logged in for six months and has an SSH key, that's a candidate for removal. Lucas: You can get last login from the last command or from /var/log/auth.log. But even better: track key fingerprints. When you generate a key, you get a fingerprint. Store those fingerprints in a central database — even a simple text file — alongside the employee's name and the date the key was issued. Luna: So you need a process at key creation time, not just at audit time. Lucas: Exactly. And that's where most teams fail. They set up key-based auth once, and then they never touch it again. The solution is to use SSH certificates instead of raw public keys. With certificates, you can set an expiry date. A certificate is valid for, say, ninety days. After that, the key doesn't work. Luna: So you force rotation. But that requires a certificate authority infrastructure. Lucas: It does. But it's simpler than you think. OpenSSH has built-in support for certificate authentication. You set up a CA key pair on a secure machine, sign user public keys with the CA, and configure each server to trust the CA's public key. Then you distribute signed certificates to users. Lucas: The certificate itself carries a validity interval. Once it expires, the user needs to re-authenticate with the CA to get a new one. That gives you a natural rotation cadence. And if someone leaves, you just stop signing their keys. Luna: That's way cleaner than manually removing keys from authorized_files. But what about existing keys? The ones already out there? Lucas: You need a one-time cleanup. I usually run a script that compares the current authorized_keys against a known-good list — a list of keys that have been explicitly approved. Any key not on that list gets commented out with a note in the file. That way, if something breaks, you can quickly uncomment it. Lucas: But you also want to check for keys with empty passphrases. If a private key has no passphrase and it's on a compromised machine, the attacker can use it immediately. You can check for weak keys using ssh-keygen -l -f ~/.ssh/id_rsa and looking at the bit length — anything below 2048 bits should be replaced. Luna: And there's a tool specifically for auditing SSH servers, right? I remember something called ssh-audit. Lucas: Yeah, ssh-audit is great. It probes the SSH server itself and tells you which key exchange algorithms, ciphers, and MACs are enabled. But it's more about server hardening than key management. For key auditing, I rely on custom scripts and the ssh-keygen -l command to list fingerprints. Luna: Let's step back. What's the single most impactful thing a small team can do this week to improve SSH key hygiene? Lucas: Disable password authentication entirely. If you have key-based auth working, set PasswordAuthentication no in sshd_config. That eliminates brute-force attacks on passwords. Then run the inventory one-liner I mentioned and remove any key you can't identify. Lucas: And while you're there, set MaxAuthTries to a low number — say three — to limit failed authentication attempts. Also consider setting ClientAliveInterval and ClientAliveCountMax to kill stale SSH sessions after a few minutes of inactivity. Luna: What about using SSH Agent Forwarding? I've heard that's a risk. Lucas: Huge risk. If you enable agent forwarding, a compromised intermediate server can use your agent to authenticate to other servers. Best practice is to never use agent forwarding. Instead, use a jump host or SSH ProxyJump. That way, you connect directly from your machine to the target without forwarding your keys through an intermediate server. Lucas: ProxyJump with -J is cleaner and more secure. You can even chain multiple jump hosts. And you can use the same public key infrastructure on the jump host as on your servers. Luna: So if I'm a sysadmin with fifty servers and no central key management, what's my six-month plan? Lucas: Month one: inventory all keys, disable password auth, remove unknown keys. Month two: set up a CA and issue certificates to all current employees. Month three: enforce certificate expiry — start with ninety days. Month four: implement a process for offboarding — when someone leaves, revoke their certificate immediately. Lucas: Month five: audit your servers again to make sure no one re-added password auth or old keys. Month six: automate everything. Use Ansible or a similar config management tool to distribute the CA public key and enforce sshd_config across all servers. Luna: Automation is key. Because manual processes always decay. Lucas: They do. And the worst part is that SSH keys are silent. You don't get a page when a key goes stale. You only find out during a breach post-mortem. By then, it's too late. Luna: That reminds me — if you found today's walkthrough useful, consider supporting the show. It keeps us ad-free and we can dig into topics like this every week. You can find us at buy me a coffee dot com slash fexingo. Lucas: Yeah, even a small contribution helps us keep the server bills paid and the research time covered. Seriously, it makes a difference. Luna: Alright, back to keys. Lucas, you mentioned certificate revocation — how does that work in practice? Lucas: You maintain a Certificate Revocation List, or CRL. OpenSSH supports the ssh-keygen -k operation to generate a KRL — a Key Revocation List. You publish that file to each server, and then any key or certificate listed in it is automatically rejected. Lucas: The KRL can revoke specific keys, certificates, or even entire CAs. And it's fast — the server checks the list on each connection. You can update the KRL via your config management tool, and the change takes effect immediately for new connections. Luna: So you combine certificate expiry with a CRL for instant revocation. That's a solid two-layer approach. Lucas: Exactly. Expiry handles the normal rotation. Revocation handles the abnormal — someone leaves, a key is compromised, a laptop is stolen. Between the two, you have a manageable key lifecycle. Luna: And it all starts with that first inventory. So I guess the homework for listeners is: run the one-liner this week. Lucas: Run it. Save the output. Then start cleaning. Your future self — and your security team — will thank you.