Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server SSH Hardening with Security Keys
Transcript
- Lucas: If today's tech conversation gave you something usable — a tip you’ll actually apply on your servers — I want to mention something behind the scenes. Luna: Yeah? Lucas: This show stays ad‑free because a small group of listeners chips in monthly through buy me a coffee dot com slash fexingo. That’s genuinely what makes it possible to produce this many deep‑dive episodes without sponsors. Luna: And it means we answer to you, not to an advertiser's timeline. If Fexingo has helped you save time debugging or avoid a late‑night outage, that’s exactly where to put a few bucks. Lucas: All right — back to SSH hardening. Today I want to walk through moving your Linux server's SSH authentication from plain SSH keys to hardware‑backed FIDO2 security keys. Luna: Hardware keys — like the little USB‑A or USB‑C dongle that you tap to authenticate? Lucas: Exactly. YubiKeys, Google Titan keys, SoloKeys — they all speak the FIDO2 protocol. When you generate an SSH key backed by one of these, the private key material never leaves the device. An attacker who compromises your workstation can’t steal the key and reuse it from a different machine. Luna: So even if someone gets remote access to your laptop, they still need physical possession of the token to authenticate. Lucas: Right. And that’s a massive step up from the default SSH key approach, where the private key sits on disk — often encrypted only by a passphrase that can be brute‑forced if the hash is weak. Luna: What do you need on the server side to support that? Lucas: First, OpenSSH version 8.2 or later on both client and server. That was released in early 2020, so most modern Linux distributions have it. You’ll also need the `libfido2` library installed on the client machine. Luna: And the key generation step — is that different from `ssh‑keygen -t rsa`? Lucas: It’s similar but with a different key type. You run `ssh‑keygen -t ed25519-sk -O resident -C “my‑server‑key”`. The `-sk` suffix stands for ‘security key.’ The `-O resident` flag stores the key on the token itself, so you can plug it into another machine and load the key without re‑generating. Luna: Wait — that means you don’t have to copy the private key file between workstations. You just plug in the token and it’s available. Lucas: Exactly. The public key gets written to `~/.ssh/id_ed25519_sk.pub` as usual. You copy that to the server’s `~/.ssh/authorized_keys` file. But the private half is hardware‑locked. Luna: Does the server need any special configuration to recognize that public key type? Lucas: It should work out of the box if the server runs OpenSSH 8.2+. The public key format includes a `sk‑ecdsa‑sha2‑nistp256@openssh.com` prefix or `sk-ssh-ed25519@openssh.com`. The server checks the signature during authentication — the client proves it has the hardware token by having the device sign a challenge. Luna: What about the user experience? Every time you SSH, you need to tap the key? Lucas: Yes — and that’s both the security and the slight friction. When you initiate the SSH connection, your local system prompts you to touch the token. The token lights up or blinks. A quick tap, and the connection proceeds. If you’re using agent forwarding, you might need to tap once per session, not per command. Luna: That sounds manageable for a sysadmin who’s SSH‑ing into a handful of servers a day. But what if you manage a hundred boxes? Lucas: That’s where `ssh_config` can help. You set `IdentityFile` to point at the hardware key’s public key, and you can even use `Match` blocks to apply it only to certain hosts. Some admins use a separate SSH key for their jump boxes and another for internal hosts. Luna: And if you lose the hardware token — are you locked out for good? Lucas: You need a backup plan. Most security tokens allow you to generate a second key on a separate device — store that public key in `authorized_keys` as well. Or you can keep a fallback SSH key on an encrypted USB drive stored in a safe. The important thing is to never have only one authentication method. Luna: So you add both the hardware‑backed public key and a conventional ed25519 key to the server’s authorized keys, then test that each works before disabling password auth. Lucas: Exactly. And once you’ve confirmed that your hardware key authentication works, you edit `/etc/ssh/sshd_config` and set `PasswordAuthentication no`, `ChallengeResponseAuthentication no`, and `AuthenticationMethods publickey`. Then restart sshd. Luna: Don’t forget to keep an active SSH session open while you test the new config, so if something breaks you can revert without getting locked out. Lucas: That’s Rule Number One. Open a second terminal, SSH in with the new config, and only close the first session after you confirm everything works. I’ve seen too many people skip that step and end up at the data center with a crash cart. Luna: Let’s talk about the ‘resident key’ feature a bit more. You mentioned `-O resident` during key generation. What exactly does that do? Lucas: A resident key stores the key pair on the token’s internal storage. You can then use `ssh-add -K` to load the key into your SSH agent from any machine where you plug in the token. Without resident, the private key is still in the token, but you need to have the generated private key file on disk to reference it. Luna: So resident keys are better for mobility — you can use any laptop as long as you have your token. Lucas: Right. The trade‑off is that the token has limited storage — often only 25 or so resident key slots. So you might reserve those for your most critical infrastructure. Luna: What about compatibility with older SSH clients? If you have a legacy server that’s stuck on OpenSSH 7.x, can you still use a hardware key? Lucas: Not directly — the server needs to understand the `sk-` key types. But you can work around it by using a jump box that runs modern OpenSSH. The jump box authenticates with the hardware key, then forwards the connection to the legacy server using a conventional key stored only in memory. Luna: That’s a clean architecture: the hardware key protects the perimeter, and the inner network uses short‑lived keys. Lucas: Exactly. And if you’re managing multiple servers, you can automate the public key distribution with Ansible or similar tools. Just treat the hardware key’s public key as a variable and push it to each server’s `authorized_keys`. Luna: Any gotchas with SSH agent forwarding when using hardware keys? Lucas: Agent forwarding works as usual — you add the key to your local agent, then use `ssh -A` to forward the agent socket. The remote server can then use your local agent to authenticate further hops, but each time it needs the hardware key, it prompts your local token for a touch. That can be a bit jarring if you’re hopping through three machines. Luna: Word to the wise: if you’re using SSH config with `ForwardAgent yes`, be aware that anyone with root on the intermediate box can potentially use your agent socket. Hardware keys reduce the risk of key theft, but agent forwarding still exposes the agent. Lucas: Good point. Some teams use SSH certificates instead of forwarding to mitigate that. But that’s a topic for another episode. Luna: So to sum up the hardening checklist: generate a `ed25519-sk` key, copy the public key to the server, test authentication, set up a backup key, then disable password auth. Lucas: And consider using resident keys if you move between workstations often. Also, think about the physical security of the token — it’s small and easy to lose. Some tokens support PIN protection, so even if someone picks it up, they can’t use it without the PIN. Luna: That’s a nice additional layer. The token itself can require a PIN before it releases the key for signing operations. Lucas: Right. You enable that with `ykman` or the token’s management tool. Then when you SSH, you first enter the PIN, then tap the device. That’s two factors: something you know and something you have. Luna: For a sysadmin who already uses SSH keys, moving to hardware‑backed keys is a relatively small operational change with a big security payoff. Lucas: Absolutely. And the hardware is cheap — a decent FIDO2 key costs around $25 to $50. Compared to the cost of a breach, it’s a steal. Luna: One last question: how do you handle key rotation if the token breaks or you want to replace it? Lucas: You’d generate a new key on the new token, distribute the new public key to all servers, and then remove the old public key. If you have a configuration management system, it’s a matter of updating a variable and running the playbook. Without automation, you have to touch each server manually — another reason to invest in automation early. Luna: Alright — I think we’ve given listeners a concrete path to a more secure SSH setup. Lucas: If you try it, pay attention to the backup plan first. That’s the part most people overlook. Then the rest is straightforward.