Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your Linux Server Needs a Dedicated Bastion Host
Transcript
- Lucas: Alright, let's talk about something that probably sounds like extra work but actually saves you a ton of headache: the dedicated bastion host. Luna: You mean like a jump box? A single server that all your SSH traffic routes through, rather than letting people connect directly to production boxes? Lucas: Exactly. And I want to start with a concrete number. I've seen audits where companies with two hundred servers had over four thousand open SSH ports exposed to the internet. Every one of those ports is a potential entry point. A bastion collapses that to one. Luna: Four thousand ports. That's insane. And I'm guessing most of them are running default sshd config too. Lucas: Root login enabled, password authentication turned on, maybe even SSH protocol one still active on some legacy boxes. It's a playground for bots. A dedicated bastion isn't just about security theatre — it actually reduces your blast radius. Luna: So how do you set one up without making life miserable for the team? I've seen bastions that become bottlenecks because they're under-provisioned or the config is too restrictive. Lucas: Right. The key is to keep it boring. A bastion should be a minimal Linux install — we're talking a $5 or $10 a month VPS, maybe even a Raspberry Pi if you're on a tight budget and the traffic is low. The only public service it runs is SSH on a non-standard port, and it's hardened to the gills. Luna: What hardening steps are you talking about specifically? I want to hear the checklist. Lucas: First, disable root login. Second, password authentication off — key-only. Third, install and configure fail2ban. Fourth, consider port knocking if you're really paranoid. But the biggest win is audit logging. You run auditd on the bastion and log every single SSH session — who connected, when, what they ran, and how long they stayed. Luna: That audit trail alone is worth it for compliance. SOC2, PCI, HIPAA — they all want to see who touched production and when. Lucas: Exactly. And here's the neat part: with SSH agent forwarding, your users don't need to store private keys on the bastion. They connect with their local key, the bastion forwards the agent, and they hop to the target server. The bastion never holds a key. Luna: But agent forwarding is risky if someone else has root on the bastion — they could hijack the agent socket. How do you mitigate that? Lucas: You lock down the bastion. No one gets root except the dedicated admin account. You also use the -a flag to forward only specific keys, or you use SSH certificates so keys are time-limited. Another approach is to use ssh -J instead of SSHing into the bastion interactively — that way the agent doesn't stay on the bastion at all. Luna: So the jump host option is basically transparent to the user? They just ssh -J bastion.example.com target-server and they're in? Lucas: Exactly. The client handles the connection chaining. The bastion sees the connection but doesn't get an interactive shell session unless needed. That's the modern best practice for small to medium fleets. Luna: I like that. Now, what about the argument that a bastion is a single point of failure? If it goes down, nobody can reach production. Lucas: Valid concern. But you can run two bastions with a floating IP or a DNS round-robin, or you can set up a VPN as an alternative path. The cost of that redundancy is still far lower than securing every server individually. And honestly, if your bastion goes down because of a DDoS, your production boxes might be next anyway. Luna: Good point. Let's talk about performance. Does routing all SSH through a single box cause noticeable latency? Lucas: In practice, no. SSH adds maybe a few milliseconds per hop. For interactive use, you won't feel it. For automated scripts or rsync, you might see a tiny increase, but it's usually dwarfed by network latency between data centers. We're talking single-digit milliseconds. Luna: What about the setup overhead? If I have a hundred servers, do I need to change each one's sshd config? Lucas: Yes and no. You update each server's firewall to only allow SSH from the bastion's IP. That's a one-line iptables rule or a security group update. Then you push a config management tool like Ansible to ensure sshd is locked down and keys are distributed. It's a few hours of work for a hundred nodes. Luna: And you can automate the bastion itself with the same tool. Keep it in a Git repository so any team member can review the config. Lucas: Exactly. And that's actually a nice segue. If today's conversation about securing your infrastructure gave you something usable — maybe you're thinking about setting up your first bastion, or you just learned a new hardening trick — then you might appreciate what listener support makes possible. Luna: Yeah, this show stays ad-free and focused on practical, no-fluff content because of people who chip in. It's a small way to keep these conversations going. Lucas: If you'd like to help, you can find us at buy me a coffee dot com slash fexingo. That's it — one link, no pressure. Every contribution goes straight into keeping the server lights on and the research time available. Luna: And we really appreciate it. Now, back to bastions — let's talk about the actual sshd config. Lucas, what's the first line you'd put in the config? Lucas: PermitRootLogin no. Non-negotiable. Then PasswordAuthentication no. Then I set AllowUsers to a specific user or group — bastion-users, something like that. Then I configure ClientAliveInterval and ClientAliveCountMax to drop idle connections after, say, 15 minutes. Luna: And for logging? What auditd rules do you recommend? Lucas: I watch the execve system call for all users. That logs every command. But you have to be careful — if you log everything, logs can grow fast. I filter out common noise like ls or ps from non-interactive sessions. The goal is to log when someone actually changes something: editing files, running scripts, touching databases. Luna: Right. And you want to ship those logs off the bastion immediately — to a central log server or an SIEM. Otherwise an attacker who compromises the bastion can delete the evidence. Lucas: Absolutely. Central logging is a must. I use rsyslog with TLS to forward to a dedicated log host that has its own bastion access control. That way even if the bastion is pwned, the logs are safe. Luna: What about SSH keys management? How do you handle employee turnover? Lucas: You use SSH certificates. Issue a short-lived certificate — say 24 hours — that users get from an internal CA when they authenticate. When someone leaves, you just revoke the CA cert or stop issuing new ones. You don't have to touch every server's authorized_keys file. Luna: That's the real pro tip. Certificates scale so much better than raw keys. And they integrate with your identity provider — Active Directory, LDAP, whatever. Lucas: Right. And for the bastion itself, you can use the same certificate approach. The bastion just needs to trust the CA, and all target servers trust the same CA. It's elegant. Luna: Let's talk about monitoring the bastion. What metrics do you watch? Lucas: CPU and memory are trivial — a well-tuned bastion barely breaks a sweat. What I watch is connection count, failed authentication attempts, and outbound connection patterns. If someone on the bastion suddenly starts connecting to an unusual external IP, that's a red flag. Luna: And you can use tools like netstat or lsof to see live connections. Or set up a simple cron job that diffs the connection list every minute. Lucas: Even better: use osquery to run SQL queries against the system state. You can write a query like 'select * from process_open_sockets where remote_address not like '10.%' and remote_address not like '172.16.%' — that catches anything going to the public internet. Luna: Nice. So the bastion isn't just a door — it's a sensor. Lucas: Exactly. And once you have that sensor, you can also use it to enforce network segmentation. The bastion becomes the only way into your internal network. No VPN, no direct access. That makes firewall rules much simpler. Luna: Alright, I'm convinced. But let's play devil's advocate: what about the argument that bastions add complexity and a new attack surface? Lucas: They do add a component, but the complexity is bounded. And the attack surface of a hardened bastion with one service, no passwords, and no root is far smaller than the aggregate surface of two hundred servers each running SSH on port 22 with default settings. It's a net reduction. Luna: Fair. And you can automate the bastion configuration with Ansible or Terraform so it's reproducible. Lucas: Exactly. Infrastructure as code. Your bastion should be ephemeral — if it gets compromised, you destroy it and spin up a new one from your Git repository. The whole process takes minutes. Luna: That's a good note to end on. One concrete takeaway: if you have more than five servers exposed to the internet on SSH, set up a bastion this week. It's the single highest roi security change you can make. Lucas: Agreed. And if you've already got one, check your config — are you logging sessions? Using certificates? Forwarding agent safely? There's always room to tighten. Luna: Thanks, Lucas. Great conversation. Lucas: Thanks, Luna. See everyone next time.