Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Manage Linux Server Secrets with HashiCorp Vault
Transcript
- Lucas: So you've got a Linux server running a web app. Where does it store its database credentials? If your answer is a dotenv file, or an encrypted config that gets decrypted at boot, you're not alone — but you're also running a risk that scales poorly. Luna: Right. Because once you have more than a handful of servers, those secrets end up scattered everywhere. One leaked backup, one misconfigured permissions — suddenly someone's got your production database password. Lucas: Exactly. And that's where a dedicated secrets management tool changes the game. The most well-known open-source option in the Linux server world is HashiCorp Vault. It's not new — it's been around since 2015 — but adoption has really accelerated in the last few years, especially as teams move toward dynamic secrets rather than static ones. Luna: Let's define dynamic secrets quickly. That means the database password isn't a string you write once and rotate manually every quarter. Instead, Vault generates a credential on the fly, with a time to live, and the application only gets it for as long as it needs it. Lucas: Exactly. When the app starts up, it authenticates to Vault, requests a database credential, Vault creates a user in PostgreSQL or MySQL with a short lease — say 24 hours — and the app uses that. When the lease expires, Vault drops the user. No human ever touches that password. Luna: That's a massive improvement over the old approach. But it does require you to run Vault itself. So what does a basic Vault deployment look like on a Linux server? Lucas: At its simplest, you install the Vault binary on a dedicated server — or ideally three servers in a cluster. Vault uses a 'seal' concept. When you first start it, it's in a sealed state — it can't serve any requests. You have to unseal it by providing a threshold of unseal keys. Typically you generate five keys, and any three can unseal it. Luna: That's the Shamir secret sharing scheme, right? So even if an attacker gets one key, they can't unseal Vault. Lucas: Correct. And in production, you'd distribute those keys to different team members. Then, once unsealed, Vault is ready to serve secrets. The next step is enabling a secrets engine. For database credentials, you enable the 'database' secrets engine and configure it with the connection details and a plugin for your database — like 'postgresql database plugin'. Luna: And then you create a role. That role defines what the generated user can do — which database, which tables, what permissions. Lucas: Right. So you'd run something like 'vault write database/roles/my-app' with a creation statement like 'CREATE USER... WITH PASSWORD...' and a default TTL. When an application authenticates to Vault, it reads from that role, and Vault executes the statement, creates the user, and returns the credentials. Luna: Let's talk about how the application authenticates to Vault in the first place. Because you need some initial credential to get into Vault — and if you put that credential in a file on the server, you're back to square one. Lucas: That's the classic bootstrap problem. Vault solves it with something called the Vault Agent. You run a small daemon on each application server that handles authentication on behalf of the app. The agent can use different auth methods — like AppRole, or Kubernetes service accounts if you're in containers. Luna: So the agent holds a secret ID, which is still a credential — but it's a machine-only credential that can be revoked centrally. And it can be delivered out of band, like during server provisioning. Lucas: Exactly. In practice, you'd generate an AppRole role in Vault, get the role ID and a wrapped secret ID, and inject them into the server at deployment time — maybe via your configuration management tool like Ansible or Terraform. Then the Vault Agent uses those to authenticate and get a token. The app never sees the underlying secret ID. Luna: And once the agent has a token, it can renew it automatically. So the app just requests secrets from a local socket or HTTP endpoint. Lucas: Right. Vault Agent can even template secrets directly into files. You define a template — say 'database.yaml.tmpl' — and Vault Agent renders it with the actual credentials, then updates the file when the lease is about to expire. The app reads from that file. Luna: That means the app doesn't even need to know Vault exists. It just reads a config file that gets refreshed transparently. Lucas: Precisely. And if you want to get even more advanced, you can use Vault's 'database' engine with automatic rotation. Every time the app renews its lease, Vault can rotate the password behind the scenes. The user in PostgreSQL gets a new password, but the app only sees the new one through the template. Luna: That's a huge operational win. But I want to talk about the complexity trade-off. Running a Vault cluster requires effort. You need to manage the storage backend — typically Consul or Raft — handle upgrades, monitor the seal state. Is it worth it for a smaller shop? Lucas: It depends. If you're running five servers and one database, you can probably get away with encrypted config files and a manual rotation script. But as soon as you have multiple environments, multiple databases, and automated deployments, the cost of not having Vault shows up as risk. A single exposed credential in a CI log can cost you dearly. Vault is an insurance policy. Luna: Speaking of insurance — if today's episode gave you a concrete idea for improving your server security, that's exactly the kind of value we aim for here. And the reason we can go deep without running ads is listener support. Lucas: Yeah. If this was useful, you can help keep it ad-free at buy me a coffee dot com slash fexingo. It's a simple way to say 'keep going' — and it genuinely makes a difference for us. Luna: Totally. And we appreciate every single one. So back to Vault — Lucas, you mentioned Consul as a storage backend. What are the practical choices for storing Vault's data? Lucas: Vault supports several backends. The most common two are Consul and Integrated Storage — that's Raft-based. Consul is a separate product, so you'd run a Consul cluster alongside Vault. Integrated Storage means Vault itself handles the Raft consensus, which simplifies deployment — you just run three Vault nodes and they talk to each other. Luna: For a small team, Integrated Storage is probably the way to go. One less thing to manage. Lucas: Exactly. And you can start with a single node for testing, then add nodes later. One thing to watch: if you lose a majority of nodes, Vault becomes unavailable. So three nodes is the minimum for production. Luna: And unsealing — you mentioned the five-key, three-threshold approach. In production, you'd automate that with something like a 'vault unseal' command from a trusted machine, or use auto-unseal with a cloud KMS. Lucas: Right. Auto-unseal uses a key encryption key stored in AWS KMS, Azure Key Vault, or GCP Cloud KMS. When Vault starts, it calls the cloud API to unwrap the master key. That's convenient but adds a cloud dependency. For on-prem, you might use a hardware security module or just stick with manual unseal for smaller setups. Luna: Let's walk through a concrete example. Suppose I have a Linux server running a Flask app that connects to PostgreSQL. I want to use Vault to get the database password. What's the step-by-step? Lucas: First, install Vault on a dedicated server. Start it in development mode for testing — 'vault server -dev'. That gives you an unsealed Vault with a root token. Then enable the database secrets engine: 'vault secrets enable database'. Next, configure the PostgreSQL connection: 'vault write database/config/my-postgres' with the connection URL for a privileged user that can create other users. Luna: That privileged user is important — it's like a master account that Vault uses to create ephemeral accounts. Lucas: Yes. Then create a role: 'vault write database/roles/flask-app' with a creation statement that grants SELECT on the app's tables, a default TTL of one hour, and a max TTL of 24 hours. Now, on the application server, install and configure Vault Agent. Write a config file that points to the Vault server, uses AppRole authentication, and defines a template. Luna: And that template — say 'db.conf.tmpl' — would contain something like 'DATABASE_URL=postgresql://{{.Data.username }}:{{.Data.password }}@host/dbname'. Vault Agent renders that to 'db.conf' and keeps it updated. Lucas: Exactly. Then start Vault Agent as a systemd service. The Flask app reads 'db.conf' at startup. When the lease is halfway to expiry, Vault Agent renews it automatically. If renewal fails, the app still has the old credential until it expires, so you get graceful degradation. Luna: One gotcha: the app needs to be able to re-read the config file without restarting. So either the app polls for file changes, or you use a signal. Lucas: Right. A common pattern is to have the app watch the file with inotify or just re-read on each request. For a Flask app, you might reload the database connection on config change. It's an extra engineering step, but it's worth it for zero-downtime credential rotation. Luna: Let's talk audit. Vault has a rich audit log — you can send it to syslog or a file. That logs every secret access, every token creation. For compliance, that's gold. Lucas: Absolutely. You can even enable audit logging on multiple backends simultaneously — say, one for real-time monitoring and one for archival. And because Vault is the single point of access for secrets, you get a complete trail of who accessed what and when. Luna: So the architecture is: Vault as the central authority, Vault Agent on each server as the local proxy, and the app as the consumer. It's clean, auditable, and scalable. Lucas: Exactly. And once you have that foundation, you can start adding more secrets engines — for SSH keys, certificates, API tokens. Vault can even act as a certificate authority for internal TLS. It becomes a platform for managing trust in your infrastructure. Luna: That's powerful. But I want to end with a practical reminder: Vault doesn't solve everything. If you have a compromised server, the attacker can still read the temporary credentials from memory or from the templated file. Vault reduces the blast radius — credentials are short-lived — but you still need host-level security. Lucas: Absolutely. Vault is part of a defense-in-depth strategy. It's not a silver bullet. But it moves you from 'static credentials that live forever' to 'dynamic credentials that expire in hours'. That alone raises the bar significantly. And for anyone managing Linux servers at scale, it's a tool worth investing in.