Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Automating Server Backups With Borgmatic
Transcript
- Lucas: Let me tell you about the morning I walked into the office and found that a staging server had been wiped clean by a rogue script. Not the production server, thank goodness, but it still had a week's worth of customer transactions that nobody had thought to back up separately. Luna: Oh, that sinking feeling. Was there any backup at all? Lucas: There was one — a nightly Borgmatic backup that ran automatically. I had set it up three months earlier after a close call with a different server. That morning, it took about 45 minutes to restore everything, and the only data we lost was from the current day. The rest was intact. Luna: So Borgmatic — that's the wrapper around Borg, right? The deduplicating backup tool. Lucas: Exactly. Borg is the engine. Borgmatic is a configuration-driven wrapper that makes it much easier to set up consistent backups. You define your source directories, your encryption method, your retention policy, and it handles the rest. No more piping tar through SSH and hoping for the best. Luna: And the deduplication is a big deal. I've seen people avoid backups because they think it'll eat up terabytes of storage. But Borg splits files into chunks and only stores new chunks. If you're backing up a database nightly, the first backup might be big, but subsequent ones are tiny. Lucas: Right. In practice, I've seen an 80 percent reduction in storage compared to traditional full backups. For a 50-gigabyte database, the initial Borg backup might be 50 gigs, but the next day's incremental backup might be just a few hundred megabytes. And Borg's compression helps too — by default it uses LZ4, which is fast and gives decent ratios. Luna: Let's talk about the setup. What does a basic Borgmatic configuration look like? Lucas: It's a YAML file, typically at /etc/borgmatic/config.yaml. You specify the source directories, the repository path — that's where the backups are stored — and the encryption passphrase. You also set the retention policy, like 'keep daily for 7, keep weekly for 4, keep monthly for 6.' That's the core. Luna: And encryption — Borg uses AES-256 by default. That's a must for any off-site backup. If your backup drive gets stolen, the data is unreadable without the passphrase. Lucas: Exactly. But the passphrase itself is a weak point. Don't hardcode it in the YAML file. Use an environment variable or a key file with restricted permissions. And store that passphrase in a password manager, because if you lose it, the backup is gone. Luna: How do you schedule it? Cron? Lucas: Yeah, a simple crontab entry. I run it at 2 AM every day. The command is just 'borgmatic --cron'. I also pipe the output to a log file so I can check for errors. And I have a separate monitoring script that checks if the last backup is older than 36 hours and alerts me. Luna: That's smart — because a backup that fails silently is no backup at all. I've seen setups where a disk filled up, the backup stopped running, and nobody noticed for weeks. Lucas: Exactly. That's why I also enable Borg's append-only mode. If the backup server gets compromised, an attacker can't delete existing archives. They can only append new ones. That gives you a window to detect the breach and still have your old backups. Luna: Append-only is huge for ransomware protection. If an attacker gets root on the backup server, they could otherwise delete everything and leave you with nothing. But append-only prevents that. Lucas: And Borgmatic supports it natively. You just set 'append_only: true' in the configuration for the remote repository. But note — it only works if the remote server supports it. On local drives, you can use filesystem permissions, but it's not as robust. Luna: What about restoring? Is it as simple as the backup? Lucas: It's actually easier than most. To restore the entire backup, you run 'borgmatic restore' and it restores the latest archive. To restore a single file, you can use 'borg extract' directly to pull just that file. Or you can mount the backup as a FUSE filesystem and browse it like a regular directory. Luna: Mounting is brilliant for quick checks. You can verify that a specific file is there without restoring everything. Lucas: Exactly. And that's the key thing: test your restores. I've talked to sysadmins who have perfect backup scripts but never actually restored from them. Then when disaster strikes, they discover the backup is corrupt or the encryption passphrase doesn't work. Schedule a quarterly restore test. Luna: So what's a concrete example? Say I have a small web application with a PostgreSQL database and some uploaded files. How would I set this up? Lucas: First, you'd create a user for backups, maybe 'borgmatic', with limited permissions. Then you'd create the backup repository — 'borg init --encryption=repokey-blake2 /path/to/repo'. That creates the repo with encryption. Then you write your config file. Luna: And for the database, you'd use a pre-backup hook to dump the database to a file, then include that file in the source directories. Lucas: Right. So your config would have a 'hooks' section with 'before_backup' commands that run pg_dump and save it to a temp directory. Then in 'source_directories', you include that temp directory plus the uploads folder. After the backup, a 'after_backup' hook removes the dump file. That way, the backup is consistent point-in-time. Luna: And for a remote backup, you'd use SSH. Borg supports remote repositories over SSH natively. You'd set the repository to 'user@backupserver:/path/to/repo'. Lucas: Exactly. And you can use SSH keys for authentication. Make sure the backup user on the remote server has restricted permissions — ideally only to the Borg repository path. Use command restrictions in the authorized_keys file to limit what commands can be run. Luna: What about bandwidth? If you're backing up over a slow link, Borg's deduplication helps a lot, but the initial backup could be huge. Lucas: For the initial backup, you might want to do it locally first, then transfer the repo to the remote server. Or you can use Borg's '--remote-ratelimit' option to limit bandwidth. I usually set it to 10 megabits per second during business hours and unlimited at night. Luna: One thing that trips people up is the 'borgmatic prune' step. If you don't prune old backups, the repository grows indefinitely. Lucas: Right. In the config, you set a 'retention' section with 'keep_daily', 'keep_weekly', etc. Borgmatic runs prune automatically after each backup by default. But you can also run it manually if you need to free up space. Just be careful — pruning is permanent. Luna: And there's a 'check' step too, which verifies the integrity of the repository. Borgmatic can run that after each backup, but it's resource-intensive. I usually run it weekly. Lucas: Good point. The check command uses Borg's 'check' which does a full integrity scan. It can take a long time for large repos. So I set 'check: once per week' in the config. And I always run it before I trust a restore. Luna: So, if today's conversation gave you something usable — maybe a better backup strategy, or a concrete step you can implement tonight — that's exactly why we do this show. And the way we keep it ad-free and focused on real sysadmin use cases is listener support. If you found it useful, you can buy us a coffee at buy me a coffee dot com slash fexingo. It's a simple way to say thanks and helps us keep putting episodes like this out. Lucas: Yeah, completely. We don't do sponsors, we don't do ad reads — it's just the content we think is actually valuable. So if you're in a position to chip in, we appreciate it. But no pressure. Luna: Alright, back to backups. One more thing I want to cover: what about backing up the backup server? If your backup server dies, you've lost everything. Lucas: That's the 3-2-1 rule: three copies of your data, on two different media types, with one copy off-site. For critical data, I use Borgmatic to push backups to both a local NAS and a remote VPS. The remote VPS is in a different datacenter. And I use Borg's 'borgmatic config' to manage both repositories in one config file. Luna: So you'd have two repositories listed in the 'repositories' section. Borgmatic will back up to both sequentially. That's a clean solution. Lucas: Exactly. And for the off-site repo, I use append-only mode with a separate passphrase. That way, even if the local repo gets compromised, the off-site repo is safe and append-only. Luna: What about monitoring? You mentioned a script that checks backup age. Do you use any specific tool? Lucas: I wrote a simple Bash script that runs via cron every hour. It parses the Borgmatic log file for the last successful backup timestamp and compares it to the current time. If it's older than 36 hours, it sends an alert via a webhook to a chat channel. You could also use Nagios or Prometheus, but the script is lightweight. Luna: I do something similar, but I also alert on the exit code of the borgmatic command. If it fails, the cron job sends an email. It's saved me a few times. Lucas: Yeah, email alerts are underrated. They're simple and they work. Just make sure the mail infrastructure is reliable — otherwise you get silent failures. Luna: So to wrap up: Borgmatic gives you deduplication, encryption, compression, append-only, and a clean YAML config. It's probably the most straightforward backup solution for Linux servers that I've used. Lucas: I'd agree. It's not the only option — you've got restic, duplicity, rsync with hardlinks — but for someone who wants a set-it-and-forget-it solution with strong integrity guarantees, Borgmatic is hard to beat. Start with one server, test your restores, then expand. Luna: And if you're new to it, the official documentation at borgmatic.readthedocs.io has a quick start guide that'll get you going in about 15 minutes. That's where I'd point anyone. Lucas: Perfect. Next episode, let's talk about monitoring stacks — maybe Prometheus and Grafana, or a simpler alternative. There's a lot to cover. Luna: Sounds good. Until then, keep your backups tested.