Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your Server Needs A Dedicated Log Partition
Transcript
- Lucas: A mid-size e-commerce company I know lost three hours of revenue last year because one service — a chatty microservice that logs every single API call — filled up the root partition. The server didn't crash dramatically. It just slowly stopped being able to write to disk. Databases couldn't commit, the application couldn't log errors, and by the time anyone noticed, the only fix was to boot a rescue image and clear space. Three hours of downtime for what? No dedicated log partition. Luna: And this is the kind of failure that feels completely preventable in hindsight, but it keeps happening. I've seen it at a few startups I've worked with too. The root partition is 20 gigs, and someone thinks 'eh, logs are small' until they aren't. Lucas: Exactly. The problem is that logs are unbounded by default. Most Linux distributions install with syslog or journald already running, and they'll keep accumulating entries until the disk is full. No built-in cap. So when a service goes rogue and starts writing a log line every millisecond, you've got roughly until the average sysadmin checks their morning coffee before the disk is full. Luna: But a dedicated log partition doesn't fix the full-disk problem entirely, right? It just isolates the damage. The root filesystem is safe, but the log partition can still fill up, and then what? Logs stop writing. Lucas: That's exactly right. The goal isn't to prevent the disk from ever filling — it's to prevent the root partition from filling. If your log partition hits 100 percent, your application services might stop logging, but they won't crash. The OS can still write to /var, /tmp, and all the critical system paths. Plus, you can set up alerts on the log partition at, say, 80 percent, and have time to rotate or archive before anything bad happens. Luna: So the dedicated partition is really about defense in depth. Root stays clean, and you can monitor the log space independently. What's the typical partition scheme you'd recommend for a server? Lucas: For a standard Linux server, I like separate partitions for /boot, /, /var, and /home. But specifically for logs: put /var/log on its own partition. How big? It depends on your logging volume. For a typical web server, maybe 10 to 20 gigs. For a high-traffic application or a centralized log collector, I've seen 100 gigs or more. The key is to size it based on your log retention policy and average daily log rate, plus a buffer for spikes. Luna: How do you estimate that? Do you just guess, or is there a methodology? Lucas: Start by looking at your current log growth. Run 'du -sh /var/log' over a week and see the daily delta. Multiply by your desired retention days — say 30 — then add 20 percent headroom. That gives you a baseline. Then consider worst-case: if a service goes haywire, how fast can it fill the partition? You want the partition to be large enough that you have hours, not minutes, to react. Luna: And once you've got the partition, you need log rotation. Otherwise the partition still fills up eventually. Lucas: Right, logrotate is the standard tool. It compresses, rotates, and deletes old logs based on size or time. A common config for Apache logs: rotate weekly, keep four weeks, compress, delay compress so the most recent log isn't compressed while Apache might still write to it. And you can set a size trigger too, like 'size 100M' to force rotation regardless of time. Luna: One thing I've seen trip people up is that logrotate runs as a cron job, typically daily. If your logs grow faster than that, you need a size-based trigger. Systemd's journald has its own built-in limits too, like SystemMaxUse. Have you switched to using journald exclusively, or do you still run syslog alongside? Lucas: I run both. Journald is great for structured logs and easy queries with journalctl, but traditional syslog text files are still easier to forward to a centralized logging system or grep in an emergency. So I let journald keep its binary journal, but I also have rsyslog writing to plain files in /var/log. Then I configure journald's SystemMaxUse to, say, 500 megs so it doesn't eat all the space, and I let logrotate handle the text files. Luna: That's a solid hybrid approach. But let's talk about containers. If you're running Docker or Kubernetes, logs go to stdout and stderr, and the container runtime usually handles them. Does a dedicated log partition still matter? Lucas: It matters even more, because container logs can be incredibly verbose. Docker by default writes container stdout/stderr to a JSON file on the host, and without log rotation, those files can explode. You can configure the Docker daemon with log-opts: max-size and max-file. But the underlying filesystem — the host's /var/lib/docker — is often on the root partition. So if you don't have a separate partition for Docker data, you're back to the same root-filling problem. Luna: So you'd recommend a separate partition for /var/lib/docker as well? Or at least a mount point for container logs? Lucas: Exactly. On container hosts, I put /var/lib/docker on its own logical volume or partition, often 50 to 100 gigs. Then within that, I set Docker's log driver to 'json-file' with max-size=10m and max-file=3 per container. That keeps each container's log manageable. And I also have the logrotate configuration for the host's own logs, of course. Luna: Let's go back to the monitoring piece. You mentioned alerts at 80 percent. What tools do you use for that? Lucas: The simplest is a cron job that runs df -h and checks the usage percentage. If /var/log is above 80, send an email or a Slack message. More sophisticated setups use Prometheus with node_exporter collecting disk metrics, and Alertmanager firing at thresholds. But honestly, even a simple bash script — 10 lines — is better than nothing. Luna: And what about the root partition itself? Do you monitor that too, even though logs are isolated? Lucas: Absolutely. Root can still fill up from other things: temporary files, crash dumps, package caches. I set alerts on all partitions, but with different thresholds. Root might alert at 85 percent, log partition at 80, home at 90. And I always leave some free space — never provision a partition to 100 percent of the underlying volume group. Leave 5 to 10 percent for snapshots or emergencies. Luna: One last thing: recovery. If the log partition does fill up and logs stop writing, what's the quickest way to free space without rebooting? Lucas: Delete or truncate the largest log files. Use 'du -sh /var/log/* | sort -rh' to find the biggest ones. Then either '> /var/log/somefile.log' to truncate in place, or use logrotate's force option: 'logrotate -f /etc/logrotate.conf'. If you truncate, the application may need to reopen the file handle — so sending a SIGHUP to syslog or the application is a good idea. And of course, fix the root cause: the chatty service. Luna: It sounds like a lot of planning, but the payoff is huge. Three hours of downtime for a single misconfigured partition is a hard lesson. Lucas: It really is. And the fix is so cheap: a few extra partitions at install time, a logrotate config, and a monitoring script. That's maybe an hour of work. Yet I still see production servers with a single root partition and no log rotation. It's one of those things that everyone knows they should do, but it's easy to skip until it bites you. Luna: So if someone listening is setting up a new server today, what's the one thing you'd tell them to do before deploying? Lucas: Before you deploy anything, carve out separate partitions for /var/log and /var/lib/docker. Set up logrotate with a size trigger and a retention policy. And write a cron job that checks disk usage every five minutes and warns you if any partition crosses 80 percent. That one checklist item will save you more headaches than almost any other sysadmin practice I can think of. Luna: Good advice. And it's the kind of thing that becomes muscle memory once you've been burned once. Lucas: Exactly. I've been burned. And I don't want our listeners to have to go through that same outage. So take the time now — before you go live — to plan your partitions. Your future self will thank you.