Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your Linux Server Needs a Dedicated Time-Series Database
Transcript
- Lucas: Let me start with a number: five hundred thousand. That's how many data points per second a mid-size e-commerce company was pushing into their Prometheus setup before the whole thing started falling apart. Luna: Five hundred thousand per second? That's not a hobby project. That's a serious monitoring workload. Lucas: Exactly. And their problem wasn't that Prometheus is bad — it's that they were using a general-purpose time-series tool for what had become a heavy-duty time-series workload, and they hadn't thought about the underlying storage engine. Luna: So what broke first? The queries? The disk? Lucas: Both, actually. Queries that used to finish in under a second started timing out at thirty seconds. And they started seeing gaps in their graphs — data simply not getting written because the write path was saturated. So they came to me asking, 'Should we just throw more RAM at it?' And the answer was no. Luna: Right, because the bottleneck wasn't memory — it was the storage architecture itself. Lucas: Exactly. What they needed was a dedicated time-series database. Not a general-purpose database with time-series data in it, but an engine purpose-built for this kind of workload. And that's what I want to talk about today: why your Linux server probably needs a dedicated time-series database, and when it doesn't. Luna: Let's start with the 'what' — what exactly makes a database 'time-series' as opposed to just a regular database with timestamps? Lucas: Great question. A time-series database — or TSDB — is optimized for data that arrives as a continuous stream of measurements, each with a timestamp. Think CPU utilization, disk I/O latency, network bytes in and out, request latencies. The key characteristics are: high write throughput, because you're constantly ingesting new data; time-based queries, like 'give me the last hour'; and efficient storage, because you don't want to keep every data point forever. Luna: So it's not just about speed — it's about storage efficiency too. Because raw metrics can eat disk space fast. Lucas: Exactly. A regular relational database like PostgreSQL or MySQL stores each row as a separate record with all the overhead that entails. If you're inserting millions of rows per second, the index maintenance alone becomes a nightmare. TSDBs use columnar storage, compression, and downsampling to handle that scale. Luna: Let's talk downsampling. That's a feature where you automatically aggregate old data, right? So instead of storing every millisecond reading from a year ago, you store one-minute averages. Lucas: Exactly. InfluxDB, one of the most popular open-source TSDBs, lets you define retention policies and continuous queries that downsample data after a certain age. So you keep raw data for, say, seven days, then one-minute aggregates for thirty days, then one-hour aggregates for a year. Your disk usage stays predictable, and queries on old data are still fast because you're querying pre-aggregated values. Luna: And that's something Prometheus doesn't do natively, right? Prometheus stores raw samples with a fixed retention, and if you want downsampling, you need something like Thanos or VictoriaMetrics on top. Lucas: Right. Prometheus is great for short-term alerting and ad-hoc queries, but it's not designed to be a long-term metrics archive. The company I mentioned was using Prometheus with default settings, keeping data for fifteen days, and they were already struggling. They needed something that could handle both the ingestion rate and the query load for their Grafana dashboards. Luna: So what did you recommend? InfluxDB? VictoriaMetrics? Something else? Lucas: In their case, I recommended VictoriaMetrics. It's compatible with the Prometheus query language — PromQL — so they didn't have to rewrite their dashboards. And it has a cluster mode that scales horizontally. They migrated their Prometheus remote write endpoint to VictoriaMetrics, and within a week, their queries were fast again, and the gaps disappeared. Luna: So VictoriaMetrics became a drop-in replacement for the storage backend, without changing the frontend. That's a nice migration path. Lucas: Exactly. But I want to be clear: not every server needs a dedicated TSDB. If you're running a single server with a few dozen services, Prometheus with local storage is perfectly fine. The tipping point is when you start collecting metrics from more than, say, fifty hosts, or when your query patterns involve a lot of time-range scans on high-cardinality data. Luna: What about log data? People often lump logs in with metrics, but logs are usually unstructured text. Do you put logs in a TSDB too? Lucas: You can, but I wouldn't. TSDBs are optimized for numeric measurements, not text. For logs, you're better off with a dedicated log management system like Loki or Elasticsearch. That said, some TSDBs, like InfluxDB, can handle string tags and fields, but it's not their strength. I'd keep metrics and logs separate. Luna: That makes sense. Different tools for different jobs. But what about the operational overhead? Adding another database to your stack means another service to patch, back up, and monitor. Lucas: It's a valid concern. But the alternative — a stressed Prometheus or a custom solution — often causes more operational pain. With a dedicated TSDB, you get built-in replication, retention policies, and often a simpler backup story. For example, VictoriaMetrics has a single binary that you can run as a systemd service. No dependencies. You just point your metrics source at it, and you're done. Luna: And you can start small. You don't need a cluster from day one. Lucas: Exactly. Start with a single node, set a thirty-day retention, and see if it handles your workload. If it does, great. If you outgrow it, most TSDBs support clustering or sharding. But for 90 percent of setups, a single node is plenty. Luna: So let's talk practical deployment. You're setting up a new Linux server today. What's your go-to stack for metrics? Lucas: Simple: Prometheus for scraping targets and alerting, VictoriaMetrics as the long-term storage backend, and Grafana for dashboards. All open source, all well-documented. I'd install VictoriaMetrics as a one-line systemd unit, configure Prometheus to remote write to it, and set a retention policy of thirty days for raw data and six months for downsampled. Luna: And for alerting? Prometheus Alertmanager handles that? Lucas: Yes. Prometheus does the evaluation, Alertmanager handles deduplication and routing. That part doesn't change. The TSDB just replaces the local storage on the Prometheus server. It's a surprisingly clean separation. Luna: I want to push back a little on the 'you need a TSDB' idea. Because some sysadmins might hear this and think, 'Great, I'll install InfluxDB on every box.' But there's a real cost in complexity. When do you say no to a TSDB? Lucas: Good pushback. I'd say no when you have fewer than ten servers, or when your metric volume is under ten thousand data points per second. At that scale, a simple Prometheus instance with local disk is faster to manage. Also, if your monitoring is purely reactive — you look at graphs when something breaks — you probably don't need long-term storage. A seven-day retention in Prometheus is fine. Luna: So the threshold is really about scale and query patterns. If you're constantly going back three months to diagnose an intermittent issue, that's a sign you need a TSDB. Lucas: Exactly. And that's the scenario I see most often: a team that's grown from twenty servers to two hundred, and they're still using the same monitoring setup they had when they were small. The graphs show gaps, queries time out, and nobody knows why. That's the moment to introduce a dedicated time-series database. Luna: Before we wrap up, I want to come back to something you said earlier — about the e-commerce company. They were pushing half a million data points per second. What kind of hardware did they end up using for VictoriaMetrics? Lucas: They used a single virtual machine with eight CPU cores, thirty-two gigs of RAM, and a thousand-gig SSD. That's it. It handled the load with room to spare. The key was that they set up proper downsampling — kept raw data for only three days, then five-minute aggregates for ninety days. Their disk usage stayed under 200 gigabytes. Luna: So the TSDB's compression and downsampling did the heavy lifting. They didn't need a cluster. Lucas: Right. And that's the takeaway: a dedicated time-series database isn't a silver bullet, but when your metrics outgrow your general-purpose tools, it's the cleanest solution. Start small, set retention policies, and let the database handle the scale. Luna: If today's conversation gave you something usable — a tool to try, a design pattern to rethink — that's exactly what we hope for. This show stays ad-free and listener-supported, and a handful of people already chip in monthly at buy me a coffee dot com slash fexingo. That's what keeps episodes like this coming. Lucas: Yeah, it's a small thing that makes a big difference. And we appreciate it. Alright — next time, I want to talk about something that trips up even experienced sysadmins: how to properly size swap space on modern Linux servers. Luna: Oh, that's a good one. There's a lot of outdated advice out there. Lucas: Exactly. We'll sort through what actually makes sense for today's hardware.