Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why You Should Monitor Server Disk Latency Now
Transcript
- Lucas: If your server monitoring dashboard shows CPU at 30 percent, memory at 50 percent, and the application is still crawling — stop hunting for the process that's eating resources. Look at disk latency first. Luna: Yeah, because everything else can look fine and the real problem is that the storage layer is silently choking. Lucas: Exactly. And I think that's one of the most underappreciated blind spots in server administration. Most people monitor disk usage — how full is the partition — but almost nobody monitors how long it takes to actually read or write data. Luna: So what's the number you actually watch? Is it iowait? Lucas: Iowait is a starting point, but it's noisy. On a multi-core system, a small spike in iowait can look like nothing while your app is wedged. I prefer to look at average I/O latency per device, and the 99th percentile tail latency. A real case: last year, a mid-sized e-commerce company — I won't name them, but they do about 50 million in annual revenue — lost 12 percent of checkout conversions over a three-week period. Their dashboards were green across the board. CPU, memory, network, disk usage — all green. Luna: But the site was slow during checkout? Lucas: Painfully slow. Page loads that should take under a second were taking four to six seconds. The engineering team spent days profiling application code, adding query indexes, caching — none of it helped. Finally, someone ran iostat and saw that the average write latency on their database volume was 120 milliseconds. That's about ten times what it should be for an ssd backed volume. Luna: And nobody had been watching that because the disk wasn't full. Lucas: Right. The disk was only 40 percent full. But the I/O scheduler was drowning in concurrent writes because their database had grown to a point where checkpoint flushes were saturating the queue depth. The fix was straightforward — move the WAL logs to a separate faster volume with a higher queue depth — but it took three weeks to find because they weren't monitoring the right metric. Luna: So what's the single command you'd run right now to check if your server has a disk latency problem? Lucas: I'd start with iostat -x 1. The extended statistics. Look at the 'await' column — that's the average time for I/O requests to complete, including wait time in the queue. Anything above 10 milliseconds for SSDs is suspicious. Above 20 milliseconds, you've got a problem. Then look at '%util' — if that's consistently above 90 percent, your device is saturated. Luna: But %util can be misleading on modern drives, right? Because NVMe drives can have very high queue depths. Lucas: That's a great point. %util essentially tells you if the device was busy doing something during the sampling interval. But an NVMe drive can be 100 percent busy and still complete thousands of I/Os per second with low latency. So %util alone isn't sufficient. That's why I also watch 'avgqu-sz' — the average queue size — and 'r_await' and 'w_await' separately. Luna: And 'svctm'? I've seen old sysadmins still swear by that. Lucas: Avoid it. 'svctm' is essentially deprecated in modern iostat versions. It was meaningful on spinning disks from the 1990s. On SSDs with NCQ and NVMe with multiple queues, it's a meaningless number. The kernel documentation itself warns against using it. Luna: Good to know. So if I log into a server, run iostat -x 1, and see await at 30 milliseconds on my database disk — what's the first thing I do? Lucas: Check if it's a read or write problem. Look at the read/write split. If reads are slow, your data might not be cached in memory. If writes are slow, you could be hitting a journal commit bottleneck. Then run iotop to see which processes are doing the I/O. You might find that a rogue log writer or a backup script is flooding the disk. Luna: I once had a server where a developer had accidentally left a debug logging level on in production. It was writing gigabytes of logs every hour. iotop caught it in about five seconds. Lucas: Exactly. iotop is your best friend for that. But there's another layer: if the latency is intermittent, you need to capture the tail. Averages hide spikes. A device can have an average await of 5 milliseconds but still have 1 percent of requests taking 500 milliseconds. That one percent can kill your application's 99th percentile response time. Luna: So how do you catch those spikes without sitting there watching iostat for an hour? Lucas: Use sar. The sysstat package. 'sar -d' will collect disk statistics at intervals and you can look at historical data. Set your collection interval to every 30 seconds and keep at least a week of history. Then you can graph it and see if latency correlates with your application slowdowns. Luna: Okay, so sar is the retroactive tool. But for real-time, iostat and iotop. What about something like 'pidstat -d'? Lucas: Also excellent. 'pidstat -d 1' shows per-process I/O statistics. That's how you find exactly which PID is generating the most reads or writes. Combine that with iotop, and you have a very complete picture. Luna: Let's talk about the hardware side for a second. If you're buying new servers today, what should you look for to avoid disk latency issues? Lucas: First, don't put everything on one NVMe drive. Even the fastest NVMe drive has a finite queue depth and bandwidth. Separate your database data from your transaction logs. Use multiple drives, and if you're using RAID, use RAID 10, not RAID 5 or 6. RAID 5 write penalties kill latency on mixed workloads. Luna: And what about cloud volumes? EBS gp3 vs io2, or similar offerings from other providers? Lucas: In the cloud, be very careful about burst credits. Many general-purpose volumes have a baseline IOPS and a burst pool. If you exhaust the burst credits, your latency goes through the floor. Provisioned IOPS volumes are more expensive, but they guarantee consistent latency. For any latency-sensitive workload, use provisioned IOPS. Luna: It's one of those things that's easy to overlook because when you first set up a server, it's fast. The burst credits are full. Three months later, you've grown, and suddenly the application is slow. Lucas: Yes. And that's exactly what happened with that e-commerce company I mentioned. Their database volume was a general-purpose SSD with burst credits. As their traffic grew, they started consuming credits faster than they could replenish them. By the time they noticed slowdowns, the burst balance was zero. They upgraded to a provisioned IOPS volume and latency dropped back to single-digit milliseconds overnight. Luna: So the lesson is: monitor disk latency from day one. Not just disk usage. Lucas: Exactly. Add iostat to your monitoring stack. Set alerts on await. Even a simple cron job that logs iostat output to a file can save you a lot of pain later. It's one of those things that takes five minutes to set up and can prevent hours of firefighting. Luna: I think a lot of people assume that if they're using SSDs, latency is never an issue. They think of the old spinning-disk days as the only time you had to worry about I/O. Lucas: Right. SSDs are orders of magnitude faster than HDDs, but they're still subject to queue depth saturation, write amplification, and garbage collection. A modern NVMe drive can do millions of IOPS under ideal conditions, but if you have a single-threaded process doing synchronous writes, you can still bottleneck. The principles haven't changed — the numbers have just shifted. Luna: One thing I've found useful is benchmarking your storage before putting a workload on it. Run fio with a realistic profile. Lucas: Absolutely. fio is the standard. Test with a mix of reads and writes, with the same block sizes your application uses. That gives you a baseline. Then when something goes wrong months later, you know what 'normal' looks like. Luna: And speaking of knowing what normal looks like — if you are getting value from this show and want to support more deep-dive content like this, we are ad-free and listener-supported. If today's tech conversation gave you something usable, and you feel like it, the link is buy me a coffee dot com slash fexingo. Truly the smallest possible gesture — and it helps us keep the episodes coming without sponsor interruptions. Lucas: Yeah, we appreciate anyone who chips in. It's not expected, but it directly funds the research time for episodes like this one. Okay — back to disk latency: one more tool I want to mention before we wrap is blktrace. That's for when you really need to drill into block layer events. Luna: blktrace is powerful but it can generate a ton of data, right? Lucas: It can. It's not something you run casually. But if you have a persistent latency issue and iostat and iotop aren't telling you enough, blktrace will show you exactly how long each I/O request spends in each stage — from the block layer submission to the driver completion. You can identify whether the bottleneck is in the queue, the device driver, or the disk itself. Luna: So that's the nuclear option. What's the one thing you want listeners to take away from this episode? Lucas: Add disk latency to your monitoring today. At minimum, set up a cron job that runs 'iostat -x 1 5' every hour and logs it. When you have a performance incident six months from now, you'll have the data to know whether it's a storage problem or not. Most teams only realise they needed that data after the fire is already burning. Luna: Good advice. Thanks Lucas. And thanks to everyone listening. We'll be back next week with another episode.