Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Benchmark Linux Server Disk I/O with FIO
Transcript
- Lucas: If you've ever had a database server that just felt slow — queries taking twice as long as they used to, no obvious CPU or memory pressure — the first thing I now check is disk I/O. Not disk space, not SMART stats. Actual performance under load. Luna: And you're not talking about a simple 'dd' test, right? Because 'dd' gives you sequential throughput, which is almost never what your workload looks like. Lucas: Exactly. 'dd' is fine for a quick sanity check — can the disk even write? — but it won't tell you about IOPS, latency distribution, or how the drive behaves under queue depth. For that, you need FIO — Flexible I/O Tester. Luna: FIO is basically the industry standard for storage benchmarking. It's what storage vendors use when they publish their 'up to X IOPS' numbers. Lucas: Right. And the beauty is it ships in almost every distro's repos. On Debian or Ubuntu it's 'apt install fio'. On RHEL or Rocky it's 'dnf install fio'. You're usually one command away from a real benchmark. Luna: But FIO has a million knobs. If you just run 'fio' with no arguments, it'll complain. So what's the minimal useful test? Lucas: Let's start with a random-read test that mimics a typical database workload — say, 4-kilobyte blocks, queue depth 32, direct I/O to bypass the page cache. I use this all the time: 'fio --name=randread --ioengine=libaio --iodepth=32 --rw=randread --bs=4k --direct=1 --size=1G --numjobs=1 --runtime=30 --group_reporting'. Luna: So that's a 1-gigabyte file, random reads in 4-kilobyte chunks, direct I/O, queue depth 32, runs for 30 seconds. What are you looking for in the output? Lucas: Three numbers. First, IOPS — 'read: IOPS=...' — that tells you how many operations per second the device can sustain. Second, bandwidth — usually in MiB/s. Third, and this is the one people miss: the latency percentiles. FIO prints 50th, 90th, 99th, and 99.9th percentile latencies. If your 99th percentile is over 10 milliseconds on an NVMe drive, something's wrong. Luna: And by wrong you mean — controller saturation? Bad firmware? Or a shared backplane? Lucas: Any of those. Or a failing drive that still passes SMART. I once spent a week tuning PostgreSQL queries that were actually slow because the RAID controller was caching writes badly. FIO caught it in two minutes. The 99th percentile latency was 120 milliseconds on a drive that should do under 1 millisecond. Luna: So FIO exposed the hardware lie. Let's talk about matching the test to the workload. If your server runs a video transcoding pipeline, random 4K reads aren't the right test. Lucas: Right. For sequential workloads — like media streaming or log shipping — you want '--rw=read' or '--rw=write' with a larger block size, say 64K or 1M. And for mixed workloads, FIO has a '--rw=randrw' option with a '--rwmixread' percentage. I usually do 70 percent read, 30 percent write for a general database profile. Luna: And you mentioned queue depth. Why is that important? If you set iodepth to 1, you're testing single-threaded latency. At depth 32, you're testing how the device handles concurrency — which is closer to real-world server load. Lucas: Exactly. A lot of cheap SATA SSDs can do 10,000 IOPS at queue depth 1, but at depth 32 they barely budge — their controller just can't parallelize. Good NVMe drives, on the other hand, scale linearly up to queue depth 128 or higher. If you see IOPS plateauing early, you've hit the controller's limit. Luna: So for a baseline, you'd run the same test on a known-good drive — maybe a local NVMe — and compare? What's a realistic number? Lucas: Consumer NVMe drives — like a Samsung 980 Pro — can do around 500,000 random read IOPS at queue depth 32. Enterprise drives like a Kioxia CM6 can push 1 million. SATA SSDs top out around 80,000 to 100,000 IOPS. If your server's NVMe is only giving you 200,000 IOPS, that's a flag. Luna: And spinning rust? I've got old servers with 15K SAS drives that are still in production. Lucas: Those top out around 200 to 300 random IOPS per spindle. If you see 400 IOPS on a single drive, that's actually good. But the latency will be high — 99th percentile often above 20 milliseconds. That's why databases on HDDs use aggressive caching. Luna: So you've run your FIO test. You have IOPS, bandwidth, latency. What's the next step if the numbers are bad? Lucas: Check if the test itself is the problem. Run it three times — same parameters, same file size — and take the median. If all three are consistent and bad, then start isolating: is it the drive, the controller, the cable, the filesystem? Change one variable at a time. Luna: And make sure you're testing on a partition that isn't mounted with 'noatime' or some mount option that could skew results? Lucas: Actually, 'noatime' helps reduce write overhead, but for a read test it doesn't matter. What does matter is using '--direct=1' to bypass the page cache. Otherwise, FIO reads from RAM after the first pass, and you're benchmarking memory, not disk. Luna: One more gotcha: if your test file is smaller than the drive's DRAM cache — say 1GB on a drive with 2GB of cache — you're testing cache hits, not raw performance. Lucas: Good point. Use a file size at least twice the drive's cache. For enterprise drives with 4GB cache, I use 8GB or even 16GB. And set '--size=8G' and let FIO create a new file each run. Luna: So you've got your baseline. Now you can tune filesystem parameters — like 'noatime', 'nodiratime', barrier settings — and re-run the exact same test to see if it helped. Lucas: Exactly. FIO gives you a reproducible, numerical way to evaluate every change. And since it's open source and on every server anyway, there's no excuse not to have a baseline for every production box. Luna: Speaking of open source and tools we rely on every day — I think it's worth mentioning that the reason we can do episodes like this, diving into a specific tool without padding it with ads, is that the show is supported directly by listeners. Lucas: Yeah, we keep these episodes ad-free on purpose. If you find the content useful — maybe you just saved an hour debugging with FIO — there's a link at buy me a coffee dot com slash fexingo. No pressure, just a way to keep the show going without sponsors. Luna: Exactly. And it means we can cover niche sysadmin topics without worrying about advertiser appeal. Now, back to that baseline — once you have it, what's the most common surprise you see? Lucas: The biggest one is discovering that your 'fast' cloud instance has a shared disk that is quietly throttling. FIO on an AWS 'gp3' volume with baseline 3,000 IOPS should hit that number. If it only does 2,500, and you're paying for provisioned IOPS, that's a problem to escalate. Luna: Or the classic: a SAN LUN that's oversubscribed. FIO shows latency spikes during business hours, but fine at 3 AM. Lucas: Right. Run FIO at different times of day. I've automated it with a cron job that runs a 60-second test every hour and logs the results to a file. When users complain about slowness, I have historical I/O performance to compare against. Luna: That's a good practice. Do you use a job file or just inline arguments? Lucas: For automation, job files are cleaner. You define the parameters in a.fio file, then invoke 'fio mytest.fio'. It makes the test repeatable and version-controllable. I've got one for random read, one for sequential write, one for mixed — all in a git repo. Luna: And the output — do you pipe it to a parser? Because the default output is human-readable, not machine-friendly. Lucas: FIO has a '--output-format=json' flag. You can pipe that into jq and extract the exact fields you want: elapsed, iops, latency percentiles. I have a script that sends a summary to our monitoring system every hour. Luna: So you've turned a one-off debugging tool into a continuous monitoring system. That's the sysadmin way. Lucas: Exactly. Because the worst time to discover your disk is slow is during an outage. Run FIO now, when everything's fine. Know your baseline. Then when something feels off, you're not guessing. Luna: And if you want to share your own FIO job files or gotchas, we'd love to hear them. In the meantime, the next time a database seems sluggish, you know where to start. Lucas: Exactly. FIO, four parameters, thirty seconds. It'll tell you more than an hour of guessing.