Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Disk I/O Tuning with IO Schedulers
Transcript
- Lucas: You've got a production Linux server that's feeling slow — not cpu starved, not out of memory, but disk waits are climbing. Before you throw hardware at it, there's a kernel-level knob that can dramatically change how your disk behaves: the I/O scheduler. Luna: I/O scheduler — isn't that something most people just leave on the default? I remember seeing 'cfq' or 'deadline' in kernel docs but never really understood when to care. Lucas: Exactly, and that's the problem. The default scheduler your distro picks might be optimized for a 2005 era spinning hard drive, not the NVMe or virtual disk you're actually running. And the wrong scheduler can add tens of milliseconds of latency per I/O, which adds up fast. Luna: If today's conversation gives you something you can use right away, that's the whole point of this show — and the reason we keep it ad-free is listener support. If you found it useful, buy me a coffee dot com slash fexingo helps us keep doing these deep dives. Lucas: Yeah, absolutely. So let's decode the three main schedulers in the modern kernel. There's CFQ — Completely Fair Queuing — which tries to give every process a fair share of I/O time. Then there's Deadline, which prioritizes latency by setting expiration times on requests. And finally, noop, which is basically a simple FIFO queue with some merging. Luna: And noop is what you'd typically want for SSDs or virtual disks, right? Because the hardware itself handles the reordering. Lucas: Right. Noop is the least overhead — it just passes requests through with minimal logic. For a raw NVMe drive or a san backed LUN, noop is often the best choice because the device has its own scheduler. But for a traditional spinning hard drive, noop can actually hurt because it doesn't try to minimize seek times. Luna: So where does Deadline fit in? I've seen that recommended for database servers. Lucas: Deadline is great when you have mixed workloads — some reads, some writes — and you care about read latency. It puts a 500-millisecond expiration on read requests and five seconds on writes by default. If a read is about to expire, it jumps the queue. That prevents 'starvation' where a flood of writes pushes reads to the back forever. Luna: And CFQ? That's the old default on many distros, but I've heard it's being phased out. Lucas: CFQ is indeed legacy. It was replaced by 'bfq' — Budget Fair Queuing — in some newer kernels, but many enterprise distros still ship CFQ. The idea is fairness: each process gets a time slice for I/O. But the overhead is high, and for most modern workloads, it's not ideal. If you have a busy web server with lots of concurrent connections, CFQ can actually cause latency spikes. Luna: So how do you check which scheduler you're currently using? I assume there's a sysfs file. Lucas: Yeah, it's straightforward. For a given block device, say /dev/sda, you can cat /sys/block/sda/queue/scheduler. That will show you the available schedulers in brackets, with the current one in square brackets. So you might see 'noop cfq' — meaning deadline is active. Luna: And to change it on the fly without a reboot? Lucas: Just echo the scheduler name into that same file: echo deadline > /sys/block/sda/queue/scheduler. It changes instantly. That's great for testing — you can switch, run your benchmark, switch back. Luna: But that change won't survive a reboot. How do you make it permanent? Lucas: Two main ways. One is via the kernel boot parameter 'elevator=' — you add that to your GRUB config. But that sets it for all block devices. The more granular approach is a udev rule. You create a file like /etc/udev/rules.d/60-iosched.rules with something like: ACTION=='add|change', KERNEL=='sd*', ATTR{queue/scheduler}='deadline'. That matches any SCSI disk. Luna: Nice. And you can get even more specific with ATTR{size} or by matching the device by its ID. Lucas: Exactly. So let's talk about a concrete scenario. Say you have a PostgreSQL database server on a machine with a RAID10 array of spinning drives. What scheduler would you pick? Luna: I'd lean toward Deadline, because databases are read-sensitive and you don't want a write burst to delay critical queries. Lucas: That's the textbook answer. But what if that same server also has an SSD for the WAL log? You'd probably want noop for the SSD, because the SSD's internal FTL handles ordering. And Deadline for the spinning data drives. Luna: So you'd have two different schedulers active on the same machine, one per device. That's fine with udev rules matching by device path. Lucas: Right. And this is where knowing your workload matters. There's no universal 'best' scheduler. You have to test. The good news is that the sysfs interface makes A/B testing trivial. You can run fio or a synthetic benchmark, switch schedulers, and compare latencies and throughput. Luna: What about multi-queue block layer? I know newer kernels have blk-mq, and that changes the scheduler landscape. Lucas: Great point. Since kernel 5.0, the single-queue I/O schedulers are legacy for most devices. With blk-mq, the available schedulers are typically 'none' — which is like noop — 'kyber', which is a latency-oriented scheduler, and 'bfq', which is the fairness one. Kyber is interesting because it dynamically adjusts queue depth to meet latency targets. Luna: So for a modern NVMe drive, you'd likely use 'none' or 'kyber'? Lucas: For NVMe, 'none' is common because the hardware is so fast that any scheduler overhead just slows things down. But 'kyber' can help if you have mixed workloads and want to protect read latency. The key is to verify with your actual workload. Luna: Let's talk about tuning the parameters within a scheduler. For Deadline, you can adjust the read and write expiration times, right? Lucas: Yes. Under /sys/block/sda/queue/iosched/, there are files like 'read_expire' and 'write_expire'. Default is 500 ms for reads, 5000 ms for writes. If you have a time-sensitive application, you might lower read_expire to 200 ms. But be careful — too aggressive can cause more seeking on spinning rust. Luna: And for CFQ or BFQ, you can adjust slice_idle and quantum. Those control how long a process gets to submit I/O before being preempted. Lucas: Right. But honestly, if you're tuning CFQ or BFQ, you're already in a niche use case — like a desktop system trying to keep UI responsive during heavy writes. On a server, Deadline or none/kyber is almost always the better starting point. Luna: What about cloud instances? I've spun up VMs on AWS and GCP and never once thought about the I/O scheduler. Lucas: That's actually a common pitfall. Many cloud images default to CFQ or deadline. But since the underlying storage is network-attached or virtualized, the guest scheduler has no idea about the actual physical layout. In that case, noop or none is usually best, because the hypervisor or storage array does the ordering. Luna: So a quick win for anyone running Linux in the cloud: check your scheduler and likely switch to noop or none. Lucas: Exactly. And if you're on a major cloud provider, they often have documentation recommending the scheduler. AWS, for example, suggests noop for EBS volumes on Nitro instances. But many people never apply that. Luna: Let's also touch on how to measure whether your change made a difference. What tools do you use? Lucas: iostat is the go-to. Look at 'await' — the average service time per I/O. If you switch from CFQ to deadline and see await drop from 10 ms to 2 ms, that's a win. Also watch '%util' — but be careful, 100% util doesn't necessarily mean saturation with modern drives. Luna: And for more granular latency percentile data, you'd use something like 'bcc' tools or 'iostat -x' with extended stats. Lucas: Yeah. The 'biosnoop' tool from BCC traces individual I/O completions and can show you the distribution. That's invaluable for spotting outliers. Luna: So to wrap up: the I/O scheduler is a low-risk, high-impact tuning parameter. It's easy to check, easy to change, and you can test instantly. Don't leave it on the distro default without thinking about your workload. Lucas: And if you take one thing away from today: know what scheduler your disks are using, and have a reason for it. That's the difference between hoping your server is fast and knowing why it is.