Skip to content 
I show step by step how the I/O wait analysis with iostat and vmstat makes bottlenecks visible and which Linux server metrics count for fast response times. In doing so, I set clear thresholds, interpret typical patterns and suggest concrete measures for optimizing I/O and CPU in.
Key points
- iostat and vmstat provide a complementary view of CPU and storage load.
- wa via 15-20% and %utile via 80% show an I/O bottleneck.
- await and avgqu-sz explain latency and queues.
- mpstat detects unevenly distributed load across CPU cores.
- Tuning ranges from MySQL to kernel parameters and storage.
What does I/O Wait mean on Linux servers?
Under I/O wait, the CPU waits idly because it is waiting for slower memory or network devices, which is known as wa-value in tools such as top or vmstat. I evaluate this percentage as the time in which threads block and requests are completed later, which users directly experience as sluggishness. Values above 10-20% often indicate an exhausted Storage-subsystem, for example when HDDs, RAID arrays or NFS mounts are at capacity. In hosting setups with databases, unindexed queries and write peaks add up to unnecessary waiting times on the Disc. If you want to brush up on the concepts, you can find the basics at Understanding I/O Wait, before I go to the practice.
Quick start: read vmstat correctly
With vmstat, I can check the most important Linux-key figures and recognize initial patterns without much effort. The vmstat 1 10 call provides ten snapshots in which I pay particular attention to the wa (I/O wait), bi/bo (block I/O) and si/so (swap) columns. For me, high bo values with simultaneously increasing wa indicate many blocking write accesses, which often indicates buffer limits or slow media. If si/so remains at zero, but wa rises significantly, this borders the suspicion more strongly on pure Storage-limit. In multi-core hosts, I combine vmstat with mpstat -P ALL 1, because I/O wait often only affects individual cores and therefore appears more harmless on average than it actually is.
CPU fine image: us/sy/st, run queue and context switch
With vmstat and mpstat I read more than just wa: High usThe "computing-heavy" application work is shown in the following sections, sy indicates kernel/driver work, for example with intensive I/O. In virtualized environments I pay attention to st (Steal): High st values mean that the VM loses CPU time, which artificially inflates latencies with identical I/O patterns. I also compare the run queue (r in vmstat) with the number of CPUs: A permanently higher r than the number of CPUs indicates congestion at the CPU, not at the Storage. Many context changes (cs) in combination with small synchronous writes are an indicator of chatty I/O patterns. This way I avoid misinterpreting CPU scarcity as an I/O problem.
Understanding iostat in depth: important metrics
iostat -x 1 gives me extended Disc-metrics that cleanly describe latency, utilization and queues. I start the measurement for load peaks and correlate %util, await, svctm and avgqu-sz to distinguish cause and effect. If %util rises to 90-100%, while await and avgqu-sz also go up, I see a saturated Plate or a limited volume. If await shows high values with moderate %util, I check for interference from caching, controller limits or isolated large requests. r/s and w/s bring frequency into the picture, while MB_read and MB_wrtn explain the volume, which provides me with important comparative values for dedicated SSD and NVMe setups.
NVMe, SATA and RAID: what %util, await and queue depth mean
I make a strict distinction between media types: HDD show higher await-values even with a moderate cue, because head movements dominate. SSD/NVMe scale well with parallelism, which is why a higher avgqu-sz can be normal as long as await remains within limits. On NVMe with multiple submission/completion queues, I read %util more cautiously: Individual devices can already be at the limit at 60-70% if the app does not generate enough parallelism or the queue depth (nr_requests, queue_depth) is too small. In the RAID I check whether stray random I/O encounters stripe sizes that are too small; then the await and %utile unevenly on the member disks. I therefore look at iostat per member device, not just on the composite volume, to make hotspots visible. For log-structured layers (e.g. copy-on-write), I expect slightly higher latencies for writes, but compensate for this with enlarged writeback windows or app-side batching.
Diagnostic workflow for long waiting times
I start each analysis in parallel with vmstat 1 and iostat -x 1 so that I can see CPU states and device statuses synchronously and assign them to time periods. I then use mpstat -P ALL 1 to verify whether individual cores are running unusually high. wa which prevents incorrectly interpreted mean values. If there are indications of a cause, I use pidstat -d or iotop to see exactly which PID is using the most I/O shares. In hosting environments with databases, I first differentiate read peaks from write peaks, as write-back strategies and fsync patterns generate very different symptoms and thus enable targeted Measures make it possible. For more in-depth storage questions, an overview like the one at I/O bottleneck in hosting, before I turn the kernel or file system screws.
Clearly delineating virtualization and containers
In VMs I consider wa together with st (Steal) and additionally measure on the hypervisor, because only there the real devices and Cues are visible. Storage aggregations (thin provisioning, dedupe, snapshots) move latency peaks down into the stack - in the VM, this has the following effects await jumps, while %util remains inconspicuous. I limit or decouple in containers I/O with Cgroup rules (e.g. IOPS/throughput limits) in order to Noisy Neighbors to tame them. I document the limits per service so that measured values are reproducible and alarms retain their context. Important: A high wa in the VM can be triggered by host-wide backups, scrubs or rebuilds - I correlate times with host jobs before touching the app.
Limits, thresholds and next steps
I use a few clear thresholds to decide whether there is a real bottleneck and what action to take to noticeably stabilize performance. I take into account the type of media, workload and latency requirements, because the same figures on HDD and NVMe have different implications. I use the following table as a quick guide that I use in my playbooks. I measure several times over minutes and under load so that outliers don't generate false alarms and I can recognize trends. I use this as a basis for targeted action instead of reflexively replacing hardware or Parameters massively.
| Metrics | Threshold | interpretation | Next steps |
|---|---|---|---|
| wa (vmstat) | > 15-20% | Significant I/O waiting time | Check iostat; find the cause with pidstat/iotop; check caching and writes |
| %utile (iostat) | > 80-90% | Device utilized | correlate await/avgqu-sz; check queue depth, scheduler, RAID and SSD/NVMe |
| await (ms) | > 10-20 ms SSD, > 30-50 ms HDD | Increased latency | Differentiate between random vs. sequential; customize file system options, writeback, journaling |
| avgqu-sz | > 1-2 persistent | Queue grows | Throttle/increase parallelism; optimize I/O pattern of the app; check controller limits |
| si/so (vmstat) | > 0 under load | Storage bottleneck | Increase RAM; query/cache tuning; check swappiness/memory limits |
Causes in the stack: DB, file system, virtualization
With databases, I often see unindexed joins, buffers that are too small and excessive fsync calls as the actual Causes for high await values. I check query plans, activate logs for slow statements and adjust sizes such as InnoDB buffer pool, log file sizes and open files objectively. At file system level, I look at mount options, journal modes and alignment to the RAID stripe, because the wrong combinations cause waiting times to explode. In virtualized setups, I don't rely on measurements in the VM alone, but look at the host, because that's where the real block devices and Cues become visible. This allows me to clearly separate the effects of deduplication, thin provisioning or neighboring VMs from the application patterns.
File system and mount options in detail
I evaluate file systems in the light of the workload: ext4 in ordered or writeback mode minimizes barriers to write intensity, while XFS scores with its log design for parallel workloads. Options such as noatime or relatime reduce metadata writes, lazytime moves timestamp updates to the writeback in bundles. For journaling, I check the commit intervals (e.g. commit=) and check whether force flushes (barriers) are exacerbated by controller cache policies. On RAID I pay attention to clean alignment to the stripe (Parted/FDISK with 1MiB start), otherwise await by Read-Modify-Write even with supposedly sequential patterns. For databases, I often use O_DIRECT or direct log flush strategies - but only after measurement, because the file system cache can dramatically reduce the read load if the working set fits into it.
Tuning: from the kernel to the app
I start with simple wins, for example query indexing, batch writing and sensible connection pooling configuration, before I start at system level. For writeback, I adjust vm.dirty_background_ratio, vm.dirty_ratio and vm.dirty_expire_centisecs in a controlled manner so that the system writes contiguously and generates less blocking without clogging memory. On block devices, I check I/O scheduler, queue depth and read-ahead because these controls directly shape latency and throughput. On RAID controllers, I tune stripe size and cache policy, while on SSD/NVMe for firmware, TRIM and consistent overprovisioning settings. After each change, I verify with vmstat and iostat over several minutes whether await drops and %util remains stable before moving on to the next step.
Interrupts, NUMA and affinities
I monitor IRQ load and NUMA topology because both have a noticeable effect on latencies. NVMe-I bind interrupts to the CPUs of the controller's NUMA domain via affinity so that data paths remain short. If the IRQ storm is running on a core, I see high sy and the rest of the CPUs appear to be idle; mpstat exposes this. I only allow irqbalance if the distribution matches the hardware - otherwise I set specific affinities. I also check whether the application and its I/O work in the same NUMA zone (storage location), because cross-socket accesses cause waiting times in await can be masked.
Automate measurement and make it visible
To make sure I recognize trends, I automate measurements and integrate iostat/vmstat into monitoring tools, which can display historical data. Data save. I set alarms conservatively, for example from wa > 15% over several intervals, combined with thresholds for await and %util to avoid false alarms. For overall metrics screens, I use dashboards that juxtapose CPU, storage, network and app metrics so that correlations are immediately visible. If you need an introduction to metrics, you can find them at Server metrics compact context for daily work. What is important to me is a repeatable process: measure, form a hypothesis, make adjustments, measure again and repeat the results. Results document.
Reproducible load tests with fio
If I lack a real load or want to verify hypotheses, I use short-lived fio-tests. I simulate representative patterns (e.g. 4k random read, 64k sequential write, mixed 70/30 profiles) and vary iodepth, to set the sweet spot window between await and throughput. I strictly separate test paths from production data and note boundary conditions (file system, mount options, scheduler, queue depth) so that I can classify results correctly. After tuning, the same tests are used as a regression check; only when await and %utile consistently look better, I apply changes to the surface.
Recognizing error patterns: typical patterns
If I observe high wa with simultaneously high %utile and increasing avgqu-sz, everything speaks for saturation on the Device, i.e. real physical limits. High await values with moderate %util tend to indicate controller or caching peculiarities, such as barriers, write-through or sporadic flushes. Rising si/so values together with dips in the cache clearly indicate a lack of RAM, which artificially inflates I/O and increases waiting times. If the disk remains inconspicuous, but the app frames large, sync-heavy writes, I shift the work to asynchronous writing, pipelining or Batch-mechanisms. In NFS or network storage setups, I also check latency, MTU, retransmits and server-side caching, because network time is directly masked as I/O latency here.
NFS/iSCSI and distributed storage
At NFS and iSCSI, I differentiate between device and network path: iostat only shows what the block layer sees - I also detect retransmits, latencies and window problems via network metrics. High await with moderate %utile on the virtual block device is typical when the network stalls or the server-side cache cools down. For NFS I check mount options (rsize/wsize, proto, sync/async) and the server side (threads, export policies, cache), for iSCSI the session and queue parameters. I schedule maintenance windows for server jobs (scrubs, rebuilds, rebalancing) so that they don't saturate the shared storage at peak times and so wa on all clients.
Practical examples: three short scenarios
Blog cluster with writing tips
At prime time, comments and invalidate caches increase, whereupon await and avgqu-sz in iostat increase significantly, while %util sticks to 95%. I switch writeback parameters slightly higher, optimize cache invalidation at path level and strengthen the InnoDB log buffer so that there are fewer small sync writes. After that, await drops measurably, bo values remain high, but wa drops, which immediately reduces response times. At the same time, I replace individual HDDs with SSDs for the journal, which additionally relieves the bottleneck. The pattern shows how Batch-Combine writing and faster journaling.
Store with reading peaks and search indices
Promotions generate heavy read load, r/s shoots up, await remains moderate, but the app still reacts sluggishly to filter navigation. I recognize many unbuffered queries without suitable indexes that exceed the file system cache working set. With targeted indexing and query rewrite, r/s drops, the hits are more often in the cache, and iostat confirms lower MB_read with the same throughput. At the same time, I increase read-ahead slightly to serve sequential scans more efficiently, which further reduces latencies. This is how I check that Read-patterns lead to cache hits again.
VM host with „Noisy Neighbor“
In individual VMs, top shows high wa, but iostat in the VM only sees virtual devices with misleading utilization. I additionally measure on the hypervisor, see saturated real block devices and identify a neighbor VM with aggressive backups as the cause. Due to bandwidth limits and changed backup windows, await and %util stabilize throughout the host. I then measure again in the VM and on the host to confirm and prevent the effect on both sides. This confirms that real Devices-metrics always show the truth at the host.
Checklist for the next incident
I start logs and measurements first so that no signals are lost, and keep vmstat 1 and iostat -x 1 running for several minutes. Then I time-correlate peaks with app events and system timers before pinning down individual processes with pidstat -d and formulating hypotheses. The next step checks memory, swap and cache hits so that RAM shortages are not mistaken for Disc-problem appears. Only when I have isolated the cause do I change exactly one thing, log the setting and evaluate the effect on await, %util and wa. In this way, I keep the analysis reproducible, learn from every incident and reduce the time until the Solution clearly.
Frequent misinterpretations and stumbling blocks
I am not fooled by isolated peaks: Single seconds with high wa are normal, only persistent plateaus indicate a structural bottleneck. %utile close to 100% is only critical if await goes up at the same time - otherwise the device is simply busy. On SSD/NVMe is a higher avgqu-sz often intentional in order to utilize internal parallelism; I only throttle when latency targets are missed. I check CPU frequency scaling: Aggressive power saving can increase response times and thus reduce latency. wa seem to worsen. And I separate application TTFB from storage latency - network, TLS handshakes and upstream services can produce similar symptoms without iostat „is “guilty".
Brief summary for admins
The I/O wait analysis with iostat and vmstat works quickly when I read wa, await, %util and avgqu-sz together and relate them to workload context. I first identify whether there is real device saturation or whether memory and app patterns are driving the latency, and then select the appropriate lever. Small, targeted adjustments to queries, writeback parameters, schedulers or queue depth often have the greatest effect before expensive hardware changes are necessary. Measurement, hypothesis, change and re-measurement remain my fixed sequence so that decisions remain comprehensible and repeatable. This is how I keep Linux-server is responsive and ensures noticeably better Response times under load.
