Glossary
Storage latency
Storage latency is the time between an application issuing a storage request and receiving the response, measured for each operation. It is the measure users notice most directly. It is reported as an average and as percentiles, and the slowest percentiles usually say more about experience than the average does.
Averages and percentiles
An average blends fast and slow requests into one calm number. Take 1,000 requests in which 980 finish in 2 milliseconds and 20 take 200 milliseconds. The average comes out near 6 milliseconds, which looks healthy. The 99th percentile, written p99, is 200 milliseconds. An application that issues dozens of storage requests to build one page or complete one transaction meets the slow ones constantly, so users report delays while the dashboard average stays flat. Monitoring and service targets that use p99 or p99.9 catch what the average hides. Percentiles also need a time window: a p99 taken over a whole day can bury a fifteen-minute spike that a five-minute window shows clearly, and that spike is often when the complaints arrive.
Which layers add to it
Each request accumulates time in several places. The media contributes microseconds on NVMe flash and milliseconds on hard drives. The network adds a fraction of a millisecond inside a data center and far more between distant sites. Writes add the time to store every protected piece before confirming, as described under read/write performance, plus a round trip when a second site must hold a synchronous copy. On top of all of these sits time spent waiting in a queue.
Why does latency climb with load?
Queueing is the layer that changes most. At low utilization a request rarely waits behind another. As a device or link approaches saturation, waiting time grows steeply rather than in proportion, so a modest increase in load can multiply response times. A system that answers quickly in an idle test can answer slowly at the busiest hour on the same hardware. Sizing for headroom at peak keeps a system out of that steep region, and storage QoS stops one workload from pushing others into it.
What a value means
A latency figure is judged against the application waiting on it. Databases and virtual machines notice single-digit milliseconds. Backup and archive jobs tolerate far more, because they depend on throughput, and spending to lower their latency changes little. A latency figure also needs the load at which it was taken: the IOPS rate and the p99 together describe an operating point, and either alone does not.
Frequently asked questions
Does flash remove latency problems?
It removes most of the media delay, but queueing, network distance and write protection remain. A flash system run close to saturation can show worse p99 latency than a lightly loaded disk system.