Glossary

Storage Latency

Storage latency is the time between issuing a request and getting the answer back. It is measured per operation, usually in microseconds or milliseconds, and it is a different thing from throughput — a system can move enormous volumes of data and still respond slowly.

Most of it is not the storage media. It is queueing, network round trips and software layers, and knowing which is which is what makes latency fixable.

Latency, throughput and IOPS are three different questions

These get used interchangeably and they measure different things.

Latency is how long one operation takes. Throughput is how many bytes per second the system moves. IOPS is how many operations per second it completes.

They are not the same, and improving one often costs another. A system can be tuned for throughput by batching requests together — larger, more efficient transfers, and every individual request waits longer. Tape is the extreme case: excellent throughput once it is streaming, and latency measured in minutes because a robot has to fetch a cartridge.

Which one matters depends entirely on the workload. A database doing small random reads is latency-bound; a backup job writing a 10 TB stream is throughput-bound. Tuning for the wrong one makes things worse.

Where the time actually goes

A read request passes through several layers, each adding time. The useful exercise is knowing roughly what each one costs, because that tells you which one is worth attacking.

Two things fall out of this table. First, on a hard drive the media is the bottleneck and nothing else comes close. Second, on flash the media is often the smallest component — a 60-microsecond NVMe read behind a 400-microsecond network round trip means the drive accounts for about a seventh of what the application experiences. Buying faster drives to fix that is buying the wrong thing.

The floors you cannot move

Some latency is a property of physics or mechanics and no product removes it.

A 7,200 RPM hard drive completes one rotation in 8.3 milliseconds. On average, the sector you want is half a rotation away, so rotational latency alone averages about 4.2 milliseconds before the head has even moved. Add seek time and a random read on spinning media lands in the 5–15 millisecond range regardless of what is in front of it.

Light in fibre travels at roughly 200,000 km per second — about 5 microseconds per kilometre. A round trip to a site 100 km away therefore costs about 1 millisecond in propagation alone, and a request to a data centre on another continent cannot be answered in less than tens of milliseconds no matter what the storage does. This is why a fast storage system in the wrong place feels slow, and why caching close to the application beats optimising the far end.

Everything else — queue depth, protocol overhead, filesystem layers, retries — is engineering, and can be reduced.

Averages hide the problem

Average latency is the least useful number in storage monitoring, because it is dominated by the many fast operations and says nothing about the slow ones.

Percentiles are what matter. p50 is the median. p99 means one request in a hundred is at least this slow. p99.9 means one in a thousand. A system averaging 2 ms with a p99 of 200 ms is not a fast system — it is a system where a hundredth of requests take a fifth of a second, and users will notice that far more than they notice the median.

Tail latency matters more than it first appears because of fan-out. If a single user action triggers 100 storage requests and waits for all of them, the odds that at least one hits the p99 are high — so the user's experience is closer to the p99 than to the median. The more a system parallelises, the more the tail governs what people actually feel.

Tails come from specific causes worth chasing individually: a drive that is failing but has not been removed, garbage collection on an SSD, a rebuild competing for the same devices, or queueing during a burst.

Why latency explodes near saturation

The single most common latency problem is not slow hardware. It is queueing.

When a device is busy, incoming requests wait. As utilisation rises, latency rises with it — but not linearly. It climbs gently up to about 70% utilisation, then rises steeply, and near 100% it goes vertical. A device at 95% utilisation can show latency an order of magnitude worse than the same device at 70%, with identical hardware and identical requests.

This is why a system that has been fine for months degrades suddenly rather than gradually: it crossed the knee of the curve. It is also why headroom is not waste. Running storage at 60–70% of capability leaves room to absorb bursts and background work without the queue building.

Two practical consequences. Watch queue depth, not just utilisation — a rising queue is the leading indicator, and latency is the lagging one. And be careful with background tasks: rebuilds, scrubs and lifecycle transitions all consume device time, and scheduling them into a busy period is a self-inflicted latency incident.

What actually reduces it

In rough order of how much they usually help:

• Cache closer to the application. A read served from memory is a hundred-microsecond problem turned into a hundred-nanosecond one. This is the largest single lever available.
• Reduce distance. If the physics floor is the problem, no tuning helps — the data or a copy of it has to move nearer the compute.
• Keep utilisation off the knee. Capacity headroom is latency headroom.
• Remove layers. Each protocol hop, gateway and abstraction adds time. A native client that talks directly to storage nodes avoids a round trip that a gateway architecture cannot.
• Match media to workload. Flash for anything latency-sensitive, spinning disk for capacity and streaming. Mixing them under one policy gives you the worst of both.
• Parallelise. This does not reduce per-operation latency, but it stops one slow operation blocking everything behind it.

What does not help: buying faster drives when the network or the queue is the bottleneck. Measure where the time is going before spending anything.

Latency in object storage specifically

Object storage adds two things to the picture. Requests arrive over HTTP, which means TCP and TLS setup unless connections are reused — persistent connections are worth having, because a handshake per request can cost more than the storage operation. And every request involves a metadata lookup to resolve the key to a location before any data moves.

The other consideration is object size. Small objects are dominated by fixed per-request overhead, so latency per object is roughly constant regardless of how small they get, and thousands of tiny objects will always be slower than one large one holding the same bytes. Large objects are dominated by transfer time, where throughput matters more than latency.

This is why object storage is a good fit for backup, archive and large-file workloads, and a poor fit for the small random reads a transactional database issues.

What Scality measures

Scality publishes measured figures rather than round numbers: 511 microseconds for a GET and 741 microseconds for a PUT. Those are useful precisely because they are specific — they describe a tested configuration rather than a marketing claim, and they sit where you would expect for flash-backed object storage over a local network.

The general caution applies here as much as anywhere: treat any unqualified latency claim with suspicion, from any vendor. A latency number without the object size, the queue depth, the media and the network it was measured on describes nothing you can plan against.