Glossary
Storage performance tuning
Storage performance tuning is the repeated work of finding what limits a storage workload and removing it, through changes to the application, client, network or storage system. It runs as a measured loop rather than a list of settings, and it ends where a physical limit is reached.
The tuning loop
- Take a baseline under production-like load, recording latency at the 99th percentile as well as the average, alongside the operation rate and data rate. The busiest hour gives the most useful baseline. The measures themselves are described under storage performance.
- Locate the narrowest stage. A request passes through the client, the network, the storage servers and the media, and the stage that is saturated while the others wait is the one worth changing.
- Change one thing at that stage, and only one, so any gain or loss can be attributed.
- Repeat the same test under the same load and compare the result with the baseline.
- Keep the change or revert it, write down what happened, and start again from the new baseline, because removing one limit moves the bottleneck somewhere else.
Why the client is checked first
An application that sends one request, waits for the answer and only then sends the next leaves a large storage system mostly idle, however capable it is. Very small requests have a similar effect, since each one carries fixed overhead that moves no data. Raising the number of requests in flight and the size of each request is frequently the first change that moves a result, and it costs nothing in hardware. How the application orders its reads and writes, covered under sequential vs random I/O, often outweighs any setting on the storage side.
What tuning cannot change
Tuning removes avoidable limits. It cannot widen a physical one. A network port has a maximum rate, and once a client fills it, more throughput needs more ports or more servers. A hard drive delivers a roughly fixed number of scattered operations per second whatever it is told. At that point the answer is hardware, and in scale-out storage that means adding servers that bring their own drives and links.
How tuning differs from benchmarking and QoS
Benchmarking produces a comparable measurement under fixed rules. Tuning uses measurements to change one particular system for one particular workload. On shared storage, a gain for one application can come out of the headroom of others, and storage QoS is the control that decides how that headroom is divided.