Glossary
Storage performance degradation
Storage performance degradation is a decline in the performance a storage system delivers (higher latency, lower IOPS or lower throughput) compared with its earlier behaviour under the same workload. Some causes build slowly as a system fills and load grows; others arrive at once, when a component fails or a background process starts.
Why performance degradation matters at scale
Large platforms rarely fail by stopping. They slow down. A backup that used to finish at 4 a.m. finishes at 7, a nightly ingest runs into the business day, and an AI pipeline that kept its GPUs fed starts leaving them idle. Because the decline is gradual, the consumers of the platform often notice it before any dashboard does, and by then the remedy is a hardware purchase on a short lead time.
Some sources of degradation are permanent features of operating at scale. A cluster with tens of thousands of drives loses drives every week, so the system spends a real share of its life rebuilding.
Gradual and acute causes
| Cause | Onset | Measurement most affected |
|---|---|---|
| Rising capacity fill | Gradual | Write latency, sequential throughput |
| Load growth and rising utilisation | Gradual | Latency at every percentile |
| Working set outgrowing cache | Gradual | Read latency |
| Metadata growth | Gradual | Lookups, listings, small-object operations |
| Drive or node failure and rebuild | Acute | Throughput and tail latency |
| Contention from another workload or background job | Acute or recurring | Latency for the affected workload |
| Thermal throttling, firmware faults, packet loss | Acute | Any |
How fill, load and cache growth slow a system
Flash slows as it fills. With less free space, garbage collection copies more still-valid data for every block it reclaims, so write latency rises and becomes less predictable. Hard drives lose sequential throughput as data lands on inner tracks, and fragmented free space breaks large files into scattered pieces.
Load growth is non-linear. Queueing delay scales roughly as 1 ÷ (1 − utilisation), so a move from 60% to 90% busy takes response time from 2.5 to 10 times service time, a fourfold rise from a 1.5-fold rise in load. A platform can feel comfortable for a year and then degrade within a quarter.
Cache effectiveness erodes as data sets grow. With 0.1 ms cache hits and 8 ms back-end reads, average read latency is about 0.26 ms at a 98% hit ratio and about 0.5 ms at 95%. A three-point drop in hit ratio nearly doubles read latency.
Failure, degraded mode and rebuild
When a drive or node fails, a protected system keeps serving data in degraded mode, reconstructing missing pieces from replicas or parity at extra cost per read. It then rebuilds the lost redundancy, and rebuild traffic competes with application traffic for the same drives, CPU and network.
Rebuild time scales with the amount of data to reconstruct and the rate at which it can be rewritten. A full 20 TB drive rebuilt at 100 MB/s takes 20 × 10¹² ÷ 10⁸ = 200,000 seconds, about 56 hours, and longer if the rebuild is throttled to protect foreground work. Traditional RAID funnels that work into one spare drive. Distributed erasure coding spreads reconstruction across many drives, which shortens the window of reduced protection that data durability depends on and spreads the performance cost thinly.
Node failures scale the same problem up. Losing a whole server puts every drive it held into reconstruction at once, so the rebuild load on the surviving nodes, and the slowdown tenants see, depends on how widely the protection scheme spreads data across the cluster.
What performance degradation means for storage operations
For a lean team running petabytes, the normal state of the platform includes some degradation most of the time. Capacity plans built on figures from an empty, healthy cluster overstate what the same platform delivers eighteen months later, 80% full, during a rebuild.
- Fill level is a performance variable. Running a large cluster close to full buys capacity efficiency with higher write latency and longer rebuilds.
- Rebuild behaviour decides how bad a bad day gets. Rebuilding only written data, spread over many drives, keeps the slowdown short and shallow; full-drive rebuilds to a single spare keep it long and concentrated.
- Background work shares resources with tenants. Scrubbing, replication to a second site, lifecycle transitions and mass deletes all draw on the same drives, and on a multi-tenant platform one tenant's cleanup is felt by everyone.
- Tail latency moves first. A p99 that doubles while the median holds steady points to queueing or background interference months before averages register it, and comparisons only hold when the workload is the same.
Rebuild behaviour in Scality RING
When a drive fails, Scality RING writes data across the remaining drives in the server and rebuilds only data that was written, within that server. A failed 20 TB drive holding 6 TB of data therefore needs 6 TB reconstructed, 30% of a full-drive rebuild, with the writes landing on several drives at once.
Because erasure coding schemes are defined per storage class, the balance between capacity overhead and rebuild work can differ between data sets held on the same RING.