Every storage system tolerates a defined number of simultaneous failures. What matters operationally is how long it spends at reduced protection after one occurs, because that window is when a second failure becomes consequential. Rebuild time is therefore a risk measure rather than a performance detail, and it is determined by design choices that are made long before the first drive fails.
Drive capacities have grown much faster than drive throughput, which means the same rebuild takes longer on each hardware generation unless the architecture spreads the work differently. A system that reconstructs onto a single replacement device is limited by that device regardless of how large the cluster is.
When a device fails, the data it held has to be reconstructed. In a replicated system that means copying from surviving replicas. In an erasure coded system it means reading enough surviving fragments to recompute the missing ones, which involves more reading and some arithmetic but moves less data overall in most configurations.
Both approaches read from many places. The difference that matters most is where the reconstructed data is written. If it is written to one replacement device, that device's write speed is the ceiling, and a large modern drive can take many hours to fill. If it is distributed across the whole cluster, the ceiling is the aggregate write capacity of many devices and the rebuild completes far sooner.
That distinction, rather than the choice between erasure coding and replication itself, is usually the largest factor in rebuild duration.
| Factor | Effect on rebuild time | Decided when |
|---|---|---|
| Amount of data on the failed device | Directly proportional | Drive capacity selection |
| Number of devices participating | Fewer devices, longer rebuild | Data placement design |
| Where reconstructed data is written | Single target is the common ceiling | Architecture |
| Rebuild rate setting | Trades client performance against exposure | Configuration, adjustable live |
| Client load during the rebuild | Competing work extends it | When the failure happens |
| Object size distribution | Many small objects add per-object overhead | Workload |
| Network capacity between nodes | Caps how fast fragments can move | Fabric design |
| Cluster fill level | Fuller systems rebuild more slowly | Capacity planning |
The object size row is often overlooked. Reconstructing a petabyte held as a few million large objects is a different job from reconstructing a petabyte held as several billion small ones, because the second carries metadata work per object that the first does not.
Larger drives reduce cost per terabyte and increase the amount of data that has to be reconstructed when one fails. Where rebuild work is distributed across many devices this matters less, because the reconstruction of a large drive is shared. Where it is not, drive capacity translates almost directly into hours of exposure.
This is worth raising during procurement rather than afterwards. The question to ask is how long a rebuild takes for the largest drive in the configuration, at the fill level the system will normally run at, while serving a typical client load. A figure taken from an empty, idle system is not the number that will apply.
Node failures deserve the same question. Losing a whole node is a much larger reconstruction than losing one drive, and it is the more likely event in a power or hardware incident.
Most systems allow the reconstruction rate to be tuned. Faster rebuilds restore full protection sooner and take more resources from clients while running. Slower rebuilds preserve application performance and extend the exposure window.
The right setting depends on how much redundancy remains. A configuration that tolerates four failures and has lost one can afford a measured rebuild. The same configuration that has lost three should prioritize reconstruction over everything else. Some systems adjust this automatically based on remaining redundancy, and where they do not, it belongs in an operational procedure with a named decision maker.
The practical preparation is knowing how to change the setting, who may authorize it, and what the effect on client performance will be. Discovering all three during an incident is what leads to a rebuild being left at its default while a second failure approaches.
Scality RING distributes data across the nodes in the system so that reconstruction after a failure draws on many devices in parallel and writes to many devices in parallel, rather than funnelling recovery through a single replacement. That is the property that keeps rebuild time from scaling with drive capacity, and it means a larger deployment absorbs an individual failure with proportionally less impact.
The figures worth establishing for a specific deployment are the rebuild duration for a full node at the expected fill level, and the read throughput available to clients while that rebuild runs. Both belong in operational documentation, since the second is what a recovery performed during a rebuild will actually see.
Fail a drive deliberately in a test or pre-production system, under representative load, and record how long the rebuild takes and what happened to client performance. Repeat for a node if the configuration allows it. That produces two numbers that are otherwise guesses, and both feed directly into risk and recovery planning.
Repeat the measurement after a significant expansion or a hardware refresh. Rebuild behavior changes as the cluster grows and as drive capacities increase with each generation, so a figure from the original deployment may understate the current exposure considerably.