Recovery plans assume the backup repository is healthy. That assumption holds most of the time and fails in exactly the circumstances that produce a recovery: a data center event that damages hardware, a power incident that takes several nodes down at once, or simply the ordinary coincidence of a drive failing during the week a restore is needed. In those cases the repository is serving restores while it is also rebuilding itself, and both operations compete for the same resources.
The question infrastructure teams should be able to answer is what the repository delivers in that condition rather than in a healthy one. A restore rate measured on a fully functional system is the best case, and recovery timelines built on it have no margin for the most likely complication.
What a rebuild actually costs
When a drive or node fails in an erasure coded or replicated system, the data it held is reconstructed from the remaining copies or fragments. That reconstruction reads from many devices and writes to the replacement, consuming throughput that would otherwise serve clients. It also consumes processor capacity for the reconstruction arithmetic and network capacity for moving fragments between nodes.
How much it costs depends on the protection scheme and on how aggressively the system is configured to repair. A fast rebuild restores full protection sooner and takes more from client performance while it runs. A slow rebuild preserves performance and leaves the system exposed for longer. Most systems expose this as a tunable, and the setting is usually left at its default because the tradeoff is only visible during an incident.
The scale of the effect also depends on the layout. Systems that spread reconstruction across many devices recover faster and with less impact on any single device than those that concentrate it, which is part of the reason the choice between erasure coding and replication is a recovery decision rather than only a capacity one.
The numbers worth knowing before an incident
Four figures turn this from a general concern into a planning input.
| Figure | How to obtain it | Why it matters |
|---|---|---|
| Rebuild duration for one drive | Vendor guidance, confirmed by observation | Length of the degraded window |
| Rebuild duration for a whole node | Same, at node scale | Much longer, and the more likely case in a site event |
| Read throughput while rebuilding | Measure during a deliberate failure test | The real restore rate in the worst case |
| Tolerated simultaneous failures | Protection scheme configuration | How much headroom exists before data is at risk |
The third row is the one that changes plans. If degraded read throughput is 60 percent of healthy throughput, every recovery timeline should be quoted with that possibility stated. If it is 25 percent, the recovery objective may not be achievable during a rebuild at all, and that is worth knowing before it is promised.
Prioritizing repair against recovery
During an incident, two things want the same capacity: restoring the business and restoring protection. There is no universally correct answer, and the balance depends on how exposed the system is.
If the repository is one failure away from data loss, completing the rebuild takes precedence, because losing the repository during a recovery ends the recovery. If it retains comfortable redundancy, throttling the rebuild to give restores more throughput is defensible for a period. What matters is that the decision is made deliberately, by someone who knows the current protection level, rather than left to a default while everyone assumes the other outcome.
Most systems allow the rebuild rate to be adjusted while running. Knowing how to do that, and who is authorized to, belongs in the runbook rather than in a support call placed at two in the morning.
Designing so the repository is not a single point
The deeper mitigation is not to depend on one repository. A second copy on separate infrastructure, ideally at another location, means a degraded primary is an inconvenience rather than a constraint on the whole recovery. This is the operational reason behind keeping multiple copies, distinct from the security reason of keeping one of them isolated.
Where a second copy exists, the plan should state which one restores are served from under which conditions, and both should have been tested as a restore source. A secondary copy that has never served a restore is an assumption, and the time to discover a difference in configuration between the two is not during an event.
Capacity headroom matters as well. A repository running close to full rebuilds more slowly and has less room to absorb reconstruction, so the capacity forecast and the recovery plan are connected. Planning for retention growth with enough margin keeps the system in the range where it recovers quickly.
Where Scality fits in degraded operation
Scality RING distributes data across nodes so that reconstruction after a failure draws on many devices in parallel rather than concentrating work on a small number, and the system continues serving reads throughout. Because capacity and throughput both scale with node count, a larger deployment absorbs a single failure with proportionally less impact on client performance.
The figure worth establishing during design or evaluation is read throughput with a node deliberately failed, since that is the condition a recovery may occur in. It is a reasonable test to ask for, and the answer belongs in the recovery plan alongside the healthy figure, so the timeline the business is given reflects the situation rather than the best case.
Test a restore with something broken
The exercise is straightforward and rarely run: fail a drive or a node deliberately, then perform a restore of realistic size while the rebuild is in progress, and record the rate. That single measurement converts a category of uncertainty into a number that can go into a plan.
Run it when the system is new, and again after any significant expansion, since the impact of one failure changes as the deployment grows. Teams that have this figure can state their recovery timeline with a qualifier that holds up. Teams that do not are quoting a best case and describing it as a commitment.