A restore test usually covers one system. It confirms the backup is valid, the procedure works and the timing is acceptable, and it says almost nothing about what happens when 400 systems have to come back at once. At that scale the limits are different: aggregate read throughput from the repository, the capacity of the restore targets, the number of concurrent jobs the backup software will run, and the order in which things have to happen.
The gap between the two situations is large enough that recovery objectives derived from single system tests are frequently unachievable in a real event. Working out the real number in advance is arithmetic plus one measured figure, and it usually reframes the conversation about what the organization can promise.
A single restore runs with the whole repository, the whole network and the whole target platform available to it. It measures the best case for one workload and nothing about contention.
In a mass restore, every job competes for the same read bandwidth from the backup repository, the same network path and the same write capacity on the target storage. Adding parallel jobs increases aggregate throughput up to a point, after which it produces queuing and each individual restore slows down. The total time is governed by the aggregate rate and the total volume, not by the speed of any one job.
There is also a coordination cost that does not exist at small scale. Hundreds of restores need tracking, failures need retrying, and dependencies between systems need respecting, all while people are working long hours. Plans that assume smooth execution tend to overrun for reasons that have nothing to do with storage.
Three numbers give an honest estimate: the total volume to be restored, the aggregate restore throughput the environment can sustain, and the fixed overhead per workload. The first is available from backup reports. The second has to be measured. The third is often underestimated.
| Factor | How to establish it | Effect on the timeline |
|---|---|---|
| Total volume in scope | Sum of protected data for the systems in the recovery tier | Sets the floor with aggregate throughput |
| Aggregate restore throughput | Measure with many concurrent jobs, not one | The single most important number |
| Concurrency limit of the backup software | Configuration and licensing | Caps how much of the throughput is usable |
| Target storage write capacity | Write throughput of the platform being restored onto | Frequently the real bottleneck |
| Per workload fixed overhead | Time to provision, boot, verify and hand over | Dominates for large numbers of small systems |
| Dependency sequencing | The recovery order map | Reduces achievable parallelism |
| Staff availability | Shift planning across a multi-day event | Determines sustainable pace |
Per workload overhead deserves emphasis. If each restore needs 15 minutes of provisioning, verification and handover, 400 workloads carry 100 hours of overhead even if the data moved instantly. Automation and orchestration reduce this more effectively than any storage change.
Almost no organization needs everything back simultaneously, and the plan that tries to deliver that will deliver nothing quickly. Grouping systems into recovery waves, with the first wave small enough to complete in hours, produces a much better outcome than a flat queue.
The first wave should contain the foundation that everything else needs, which is the dependency order rather than a business priority list. The second contains the applications the business has agreed matter most. Later waves contain the remainder. Agreeing that grouping in advance, with the business rather than within IT, avoids the most costly delay in a real event, which is renegotiating priorities while the clock runs.
It is worth stating what each wave means in practice, including what a partial service looks like. A business that understands it will have its order system before its reporting system can plan around that. A business that expects everything at once cannot.
Attention naturally goes to the backup repository, but in a mass restore the write side is often the limit. If production storage was lost, the replacement has to be available, provisioned and capable of absorbing the restore rate. If the recovery runs onto a smaller standby environment, its write throughput and its capacity both cap the plan.
Two questions settle this. Does capacity exist to receive everything in scope, and at what rate can it be written? A plan that assumes hardware will be procured during the event should include the procurement lead time in the timeline, because that is frequently the largest single number in the whole exercise.
Virtualization platforms add a further consideration: management plane capacity and the rate at which machines can be registered and started, which has its own ceiling separate from data movement.
A backup repository built on Scality RING or ARTESCA reads at an aggregate rate that scales with the number of nodes, which is the property that matters when many restore streams run at once. The figure worth establishing during design is aggregate read throughput under high concurrency rather than the speed of a single stream, since that is the condition a mass restore creates.
Because the repository is a single namespace rather than a set of separate appliances, restore jobs are not confined to whichever device holds a particular copy, and capacity and throughput grow together as nodes are added. Sizing the repository for the read rate a full recovery would demand, rather than only for the write rate that backups need, is what keeps restore speed in line with the objective the business was given.
A full scale rehearsal is impractical for most organizations, but a partial one is not. Restore 20 or 30 systems concurrently into an isolated environment, measure the aggregate rate and the per workload overhead, and extrapolate. That produces an estimate based on measurement rather than on optimism, and it usually reveals a ceiling nobody knew about.
Record the result and repeat it annually. Protected volume grows steadily and restore capacity usually does not, so an estimate that was comfortable two years ago may now exceed the objective by a considerable margin. Finding that in a rehearsal is inexpensive. Finding it during a recovery is not.