VM backup at scale behaves differently from the same process at a few hundred machines. The method does not change: a snapshot is taken, changed blocks are read, data is reduced and sent to a repository. What changes is where the limit sits. At small scale the bottleneck is usually obvious and singular. At several thousand virtual machines, a backup system has half a dozen stages that can each become the constraint, and relieving one moves the queue to the next. Teams that scale by adding capacity to the stage they suspect often see no improvement at all.
This post walks through the stages of a large virtual machine backup, where each one saturates and how to tell which is limiting at any given time.
A virtual machine backup passes through a fixed sequence, and throughput is set by the slowest stage. Backup software typically reports which stage each job spent its time waiting on, and those statistics are the most useful diagnostic available.
| Stage | What happens | How it saturates at scale |
|---|---|---|
| Scheduling and catalog | Jobs are queued, sessions tracked in a database | Database contention, slow job start, long retention processing |
| Snapshot | Hypervisor creates and later removes a snapshot | Limits on concurrent snapshots per host and datastore, slow consolidation |
| Source read | Changed blocks read from production storage | Production storage latency rises, impact on running workloads |
| Proxy processing | Compression, deduplication, encryption | Processor and memory exhausted, tasks queue for a free slot |
| Network | Data moves from proxy to repository | Shared links saturate, especially between sites |
| Repository write | Data and metadata written to backup storage | Ingest throughput, request rate and metadata operations |
At scale, the backup server and its database become a bottleneck before any data moves. Thousands of jobs, each with session records, restore point metadata and retention processing, generate a steady load on the configuration database. Symptoms include jobs that sit in a pending state, a console that is slow to respond and retention tasks that run into the next night's window. The remedies are architectural: a properly sized, separate database server, fewer and larger jobs organized by policy instead of one job per machine, and where the product supports it, several backup servers under a central manager.
Each hypervisor host and datastore tolerates only a limited number of snapshots being created and removed at once. Removing a snapshot means merging the changes made during the backup into the base disk, and on a busy machine with a long-running backup that merge is heavy. Too many in parallel and production latency suffers. Backup software enforces limits per host and per datastore for that reason, and at scale those limits, more than anything in the backup infrastructure, can determine how many machines are processed per hour. Shorter backups help most, because a snapshot that exists for five minutes is cheap to remove.
Proxies do the processor-intensive work, and each handles a fixed number of concurrent disks. Too few, and jobs wait for a slot. The intuitive response is to add proxies and raise concurrency, which helps until the source storage or the repository becomes the limit. Past that point, more concurrency makes each stream slower and lengthens the time snapshots stay open. The right concurrency is the highest value at which per-stream throughput has not yet fallen, and it is found by measurement.
A repository that performs well with twenty streams may behave very differently with four hundred. Two properties matter. Aggregate ingest throughput has to scale with the number of streams, which favors storage that spreads writes across many nodes over a single controller. And on object storage, backup software writes data as large numbers of objects, so the request rate and metadata operations grow with stream count as well as with volume. Retention processing adds bursts of deletes and listings on top.
When the repository is the limit, job statistics show the target as the bottleneck and proxies sit partly idle. The posts on S3 request performance and small object storage explain what drives request-side limits, and S3 load balancing covers spreading client connections across the storage nodes so that no single endpoint becomes the choke point.
With thousands of machines, the backup window is governed by queueing. Jobs that all start at the same hour produce a peak that the infrastructure cannot absorb and then a long idle tail. Spreading start times, grouping machines so that large and small ones are mixed and letting the scheduler run continuously through the window flatten the peak. Incremental-forever schemes with synthetic operations performed on the repository remove the weekly spike of full backups, though they shift work onto the repository, which then has to be sized for it.
The useful metric is machines protected per hour, tracked nightly. A falling trend is an early sign that some stage has reached its limit, well before jobs begin to miss the window.
Everything above concerns backup. A large restore runs the pipeline in reverse and stresses different points. Reads from the repository are more random than writes were. Production storage has to absorb full machine images, not increments. And the hypervisor has to register and start hundreds of machines. An estate tuned to finish its backups by morning may still take a week to restore, because nobody measured the reverse path. The posts on mass restore planning and restore speed as the real recovery time address that side.
Scality RING exposes an S3 interface used by backup applications as a repository. It distributes data across the nodes in the system, spreading ingest and read load across many devices, which is the property that matters when hundreds of streams write or restore at once. Nodes and servers can be added to a live system without taking the namespace offline, so repository throughput and capacity can grow as the protected estate does.
RING emits operational metrics and logs that the operator keeps locally, so request rates, throughput and background activity can be compared with the backup software's own bottleneck statistics. It supports versioning and S3 Object Lock for immutable restore points.
Scaling virtual machine backup is a matter of finding the current constraint, relieving it and measuring again. The stages are known in advance. Which one is limiting on a given night is something only the statistics can show.