Adding capacity to a running object storage cluster is documented as a short procedure and experienced by operators as a multi-week condition. The nodes join, the system begins moving data to use them, and for a period the cluster is doing two jobs at once: serving clients and redistributing what it already holds. Which job gets priority, and what happens if a drive fails while both run, decides how the expansion feels.
The obvious approach is to add the nodes and let the system sort itself out. That works, in that the cluster ends up balanced, but it leaves the timing and intensity of the rebalance to a default chosen without knowledge of the workload. Expansions that go badly usually go badly for that reason.
The second complication is arithmetic. Capacity is the visible reason for an expansion, but protection schemes and failure domains are defined in node and rack counts, and those change as nodes join. A cluster that could survive the loss of a rack before an expansion may not survive it during one.
Why new nodes fill unevenly
New nodes arrive empty, and most placement algorithms weight by free space. The immediate effect is that a large share of new writes lands on the newest hardware, which is correct long-term behavior and an uncomfortable short-term one. A handful of empty nodes absorbing most of the write traffic becomes a hotspot, particularly if they share a top-of-rack switch with each other and not with the rest of the cluster.
The effect is sharpest when the expansion is small relative to the cluster. Adding four nodes to a hundred changes aggregate capacity slightly and the free-space ranking dramatically, so those four take a disproportionate share of writes until their utilization nears the average. Adding forty spreads the same effect across enough hardware to be unnoticeable.
Rebalancing corrects this from the other direction, moving existing objects onto the new nodes so that placement of future writes returns to even. Until it finishes, per-node utilization is the wrong measure of how the expansion is going and per-node write latency is the right one.
Rebalancing and the load it adds
A rebalance reads objects from their current locations and writes them to new ones, consuming the same disks, the same network and, in erasure-coded systems, the same reconstruction path client traffic uses. Most systems allow the rate to be throttled, and that throttle is the most consequential setting during an expansion. Its default is rarely correct for a cluster also serving production.
Throttling too aggressively carries its own risk, since it extends the period the cluster spends in a transitional state. A rebalance that runs for six weeks is six weeks in which other maintenance is harder to schedule and any failure lands on a system already doing background work. The useful framing is not fast against slow but how long the transition should last, with the throttle set to hit that duration.
The interaction deserving attention is a drive or node failure during a rebalance. Repair and redistribution compete for the same resources, and repair is the one with a durability deadline. Systems differ in whether they prioritize repair automatically, and the behavior is worth confirming rather than assuming, for the same reasons that make rebuild time worth measuring in advance.
| Expansion stage | What the cluster is doing | What to watch |
|---|---|---|
| Nodes racked and joined, no data placed | Membership and capacity records updated | Every node reported as a member before anything moves |
| New writes directed at empty nodes | Free-space weighting sends most writes to the newest hardware | Write latency on new nodes, uplink use for their rack |
| Background rebalance running | Existing objects read and rewritten to new locations | Client latency percentiles, queue depth, inter-rack network load |
| Rebalance overlapping a drive repair | Repair and redistribution share disks and network | Whether repair is starved, and its completion estimate |
| Rebalance complete | Placement even, failure domains recalculated | Utilization spread and distance from the fullness threshold |
Failure domain arithmetic when a rack or site grows
Protection schemes distribute fragments across failure domains, and the number of domains determines what the scheme survives. A cluster configured so that a full rack can be lost depends on there being enough racks to spread across. Adding nodes inside existing racks raises capacity without raising that count; adding a new rack changes it.
During the expansion the arithmetic is temporarily worse rather than better. New nodes hold little data, so the effective spread of any given object remains across the original domains until rebalancing moves fragments. The cluster reports more capacity and more nodes while its real tolerance for a rack failure is unchanged, a gap worth stating in the change record.
Site expansion has the same shape at a larger scale. Adding a third site changes which placement policies are possible, but only once enough data has moved to make that site a real participant. The choice between replication and erasure coding across sites determines how much data moves and how long it takes, the operational side of the trade-off between the two schemes.
Mixed hardware generations in one cluster
Expansions almost always introduce denser drives, more cores and faster networking than the existing nodes have, since that is what is available to buy. A cluster with mixed generations places data by capacity unless told otherwise, so the newest nodes hold the most data, serve the most requests, and form the largest single failure domain.
Denser nodes also take longer to rebuild. A node holding three times the data of its neighbors takes roughly three times as long to reconstruct, and the cluster is exposed for that window. Whether that is acceptable depends on the protection scheme, but the question belongs at purchase time rather than after racking, and it is the same calculation behind how a cluster is sized in the first place.
Weighting placement so that all nodes hold similar absolute capacity leaves the new hardware underused and is rarely worth it. Weighting by a fraction of capacity, so denser nodes hold more but not proportionally more, is the usual compromise.
Where Scality RING fits
RING is built so that nodes and servers can be added to a live system without taking the namespace offline, and data placement across nodes and sites is configured by the operator rather than fixed by the product. That configurability is what makes the failure domain arithmetic above something an operator controls, since placement across sites is set deliberately and both erasure coding and replication are selectable per policy.
Because RING spreads data across sites so that a whole site can be lost without data loss, a site-level expansion is a placement policy change as much as a hardware event, and the two need sequencing together. Self-healing runs as a background task, the same class of work as a rebalance, drawing on the same resources.
RING emits operational metrics and logs the operator keeps locally, which is how the utilization spread and repair queue behavior described here become observable during an expansion rather than after it.
How to sequence an expansion
Rack and join the nodes without changing the protection policy, and confirm the cluster reports them as members before any data moves. Then set the rebalance throttle from a target duration rather than a default. Then, if the expansion adds failure domains, change the placement policy and let that change drive redistribution instead of running two in sequence.
Suspend the expansion if a drive or node fails while it runs, and let the repair finish first. A rebalance can wait; a repair cannot. Confirming the system does this automatically, or scripting it, is worth doing before the first expansion rather than during one.
The record to keep is the node and rack counts before and after, the protection policy before and after, the throttle setting and the duration it was derived from, the per-node utilization spread at the end, and the date the rebalance completed. That last date marks when the stated failure tolerance became true again, and it anchors the next expansion or hardware refresh cycle.