Data placement is the set of rules that decides where each copy or fragment of an object physically lands: which drives, which servers, which racks and which sites. It is easy to treat as an implementation detail of the protection scheme. In practice it decides which failures a cluster survives, how evenly it performs, and how much work it takes to recover. Two clusters with identical hardware and the same protection scheme can behave very differently because their placement differs.
The arithmetic of how many fragments each fault domain can hold under a given erasure coding scheme is covered in the post on erasure coding layouts. This post looks at the placement decisions around that arithmetic: how fault domains are defined, how placement copes with an uneven physical estate, and what changes when racks, rooms and sites are added.
Fault domains are a model of the building
A fault domain is any group of components that can fail together. A server is one, because its drives share a motherboard and power supply. A rack is another if it shares a top-of-rack switch or a power distribution unit. A room with a single cooling plant is a third, and a site is the largest. Placement rules only protect against the failures they are told about, so the first step is describing the physical estate accurately.
This is where many designs drift from reality. Racks are declared as separate fault domains while sharing one power feed. Two "sites" turn out to be halls on the same campus fed by one utility connection. Servers are moved during maintenance without the placement map being updated. The protection scheme is then calculated against a hierarchy that does not match the building, and the first correlated failure exposes the gap.
How placement rules translate into survivability
The table below shows how the same cluster can be described at different levels of the hierarchy, and what each level protects against when placement respects it.
| Placement level respected | Survives | Does not survive | Typical requirement |
|---|---|---|---|
| Drive only | Individual drive failures | A server failure that takes several copies or fragments at once | Rarely acceptable beyond a single server |
| Server | Server or drive failures | A rack-level power or switch failure | Enough servers to hold each copy or fragment separately |
| Rack | Rack, server and drive failures | A room or site event | Enough racks, and racks that are independent in power and network |
| Site | Loss of an entire site | Simultaneous loss of more sites than the scheme allows | Two or more sites with enough capacity and bandwidth between them |
Higher levels are not free. Placing copies across racks and sites means repairs cross rack and site links, and site-level placement makes every write depend on the inter-site network, a constraint explored in the post on stretched clusters.
Uneven estates and placement balance
Placement rules assume the fault domains they spread data across are roughly comparable. Real estates are rarely tidy. One rack may hold newer, larger servers. One site may have twice the capacity of another. When placement insists on an equal share per fault domain, the smallest domain fills first and limits usable capacity for the whole cluster. When placement weights by capacity instead, the largest domain holds a disproportionate share of each object's protection, and losing it hurts more.
There is no single right answer, but there is a wrong one: not knowing which approach the cluster uses. The practical checks are to compare fill levels across fault domains regularly, and to recalculate what a loss of the largest domain would mean for data that has more than its fair share there. The same imbalance appears after hardware refreshes, a topic covered in the post on replacing aging nodes.
Placement and performance
Placement also shapes performance. Data spread evenly across many drives and servers gives every request a large pool of devices to read from and repair from. Data concentrated on a subset creates hot spots: some servers busy while others idle, and rebuilds funneled through the few devices that happen to hold the affected data.
Placement choices that improve resilience can cost performance. Site-level placement adds latency to writes, and in some designs to reads during a site outage. Rack-level placement increases east-west traffic across the data center network. These costs are acceptable when they are chosen deliberately and measured, and unwelcome when they appear as unexplained slowness after a topology change.
When placement has to change
Placement is not set once. Adding a rack, opening a second room or bringing a new site online changes the hierarchy, and existing data has to be moved to match. That movement is a background workload of the same kind as a rebalance, with the same effect on client traffic, and the same need to be throttled and monitored. The sequencing concerns described in the post on object storage expansion apply whenever placement policy changes, not only when capacity is added.
A useful discipline is to treat a placement change like a schema change: write down the old and new hierarchy, calculate what each failure scenario means before and after, estimate how much data must move, and only then start the migration.
Where Scality RING fits
In Scality RING, data placement across nodes and sites is configured by the operator rather than fixed by the product, and erasure coding and replication are selectable per policy. That makes the fault domain hierarchy an explicit design input: placement can be matched to the servers, racks and sites actually deployed and changed as the estate grows.
RING distributes data across the nodes in the system, so reads and reconstruction draw on many devices in parallel instead of concentrating on a few. It can span multiple sites, with data spread so that a site can be lost without data loss, and it emits operational metrics and logs the operator keeps locally, which is how fill levels and load across fault domains can be compared over time.
A placement review checklist
- Does the declared fault domain hierarchy match the building, including shared power, cooling and network?
- Which level of the hierarchy does each protection policy respect, and which failures does that cover?
- How do fill levels compare across racks and sites, and what would losing the fullest one mean?
- What extra latency and network traffic does the current placement add, and has it been measured?
- When the hierarchy last changed, was existing data moved to match, and is that confirmed?
Placement rarely gets attention until a failure tests it. Reviewing it while everything is healthy is how a cluster ends up surviving the failures it was designed for.