A stretched cluster is a single storage system whose nodes are spread across two or more sites close enough to behave, for most purposes, like one data center. Writes are acknowledged only once they are protected across sites, so the loss of a whole site does not lose acknowledged data. The design is attractive because it removes the gap between primary and replica. It is also demanding, because every write now depends on the network between the sites.
This post looks at how stretched clusters replicate data across sites, what distance and latency do to them, and how they handle the failure everyone designs for and the one fewer people plan for: the sites losing sight of each other while both are still running. What happens to writes during a planned or unplanned move between independent sites is a related but separate question, covered in the post on object storage site failover.
What makes a cluster stretched
In a conventional multi-site design, each site runs its own storage system and data is copied between them asynchronously. The second site is always slightly behind. In a stretched design there is one system, one namespace and one set of metadata, with placement rules that put copies or fragments of every object in more than one site. A write is not acknowledged to the client until enough of those copies or fragments exist to survive a site loss.
The result is a recovery point objective of zero for site failure: anything the application was told is stored remains stored if a site disappears. The cost is that each write waits for the slowest site needed to protect it.
Distance, latency and write performance
Light in fiber travels at roughly 200 kilometers per millisecond, so each 100 kilometers of fiber path adds about one millisecond of round-trip time before any switching or processing. Real paths are longer than the straight-line distance, and real round-trip times include equipment delay. A stretched cluster's write latency therefore rises with distance in a way no tuning removes.
| Inter-site round-trip time | Typical situation | Effect on stretched writes |
|---|---|---|
| Under 1 ms | Buildings on one campus or metro sites very close together | Small addition to write latency, rarely noticed by applications |
| 1 to 5 ms | Metro-area sites tens of kilometers apart | Noticeable on small writes and metadata operations, usually acceptable for large objects |
| 5 to 10 ms | Regional distance | Small-object and metadata-heavy workloads slow down markedly |
| Over 10 ms | Long distance | Generally better served by asynchronous replication between separate systems |
Bandwidth matters as much as latency. Every write crosses the inter-site link, and so does every repair after a failure. A link sized for normal ingest can saturate during a rebuild that has to recreate a whole site's worth of copies, which is why inter-site capacity should be sized for recovery, not only for steady state. The broader treatment of what affects recovery time applies with the extra constraint of a wide-area link.
Two sites or three
A stretched cluster has to decide, at every moment, which part of it is authoritative. With two sites that decision is hard. If the link between them fails, each site sees the other as gone, and neither can tell whether the other has crashed or is still serving clients.
The usual answer is a tiebreaker, often called a witness or quorum site: a third location, which may hold no data, whose vote decides which side continues. The side that can still reach the witness keeps serving writes. The other side stops accepting them. Three full data sites achieve the same with a majority rule, and add the option of surviving a site loss while still holding protected copies in two places.
Without a tiebreaker, the safe behavior after a link failure is to stop writes on at least one side, and many two-site designs fall back to an operator decision. That is acceptable if the procedure is documented and rehearsed, and a serious risk if it is not.
Split brain and why it matters
Split brain is the condition in which both halves of a stretched cluster continue accepting writes independently. Each half's data then diverges, and when the link returns there is no automatic way to decide which version of an object is correct. For object storage the damage can be subtle: two different objects under the same key, a delete applied on one side and an overwrite on the other, or versions numbered differently at each site.
Preventing split brain is the main reason for quorum rules. The test for any stretched design is simple to describe: cut the inter-site link while both sites are healthy, and confirm that only one side continues to acknowledge writes.
Data protection across sites
A stretched cluster can protect data with replication, with erasure coding, or with a mix. Replication places full copies at each site and makes reads local, at the cost of storing the data two or three times. Erasure coding spreads fragments across sites with lower overhead, but a site loss forces reads to reconstruct from fragments at the surviving sites, and repairs move more data across the link. The arithmetic of spreading fragments across sites is set out in the post on erasure coding layouts, and the general trade-off in erasure coding versus replication.
Where Scality RING fits
Scality RING can span multiple sites as a single system, and data placement across nodes and sites is configured by the operator rather than fixed by the product. Erasure coding and replication are selectable per policy, so the protection scheme for site-level resilience can be matched to the latency, bandwidth and capacity actually available between locations.
RING is designed to spread data across sites so that a whole site can be lost without data loss, with self-healing running as a background task to restore protection afterwards. It emits operational metrics and logs that the operator keeps locally, which allows inter-site latency, replication load and repair traffic to be watched during normal operation and during a site event.
Questions to answer before stretching a cluster
- What is the measured round-trip time between sites, under load, and how does it affect the workload's write latency?
- Is the inter-site link sized for a full site rebuild, not only for normal ingest?
- Where is the tiebreaker, and is it independent of both data sites' power and network?
- What happens to writes on each side when the link fails, and has that been tested?
- Which protection scheme is used across sites, and what does a site loss do to read performance?
Answered with tests rather than assumptions, these questions decide whether a stretched cluster delivers what it promises: no loss of acknowledged data when a site goes away, and no divergence when the sites cannot see each other.