When a site running object storage fails, the first question is whether applications can keep reading and writing. The second, which matters more and gets asked later, is what happened to the writes that were in flight or recently acknowledged when the site went down. The answer depends almost entirely on how data was moving between sites before the failure, and it is far better decided in advance than discovered during an incident.
This post covers failover between sites that hold separate copies of data: what happens to recent writes under each replication model, how clients are redirected, and what failback involves. The design of a single cluster stretched across sites, where every write is protected at more than one site before it is acknowledged, is covered in the post on stretched clusters.
The replication model decides the outcome
Data reaches a second site in one of two ways. With synchronous protection, a write is acknowledged only after it is safe at both sites. With asynchronous replication, the write is acknowledged once it is stored locally, and copied to the other site afterwards. Asynchronous replication is common because it keeps write latency independent of distance, but it means the second site is always some way behind.
| Replication model | Writes acknowledged when | Data at risk if the primary site is lost | Effect on write latency |
|---|---|---|---|
| Synchronous across sites | Protected at both sites | None of the acknowledged writes | Rises with inter-site distance |
| Asynchronous, low lag | Stored at the primary | Writes not yet replicated, typically seconds to minutes | Unaffected by distance |
| Asynchronous, backlogged | Stored at the primary | Everything in the replication queue, which can be hours | Unaffected by distance |
| Scheduled copy or backup | Stored at the primary | Everything since the last completed copy | Unaffected |
The third row is the one that surprises teams. Replication lag is usually small, but it grows during ingest spikes, link degradation or maintenance. If the primary site fails while a backlog is building, the recovery point is the age of the oldest unreplicated write, not the typical lag.
Measuring replication lag honestly
Replication lag can be reported in several ways that look similar and mean different things: the number of objects queued, the bytes queued, or the age of the oldest unreplicated object. Only the last one shows how much data a failover would lose. A queue of ten objects is harmless if they arrived a second ago, and serious if one of them has been waiting since yesterday because of a permissions error.
Lag should be monitored as an age, alerted on against the recovery point objective, and broken down by bucket, because a single stuck bucket can hide behind a healthy average. The metrics that reveal trouble early should include it.
What happens to writes during failover
When the primary site fails under asynchronous replication, three groups of writes need attention:
- Replicated writes are present at the secondary and need nothing further.
- Acknowledged but unreplicated writes exist only at the failed site. If that site's data survives, they can be recovered later. If it does not, they are lost, and the applications that wrote them believe they succeeded.
- In-flight writes that had not been acknowledged fail at the client, and well-behaved applications retry them against the new endpoint.
The second group is where application design matters. Systems that can replay recent writes from their own logs or queues recover cleanly. Systems that treat an acknowledged write as permanent and discard their source copy do not. A recovery runbook should state, per application, how the second group is handled, which ties into the broader question of what must come back first.
Redirecting clients
Applications find object storage through endpoint names. Failover means pointing those names at the surviving site, through DNS changes, a global load balancer or application configuration. Each has a different delay: DNS changes wait for cached records to expire, load balancers react to health checks, and configuration changes depend on deployment processes. The design considerations in the post on S3 load balancing for endpoint failures apply at site level, with the added question of who decides that a site has failed and how long that decision takes.
Automatic failover is fast but risks acting on a network partition rather than a real outage. Manual failover is slower but deliberate. Many organizations automate redirection for reads and require a decision for writes, so that two sites never accept writes for the same data at once.
Failback is its own project
Once the failed site returns, it holds data that is now out of date and, possibly, writes that never reached the secondary. Failback means reconciling the two: replicating changes made at the secondary during the outage back to the original site, deciding what to do with any orphaned writes, and only then moving clients back.
The reconciliation volume can be large if the outage was long, and the inter-site link has to carry it alongside normal traffic. Failback should be planned and rehearsed with the same care as failover, and it is often safer to run on the secondary for longer than to rush the return. Keeping replicas inside the same jurisdiction and operational boundary also matters during this period, as discussed in the post on sovereign disaster recovery.
Where Scality RING fits
Scality RING can span multiple sites as a single system, with data placement across nodes and sites configured by the operator, and it is designed so that a whole site can be lost without data loss when data is spread across sites in that way. Where separate sites are preferred, the operator decides how data is protected and placed at each, so the choice between site-level protection within one system and replication between systems can follow the latency and recovery point the workload needs.
RING exposes an S3 interface, so applications use standard endpoints that can be redirected during a site event, and it emits operational metrics and logs the operator keeps locally, which is where replication and repair activity can be watched before, during and after a failover.
Questions to settle before the next site outage
- Which buckets are protected synchronously across sites, and which replicate asynchronously?
- What is the age of the oldest unreplicated write right now, and what alert fires when it grows?
- How do clients reach the surviving site, and how long does redirection take?
- Who decides that a site has failed, and is write failover automatic or manual?
- How does each critical application handle acknowledged writes that may have been lost?
- What is the failback procedure, and when was it last rehearsed?
A site failure is a test of decisions made months earlier. Making those decisions explicitly, and measuring the numbers they depend on, is what lets a failover lose nothing that the business was not already prepared to lose.