Storage software needs updating like any other software: security fixes, bug fixes, new features and support for new hardware. For a system that applications expect to be available around the clock, the question is not whether to update but how to do it without a maintenance window. In distributed object storage the standard answer is a rolling upgrade: update one node or a small group at a time, confirm it is healthy, and move on, while the rest of the cluster keeps serving requests.
The approach is well established, but its success depends on details that are easy to underestimate: how much redundancy is given up while a node is down, how the versions coexist during the upgrade, what happens if a hardware failure arrives halfway through, and how to stop and reverse if something goes wrong.
How a rolling upgrade works
A rolling upgrade proceeds through the cluster in waves. For each wave, the operator or the upgrade tooling typically:
- checks that the cluster is healthy, with no ongoing rebuilds and full data protection;
- drains client traffic from the nodes in the wave, usually by removing them from load balancer pools;
- stops services on those nodes, updates software, and restarts them;
- waits for the nodes to rejoin, catch up on any writes they missed, and pass health checks;
- returns them to the load balancer and moves to the next wave.
From the outside, the cluster stays available throughout. Inside, it is running with reduced capacity and redundancy for the duration of each wave.
What redundancy is spent during an upgrade
A node taken down for an upgrade is, from the cluster's point of view, temporarily failed. Its data is unavailable until it returns. Protection schemes handle that, but it uses some of the failure tolerance the scheme provides.
| Situation | Effect on protection | Risk if another failure occurs |
|---|---|---|
| One node down for upgrade, scheme tolerates two failures | One failure's worth of tolerance remains | A second failure is still survivable |
| One node down, scheme tolerates one failure | No tolerance remains for affected data | A second failure can make data unavailable or lost |
| Several nodes down in one wave, within one fault domain | Depends on placement: safe if the scheme tolerates that domain's loss | Low if placement respects the domain, high if it does not |
| Upgrade started during a rebuild | Tolerance already reduced by the rebuild | Highest, and generally avoided |
The practical rule is to size each wave so that the cluster could still survive an unplanned failure during it. Upgrading by fault domain, one rack at a time when placement respects racks, is often both faster and safer than upgrading individual nodes scattered across the cluster, a point that depends on the placement design discussed in the post on data placement.
Mixed versions during the upgrade
For the length of a rolling upgrade, some nodes run the old version and some the new. They must be able to talk to each other, agree on data formats and handle requests consistently. Well-designed storage software supports this by making new features inactive until every node is upgraded, and by keeping on-disk and network formats compatible across adjacent versions.
Two consequences follow. Skipping many versions in one step is riskier than upgrading through supported paths. And new features should be enabled only once the whole cluster is on the new version, as a separate, deliberate step that can be planned and tested on its own.
Client behavior during an upgrade
Even with careful draining, some requests will be in flight when a node goes down. Clients see this as connection resets or retryable errors. Applications built on S3 SDKs generally retry transparently, but custom clients, long-running multipart uploads and applications with aggressive timeouts may not. The design of load balancing for endpoint failures decides how gracefully traffic moves away from nodes being upgraded, and the S3 error codes applications see during the process are worth checking in advance.
Knowing when to stop
A rolling upgrade should have a defined stop condition before it starts. Typical triggers include a node that fails to rejoin within a set time, a rise in client error rates or latency beyond an agreed threshold, an unrelated hardware failure, or a rebuild starting anywhere in the cluster. Pausing at a wave boundary is safe. Pushing through is how a routine upgrade becomes an incident.
Rollback deserves the same planning. Whether the previous version can be reinstated on upgraded nodes, and up to which point in the upgrade, should be known from the release notes and from testing, not discovered under pressure.
Where Scality RING fits
Scality RING is designed so that nodes can be taken out of service and returned while the system keeps serving requests, and nodes and servers can be added to a live system without taking the namespace offline. Data protection is configured per policy with erasure coding or replication, and placement across nodes and sites is set by the operator, which is what determines how many nodes or which fault domain can be upgraded at once without losing tolerance for a further failure.
Self-healing runs as a background task, and RING emits operational metrics and logs the operator keeps locally, so cluster health, client error rates and any catch-up work after each wave can be checked before the next one starts.
A rolling upgrade checklist
- Is the cluster fully healthy, with no rebuild or rebalance running?
- Is the upgrade path from the current version supported in one step?
- How large is each wave, and could the cluster survive an unplanned failure during it?
- Are clients drained from each node before it stops, and do applications retry correctly?
- What conditions pause the upgrade, and who decides to resume?
- When will new features be enabled, and how will that step be tested?
Done this way, software updates become routine background work rather than scheduled downtime, which is exactly what a storage platform with high availability targets needs them to be.