Storage service levels: Availability and durability

Storage service levels are usually written as two numbers with a lot of nines in them: one for availability and one for durability. They look similar and are often quoted side by side, but they describe different failures, are measured in different ways, and lead to very different conversations when something goes wrong. A platform can meet one comfortably while quietly missing the other.

For teams that run object storage as an internal platform, or that are evaluating a vendor's commitments, the useful work is less about the number of nines and more about what each number covers, how it is measured, and what happens during the periods when the system is technically up but not performing normally.

Availability and durability are different promises

Availability is the share of time, or the share of requests, for which the service responds successfully. When availability drops, clients see errors or timeouts, but the data still exists. When the service recovers, nothing has been lost.

Durability is the probability that stored data will still exist, uncorrupted, after a given period, usually a year. A durability failure means data is gone or damaged. It may not be noticed for months, and it cannot be fixed by waiting.

The two are related only loosely. A cluster with strong protection and slow failover can have excellent durability and mediocre availability. A system that serves requests fast from a single copy can show high availability right up to the moment a disk fails and data is lost.

What the nines mean in time

Availability targets are easier to reason about when converted to allowed downtime.

The jump from three to four nines is the one that changes operating practice. At 99.9 percent, a planned maintenance window can fit inside the budget. At 99.99 percent, maintenance has to happen without taking the service down at all, which means rolling upgrades, redundant endpoints and load balancing designed for endpoint failures.

How durability figures are produced

Durability is not measured by observation. Eleven nines of durability implies an expected loss of roughly one object in a hundred billion per year, and no single deployment holds enough data for long enough to confirm that empirically. Durability figures come from models that combine:

• the protection scheme, such as the replica count or the erasure coding layout;
• how fragments are placed across drives, servers, racks and sites;
• assumed annual failure rates for drives and servers;
• the time taken to detect and repair a failure, which sets how long data runs with reduced protection.

The last input is the one that varies most in practice. A model that assumes a six-hour repair is optimistic if a full-drive rebuild on large disks actually takes two days. Rebuild time is therefore a durability parameter as much as an operational one, and it is worth measuring rather than assuming, as discussed in the post on what affects recovery time. The choice of protection scheme feeds the same model, which is why the trade-off between erasure coding and replication is partly a durability decision.

Durability also depends on detecting silent corruption. A model that assumes every bad sector is found and repaired only holds if the system actually scrubs data and verifies checksums on a schedule.

SLA, SLO and SLI

Three related terms are often blurred:

• Service level indicator (SLI): what is measured, for example the percentage of S3 GET requests that return successfully within 200 milliseconds.
• Service level objective (SLO): the internal target for that indicator over a period, such as 99.95 percent over 30 days.
• Service level agreement (SLA): the external commitment, usually set looser than the SLO, with consequences attached if it is missed.

For an internal storage platform the SLO is the working tool. An SLA without a well-defined SLI behind it is a number nobody can verify.

Defining a useful storage SLO

A storage SLO is only as good as its definitions. Several details decide whether it reflects what applications experience:

• Measure at the client side of the endpoint. Server-side health checks can report green while a load balancer or network path is failing requests.
• Count requests, not minutes. A request-based SLI captures partial outages that a time-based one misses.
• Include latency. A request that succeeds after 30 seconds is a failure for most applications. Define success with a latency threshold at a stated percentile.
• Separate reads and writes. Many incidents affect one path and not the other, and applications depend on them differently.
• State the exclusions. Planned maintenance, client errors and requests above agreed rate limits should be listed explicitly rather than argued about afterwards.

The metrics that reveal trouble early are usually the same ones that make good SLIs.

Degraded is not the same as down

The period that service level definitions handle worst is the one where the system is up but working harder than usual: rebuilding after a drive failure, rebalancing after an expansion, or running with a site offline. Availability stays technically intact, latency rises, and protection is temporarily reduced.

A complete service level definition says what performance applications can expect in that state, how long it may last, and at what point it counts against the objective. Without that, the most common real-world degradation is invisible to the SLO.

Where Scality RING fits

Scality RING is designed so that nodes, drives and sites can fail or be added while the system keeps serving requests. Data protection is configured per policy, with erasure coding or replication, and placement across nodes and sites is set by the operator. Those choices are the inputs to the durability model and decide whether a server, rack or site loss affects availability at all.

RING distributes reconstruction work across many devices in parallel, which keeps the time spent in a reduced-protection state shorter as the deployment grows. It also emits operational metrics and logs that the operator keeps locally, so the indicators behind an internal SLO, including behavior during a rebuild, can be measured directly rather than inferred.

Questions to settle before committing to a service level

• Is the availability target measured per request or per minute, and at which point in the path?
• What latency threshold defines a successful request, for reads and for writes?
• Which durability model produced the durability figure, and what repair time does it assume?
• How long does a full-drive rebuild take at expected fill, and what client performance is available while it runs?
• What is excluded from the calculation, and is that list written down?

Answering these turns a pair of headline numbers into a commitment that applications can plan around and that operations teams can actually verify.