A dashboard showing green capacity bars, an average latency figure and an uptime percentage can sit alongside an application team reporting intermittent failures, and both can be accurate. The metrics storage dashboards display by default describe aggregate state, while the failures users experience concentrate in a small share of requests, one failure domain or one tenant. Aggregates are the wrong shape for finding those.
The usual answer is to add panels, which increases the volume of information without improving its predictive value. A dashboard with forty charts and no stated threshold is a reference tool, not a monitoring system. The question worth asking of any metric is what decision it changes, and how much warning it gives first.
A few signals do predict trouble on object storage, and they share a property. Each measures a distribution or a margin rather than a total.
Error rate split by response class
A single error rate is close to useless, because the S3 error classes mean unrelated things. Client errors in the 400 range are mostly application behavior, and a rise in 403 responses follows a credential or policy change rather than a storage fault. Server errors in the 500 range are the system's own failures, and 503 indicates the endpoint is shedding load.
Splitting the rate by class turns one ambiguous number into three useful ones. A rising 500 rate against a flat request rate is a system problem. A rising 503 rate against a rising request rate is a capacity or balancing problem. A rising 404 rate on a workload that previously found what it asked for usually means a lifecycle rule or cleanup job removed more than intended.
The rate should be a proportion of requests rather than a count, and tracked per endpoint and per tenant rather than cluster-wide. A rate that looks negligible across the cluster can be total failure for one client whose requests land on one node. Reading it correctly depends on knowing what each S3 error code means in the implementation at hand.
Percentiles rather than averages
Average latency is dominated by the common case and hides the tail, and the tail is what applications experience as failure. A workload issuing thousands of requests per second meets the 99th percentile constantly, so a low mean alongside a 99th percentile measured in seconds describes a system failing for a meaningful share of traffic while the average says otherwise.
Track the median, the 95th and the 99th percentile separately, and per operation type. GET, PUT, DELETE and LIST have different cost profiles, and LIST scales with the number of keys under a prefix rather than object size, so listing latency that grows steadily reports on key distribution rather than load. The gap between throughput and per-request behavior is the subject of why throughput is not enough.
Watch the gap between the median and the 99th percentile as a signal in its own right. A widening gap while the median stays flat means variance is increasing, which points at one slow node, one degraded disk or one saturated path rather than general overload.
Repair activity and headroom per failure domain
Self-healing systems repair continuously, so repair work is normal and the depth of the queue is the metric. A queue that drains between events has capacity to spare. A queue that never reaches zero means repair is at its limit, and the next failure extends the window during which data is held with fewer copies or fragments than the policy intends.
The figure to record is not queue depth alone but the time to drain after a known event, such as a disk replacement, and how that time trends. It is the honest estimate of exposure, and it grows as the cluster fills, since rebuild work scales with the data on the failed device. What drives it is covered in what affects recovery time.
Capacity needs the same treatment. Cluster-wide percentage used is the least useful capacity metric available, since placement is constrained by failure domain. If one rack, site or node group is near full while the cluster average is comfortable, writes that must place a fragment there degrade or fail while the dashboard shows headroom. Track free space per failure domain, and the skew between the fullest and emptiest domain as its own series.
| Signal | What it usually means | What to check |
|---|---|---|
| 99th percentile latency rising while the median is flat | Variance from one component, not general load | Per-node latency distribution, disk error counters, network path |
| 503 responses rising with request rate | The endpoint is shedding load it cannot place or serve | Connection distribution across endpoints, and per-tenant request share |
| Repair queue never returning to zero | Background repair is running at its sustained limit | Time to drain after the last known failure, and how it has trended |
| Capacity skew between failure domains widening | Placement is constrained before the cluster is full | Free space in the fullest domain, and whether expansion rebalanced |
Per-tenant skew and the dashboards that mislead
Clusters are shared, and the aggregate request rate hides how unevenly that sharing goes. One tenant issuing many small requests can consume a disproportionate share of request handling while adding almost nothing to stored capacity. Nothing on a capacity-oriented dashboard shows it, and the symptom other tenants report is latency.
Track requests per second and error rate per tenant, and look at the ratio of requests to bytes transferred. A tenant whose ratio sits far above the others is running a small-object or metadata-heavy workload, a different profile from the one the cluster was sized for. Whether it was anticipated is a sizing question, better raised before saturation.
Three dashboard conventions mislead reliably. Uptime as a percentage counts an endpoint as available while it returns errors. Capacity as one cluster-wide bar hides the domain skew above. And a chart whose default window is a day makes slow trends invisible, which matters because repair drain time, capacity skew and tail latency all move over weeks.
Where Scality RING fits
RING emits operational metrics and logs the operator can keep local, so the per-endpoint, per-tenant and per-domain breakdowns described above can be collected into whatever monitoring stack already exists, without telemetry leaving the environment. For deployments with residency constraints, that settles a question about where operational data goes.
RING's self-healing runs as a background task that detects and repairs lost or corrupted data, so repair queue depth and drain time are ordinary operational signals rather than exception conditions. Because durability is configured per policy using erasure coding or replication, the exposure a given backlog represents depends on the policy in force for the affected data, and the metric should be read against it.
RING's multi-tenant model, with accounts, buckets and quotas, gives the natural boundary for per-tenant request accounting. Since RING spans nodes and sites in one namespace, capacity headroom is best read per failure domain rather than for the namespace as a whole.
What to alert on and what to review
Alert on a small set of conditions with stated thresholds: server error rate above a defined proportion of requests over a defined window, 99th percentile latency above what the busiest application tolerates, free space in the fullest failure domain below the amount needed to absorb the loss of one domain, and a repair queue that has not drained within the expected window. Four alerts that fire rarely beat forty that fire weekly.
Review the slower series monthly, since they are not alert material. Capacity skew between domains, repair drain time after each event, per-tenant request share and the median-to-tail gap all move slowly enough that a trend is visible long before a threshold is crossed. Record the values rather than only looking at charts, since a number written down last quarter is what makes this quarter's reading interpretable.
Write down the expected value for each series and the reason behind it, then revisit after every expansion, hardware refresh or new workload. Each changes the baseline, and a threshold inherited from a smaller cluster is the most common reason an alert either never fires or never stops.