Table of Contents
Glossary
Storage Management
Storage management is the ongoing work of running a storage system: provisioning capacity, configuring how data is protected, watching for the signals that predict trouble, moving data as it ages, and controlling who can reach it.
It is not one product. It is a set of jobs that every storage platform handles somehow, and the differences between platforms are mostly differences in how much of this work is automated.
The jobs it actually covers
Storage management is a broad term with no single standards-body definition, which is why vendor pages tend to fill it with whatever the vendor sells. In practice it breaks into six jobs:
- Provisioning — deciding how much capacity a workload gets, and how that capacity is presented to it.
- Data protection — configuring replication or erasure coding, deciding where copies land, and setting retention and immutability.
- Monitoring — collecting the signals that tell you the system is healthy, and alerting on the ones that predict it will not be.
- Lifecycle — moving data to cheaper storage as it ages, and deleting it when policy says it can go.
- Access control — who can read and write what, and how tenants are separated from one another.
- Capacity planning — knowing how much room is left and how long it will last at current growth.
Each of these is straightforward on one array. What changes at scale is that they stop being tasks someone performs and have to become policies the system enforces.
The three capacity numbers, and why they never match
Almost every capacity argument comes from people quoting different numbers at each other without saying which one they mean.
| Number | What it means | Where you see it |
|---|---|---|
| Raw | The sum of every drive's stated capacity | The purchase order |
| Usable | Raw minus protection overhead and system reserve | The design document |
| Effective | Usable multiplied by any compression or deduplication ratio actually achieved | Vendor marketing |
A cluster with 1.5 PB raw, running erasure coding at 33% overhead, delivers roughly 1.13 PB usable. Reporting tools using binary units will show that as about 1.02 PB. If a vendor quotes an effective figure assuming 2:1 data reduction, the same hardware gets described as 2.26 PB. All four numbers are correct and they describe one system.
Effective capacity is the one to treat carefully, because reduction ratios depend entirely on the data. Virtual machine images and text compress well. Video, images and anything already encrypted or compressed do not. A ratio measured on someone else's workload predicts nothing about yours.
Two practical rules: agree which number a figure refers to before anyone builds a budget on it, and plan to run a cluster below its ceiling. Systems that need to redistribute data on failure need free space to redistribute it into, and one that is nearly full loses the room to repair itself.
Protection is a configuration, not a checkbox
"The data is protected" describes almost nothing. The decisions that matter are specific:
What scheme, and at what overhead. Three replicas costs 200% overhead and survives two simultaneous losses. Erasure coding at 12 data plus 4 parity costs 33% and survives four. Replication is faster for small objects and simpler to reason about; erasure coding is dramatically cheaper at scale. Systems that let you set this per class of data rather than once for the whole cluster let archives run at low overhead while latency-sensitive data keeps replication.
Where the pieces land. A protection scheme that survives four drive failures survives nothing if all the fragments sit in one rack and the rack loses power. Placement across failure domains — drive, server, rack, site — is a separate decision from the scheme itself, and it is the one more often left at its default.
Whether the copies can be changed. Replication is faithful. It copies ransomware encryption and accidental deletions to the second site exactly as reliably as it copies good data. Object Lock, in governance or compliance mode, is what makes a copy genuinely unchangeable for a retention period. Protection against hardware failure and protection against bad writes are different problems, and only the second one needs immutability.
What is actually worth watching
Most storage dashboards show capacity and throughput because those are easy to draw. Neither predicts an outage. The signals that do:
| Signal | What it tells you |
|---|---|
| Capacity trend, not capacity | How many weeks of headroom remain at the current growth rate. A cluster at 70% growing 2% a week is a bigger problem than one at 85% that is flat. |
| Latency percentiles, not averages | The average hides everything. p99 and p99.9 are where users notice, and where a failing drive shows up before it fails. |
| Rebuild state and duration | How much data is currently running below its designed redundancy, and for how long. This is the window in which a second failure causes real loss. |
| Error rates and drive health | Reallocated sectors and rising read errors precede failures. Drives usually warn before they die. |
| Incomplete multipart uploads | Failed large uploads leave fragments that consume capacity and appear in no object listing. On a busy S3 endpoint this quietly accumulates for years. |
The practical test of a monitoring setup is not how much it displays. It is whether it would have told you about the last incident before a user did.
Lifecycle: moving data down as it ages
Most stored data is read intensively for days, occasionally for months, and then almost never — but it still has to be kept. Lifecycle management is the mechanism for matching what data costs to how much it is actually used.
In S3-compatible systems this is expressed as rules attached to a bucket, optionally narrowed by prefix or object tag. The standard actions are:
- Transition current object versions to a different storage class after a set number of days.
- Transition noncurrent versions — older versions, which are usually the larger cost in a versioned bucket.
- Expire current versions after a set period.
- Expire noncurrent versions, which is how you stop version history growing without limit.
- Remove expired delete markers, the tombstones left behind when every version of an object is gone.
- Abort incomplete multipart uploads after a set number of days.
That last rule is the one most often missing and the one that most often explains unaccounted capacity. Multipart uploads that never complete leave their parts behind indefinitely, and those parts are invisible to normal object listings while still occupying space.
Two warnings. Lifecycle rules and retention locks interact: an expiry rule cannot delete an object that Object Lock is holding, and a rule written on the assumption that it can will fail quietly. And archive tiers charge for retrieval as well as storage — data moved down is cheaper to keep and more expensive to read back, which is the right trade only if the access pattern really has gone cold.
Access control and tenancy
In object storage, access control is normally expressed as identity policies attached to users or roles, plus policies attached to buckets. The two combine, and the combination is where mistakes hide — a bucket policy that grants broad read access will do so regardless of how carefully the identity policies were written.
Where several teams or customers share one system, the questions are: can one tenant see another's data, can one tenant's traffic degrade another's performance, and can one tenant consume capacity the others were relying on. Quotas answer the third. The first two are architectural and worth confirming rather than assuming.
The operationally useful habit is auditing what access actually exists rather than what was intended — permissions granted for a migration three years ago rarely get revoked, and no one notices until they are used.
Why management changes shape at scale
On a handful of arrays, managing storage means logging into each one and doing things. That approach has a hard ceiling, and it is reached well before the hardware is exhausted.
Past a few petabytes, three shifts happen. Failures stop being events and become background noise, so the system has to repair itself and report afterwards rather than wait for someone to respond. Configuration stops being per-device and becomes policy — you declare the intent once and the system applies it to everything that matches, including hardware bought two years later. And observation moves from dashboards someone watches to alerts on the specific signals above.
The single most useful question when evaluating a platform at this scale is not what its interface looks like. It is what happens at three in the morning when a drive fails and nobody is logged in.
How Scality's management layer works
Scality RING presents one logical view across on-premises, cloud and edge locations, with policy applied once rather than per system. Different erasure coding schemes can be defined per storage class, and geo-distribution rules control which sites hold which fragments — so archive data and active data do not have to carry the same overhead or the same placement.
RING repairs itself after a disk failure by rewriting the affected data across the remaining drives in the server, and it rebuilds only the data that was actually written rather than the full capacity of the drive. Scality's own engineering guidance notes that multiple simultaneous disk failures have historically required manual intervention — worth knowing, because it is the honest version of "self-healing".
Scality ARTESCA is managed through a web UI aimed at generalists rather than storage specialists, with Grafana available for metrics and Prometheus underneath it. Audit logs can be forwarded to an external SIEM — Splunk, Graylog, Elasticsearch or syslog — and a SuperAdmin REST API published as an OpenAPI specification covers account management. S3 Lifecycle policies are supported per bucket, with prefix and tag filters and the full set of transition and expiration actions listed above. Multiple ARTESCA clusters are managed together through its fleet management component, Maestro.
Both platforms support S3 Object Lock in governance and compliance modes, which is the part of the management layer that determines whether a retained copy is genuinely unchangeable.