Glossary
Distributed storage
Distributed storage spreads data and its protection across many independent servers, often in several racks or sites, so that the failure of one drive, server or location does not make data unavailable. Surviving nodes detect a loss and rebuild the missing pieces elsewhere, provided the cluster has free space to rebuild into.
No central box to fail
A traditional array depends on its controllers: if both fail, every application on it stops. A distributed system has no such centre. Each piece of data is stored on several nodes, either as full copies, as in data replication, or as data and recovery fragments through erasure coding, which needs less raw capacity for the same protection. Clients can reach data through any healthy node, and hardware is replaced a few servers at a time while the system keeps serving.
Fault domains decide what is survived
A fault domain is anything that can fail as one unit. Distributed systems place pieces so that losing one domain still leaves enough to read and rebuild the data. Designs differ mainly in the largest domain they are laid out to survive.
| Fault domain | Typical cause | What surviving it requires |
|---|---|---|
| Drive | Media wear or a failed controller chip | Pieces of each object on different drives |
| Server | Motherboard, power supply or operating system fault | Pieces on different servers |
| Rack | Shared power feed or top-of-rack switch | Pieces spread across racks, with enough racks to hold them |
| Site | Power loss, fire or network isolation | Pieces or copies in other sites, and links sized for the traffic |
Why does a nearly full cluster struggle to heal?
Repair needs somewhere to put the rebuilt pieces. After a server fails, the survivors recreate its share of data on their own free space. A cluster running close to full has little room to do so, and it stays in a degraded state, exposed to a second failure, until hardware is added. Free capacity is therefore part of the protection, and the amount needed grows with the size of the largest domain the design has to absorb. The network sets the other half of the timing: a rebuild moves a failed node's worth of data between survivors, and congested links stretch a repair of minutes into days. Larger clusters recover faster, because more nodes share the work.
Distributed, scale-out and durable are separate claims
Distributed describes where data and its protection live. Scale-out describes growth by adding nodes, and most distributed systems grow that way. High availability is about staying online through failures, while data durability is about how unlikely permanent loss is over time. Object stores, scale-out file systems and many cloud services are built on distributed storage, and a distributed file system applies the same idea to a folder tree.
Frequently asked questions
Is distributed storage the same as cloud storage?
No. Public cloud storage is one example of a distributed system, run by a provider. The same design runs on servers an organization owns, in its own data centers.
Does distributed storage replace backup?
No. It protects against hardware and site failures. A deleted or encrypted file is stored just as faithfully on every node, so separate point-in-time copies are still needed.