Glossary
Distributed File System
A distributed file system spreads files across many servers but presents them to applications as one ordinary file system — the same paths, directories and permissions, and the same open, read, write and seek operations.
The point is that applications do not have to know the data is distributed. Delivering that illusion at scale is where the engineering goes.
What it has to deliver
A local file system on one machine gives applications a set of guarantees they quietly depend on: a hierarchical path to every file, byte-range reads and writes at any offset, the ability to append to a file without rewriting it, ownership and permission bits, and locking so two writers do not corrupt each other.
A distributed file system has to provide all of that while the actual bytes live on dozens or hundreds of separate machines, any of which can fail. Three things get distributed:
- The data — file contents, usually split into chunks and spread across servers, with replication or erasure coding for protection.
- The metadata — the directory tree, file names, sizes, timestamps, permissions and the map of which chunks live where.
- The coordination — locks, leases and the ordering rules that decide what happens when two clients touch the same file at the same time.
Data is the easy part. Metadata and coordination are what separate systems that scale from systems that fall over.
Metadata is the hard problem
Every file operation touches metadata first. Opening a file means walking the path, checking permissions at each level, and looking up where the chunks are. That is several metadata operations before a single byte of data moves — and small-file workloads are almost entirely metadata traffic.
How a system handles this is the main architectural difference between distributed file systems:
| Approach | How it works | Where it runs out |
|---|---|---|
| Single metadata server | One node owns the whole namespace; data servers handle only bytes | The metadata node's memory and CPU. Simple and fast until it is the bottleneck, and it is a single point of failure unless paired. |
| Metadata server cluster | Several nodes share the namespace, usually by splitting the directory tree between them | Hot directories. If one directory holds millions of entries and every client writes to it, the split has not helped. |
| Fully distributed metadata | Metadata is sharded or hashed across every node, with no dedicated metadata tier | Operations that span shards — renaming a directory, or anything needing a consistent view across the whole tree. |
The practical consequence: a system that benchmarks well on large sequential files can be an order of magnitude slower on a directory of a million small files, and the number that predicts this is metadata operations per second, not throughput.
How clients actually reach it
There are two ways applications get to a distributed file system, and the choice has real performance consequences.
Through a standard protocol. The file system is exported over NFS or SMB and mounted like any other network share. Nothing has to change on the client, which is why this is the common path. The cost is that every request goes through a gateway, and that gateway becomes both a bottleneck and a failure domain unless there are several of them with traffic balanced across them.
Through a native client. Software installed on each client machine talks directly to the storage nodes, so a read goes straight to the node holding the data with no intermediate hop. This is faster and scales better, because adding clients adds parallelism instead of loading a shared gateway. The cost is a package to install and keep in step with the cluster version, which is fine on a managed compute fleet and awkward on machines you do not control.
Most deployments end up with both: native clients for the compute cluster that does the heavy work, and NFS or SMB for everything else.
Consistency, locking and the compromises
Local file systems give you strict semantics: a write that returns is immediately visible to every other reader. Providing that guarantee across a network, to many clients at once, costs coordination on every operation.
Systems make different trade-offs here, and it is worth knowing which one you have:
- Strict consistency — every client sees every write immediately. Correct, and the most expensive.
- Close-to-open consistency — a client that opens a file sees everything written by clients that have closed it. This is what standard NFS gives you, and it is usually enough. It is not enough for two processes writing to the same file simultaneously.
- Client-side caching with leases — a client is granted exclusive use of a file or a byte range for a period, caches aggressively, and gives the lease back when someone else asks. Fast in the common case; the complexity shows up when a client holding a lease disappears.
Locking is the related question. Advisory locks are honoured only by applications that check them. Mandatory locks are enforced by the file system. And a lock held by a client that has crashed has to time out somehow, which means there is always a window where a file is locked by nobody.
What failure looks like
A single server's file system is either working or it is not. A distributed one is usually partly working, which is harder to reason about.
When a storage node fails, the data on it is reconstructed elsewhere from replicas or parity, and until that finishes the affected files are running with less protection than the design promises. When a metadata node fails, the part of the namespace it owned is unavailable — files elsewhere in the tree keep working, which means some directories respond and others hang. And when the network partitions, a system that keeps serving on both sides risks divergence, while a system that stops on one side is unavailable for those clients. There is no option that avoids both.
The useful question when evaluating one of these is not whether it survives a node failure — they all claim that — but what clients experience during the recovery, and how long the recovery takes.
When a file system is the right answer
A distributed file system is the right choice when applications genuinely need file semantics: modifying part of a large file in place, appending to a file that is being read, or running software that expects a mounted path and cannot be changed.
Object storage takes the opposite approach. Objects are written and replaced whole rather than edited in place, the namespace is flat rather than a tree, and access is over HTTP rather than a mount. Giving up in-place modification is what lets object storage scale its metadata far past what a directory tree manages, which is why archives, backup targets and very large unstructured data sets usually end up there. Our entry on object storage vs NAS covers that comparison in detail.
In practice the split follows the workload. HPC and research computing, media production with large files edited in place, and analytics pipelines built on tools that expect POSIX paths all want a file system. Long-term retention, backup, and anything addressed by an application rather than browsed by a person are better served by objects.
How Scality approaches it
Scality RING provides native file access alongside its S3 interface, so file and object workloads can be served from the same underlying storage rather than from separate systems that then have to be kept in step. Files written through the file interface and objects written through S3 land in the same pool, protected by the same policies.
Protection is set by policy per storage class rather than once for the whole cluster, and geo-distribution rules control which sites hold which fragments — so a scratch area for active work and a long-retention archive can sit in one system on different terms.
If the requirement is genuinely object storage rather than file access, Scality ARTESCA is the S3-only option, validated from 20 TB to 8.5 PB.