Most storage failures announce themselves. A drive stops responding, a server drops off the network, an alert fires. Silent data corruption does not. A few bits on a disk change, or a block is written to the wrong place, and the storage system returns the damaged data on the next read as if nothing had happened. The common name for the slow, unannounced decay of stored data is bit rot, and at petabyte scale it is a question of when, not if.
The defense against it is not a better drive. It is a system that records what each piece of data should look like, checks it regularly, and repairs it from a good copy before the damage spreads to every copy that exists. This post looks at where silent corruption comes from, how object storage detects it, and how to tell whether detection is actually running.
Corruption can enter at several points between an application and the physical media:
The individual probabilities are small. Multiplied by billions of objects and years of retention, they are not.
Replication and erasure coding protect against losing data. They do not, by themselves, protect against storing the wrong data. If a corrupted fragment is never read, the corruption is never noticed. If it is read and returned without verification, the client receives bad data. And if a repair process copies a corrupted fragment to rebuild another, the damage is multiplied.
The weak point is time. Corruption that goes undetected for long enough can coincide with an ordinary failure elsewhere. A cluster that believes it has three good copies, of which one is silently corrupted, effectively has two, and losing a server then leaves one.
Detection rests on checksums: small values computed from data when it is written, stored separately, and recomputed when the data is read. A mismatch means the data has changed. Object storage applies checksums at several layers, and each catches a different class of problem.
| Check | When it runs | What it catches | What it misses |
|---|---|---|---|
| Transfer checksum on upload | When the client writes the object | Damage between client and storage | Anything that happens after the write |
| Per-fragment or per-block checksum on read | Every time data is read | Corruption in data that is actively read | Data that is rarely or never read |
| Background scrubbing | On a schedule, across all stored data | Corruption in cold data before it is needed | Damage that occurs between scrub passes |
| Application-level fixity check | When the owner of the data runs it | End-to-end changes, including software defects | Nothing on its own; it depends on someone running it |
Read-time checks protect hot data. Background scrubbing protects everything else, and it is the layer that most often turns out to be disabled, throttled to nothing, or never completing a full pass. Archives and long-retention data, which are read least, depend on it most.
A scrub reads every stored byte, recomputes its checksum and compares the result. At petabyte scale a full pass is a large background workload, and it competes with client traffic, rebuilds and rebalances for the same disks. The usual compromise is a continuous, throttled scrub that completes a full pass over a defined period, such as a few weeks or a quarter.
The period matters. A scrub cycle sets the longest time corruption can sit undetected in cold data, which is the window in which a second failure could turn a recoverable problem into data loss. Shorter cycles cost more I/O. Longer ones leave a wider window. Either way, the cycle should be a stated figure, measured and reported, not an assumption.
Detection is only useful if it leads to repair. When a checksum mismatch is found, the system should treat the affected fragment or copy as missing, reconstruct it from the remaining good copies or fragments, verify the result, and write it to healthy media. The event should also be logged, because a cluster of mismatches on one drive or server is an early warning of hardware about to fail.
The same verification logic applies when data is moved deliberately. Migrations and hardware refreshes are moments when corruption can be introduced or carried over, and the approach in the post on verifying migrated data is the end-to-end version of what scrubbing does continuously. Rebuild speed also matters here, because a corruption repair is a small rebuild, and the factors discussed in what affects recovery time apply.
Scality RING protects data with erasure coding or replication, selectable per policy, and distributes data across nodes so that reconstruction draws on many devices in parallel. Self-healing runs as a background task on the same cluster that serves clients, which is the mechanism by which missing or damaged data is restored to full protection without operator intervention.
RING emits operational metrics and logs that the operator keeps locally, so background activity, repair events and their distribution across drives and servers can be monitored rather than assumed. That visibility is what turns integrity checking from a property claimed in documentation into one an operations team can report on.
Silent corruption cannot be prevented entirely. It can be found early, repaired from good copies and tracked over time, and that is the difference between a statistic in a reliability model and an incident.