Data protection & recovery
Checksum
A checksum is a short value computed from a block of data by a fixed algorithm, used to detect whether the data has changed. The same data always produces the same checksum. If the data is altered, even by a single bit, the checksum computed afterward will almost certainly differ from the one stored earlier.
How does a checksum work?
When data is written, the system runs it through the algorithm and stores the result alongside it, typically in object metadata. When the data is read, transferred or audited, the checksum is computed again and compared. A match means the data is, to a very high probability, exactly what was written. A mismatch means something changed in storage or in transit.
Which algorithms are used?
Algorithms differ in speed and in what they are designed to resist.
| Algorithm | Type | Typical use |
|---|---|---|
| CRC32, CRC32C | Error-detecting code, very fast | Catching accidental corruption in storage and networking |
| MD5 | Older cryptographic hash | Legacy integrity checks, S3 entity tags |
| SHA-1 | Older cryptographic hash | Legacy systems |
| SHA-256 | Cryptographic hash | Integrity with resistance to deliberate tampering |
| xxHash and similar | Non-cryptographic, very fast | High-throughput internal verification |
Where are checksums verified?
A well-designed data path checks at several points. The client computes a checksum and sends it with the upload, and the storage system refuses the write if what arrived does not match, which catches corruption on the network. The system stores its own checksums for each piece it writes to disk and verifies them on every read. Background scrubbing verifies data that nobody is reading. And during replication or migration, checksums are carried across and compared at the destination. The S3 API supports several algorithms for upload and download verification.
What can a checksum prove, and what can it not?
A checksum proves that data matches a reference. It says nothing about whether the reference was right, so a file corrupted before its first checksum was computed will verify perfectly. Error-detecting codes such as CRC guard against accidents and offer no protection against someone who alters data and recalculates the code. Cryptographic hashes resist that, provided the stored hash is itself protected. And a checksum detects damage without repairing it. Repair needs redundancy, which is where replication and erasure coding come in. Bit rot describes the failure checksums exist to catch, and integrity verification covers how the checks are run in practice.
Frequently asked questions
Is a checksum the same as a hash?
A hash function is one way to produce a checksum. Cryptographic hashes are checksums designed so that finding two inputs with the same value is infeasible.
Is the S3 ETag a checksum?
For simple uploads it is usually the MD5 of the object. For multipart uploads it is not a plain hash of the content, so it should not be relied on for verification.
Do checksums slow storage down?
Modern processors compute common checksums very quickly, so the overhead is small compared with disk and network time.
Related terms
- Bit rot: the silent corruption checksums detect.
- Data durability: the outcome integrity checking supports.
- Object metadata: where checksums are stored.
- Multipart upload: where each part carries its own checksum.
- Erasure coding: the redundancy used to repair what a checksum flags.