Archive storage refresh is the problem every long-retention archive eventually meets: the data has to outlive the hardware it sits on. An archive kept for twenty or thirty years will pass through four or five generations of servers and drives. Each transition is a chance to lose data, to break the chain of evidence that it has not been altered or to spend a year on a migration project. How an archive handles hardware change matters more to its long-term cost and integrity than almost any feature it has on day one.
This post looks at what moving archive data across hardware involves, the ways it is done and what has to be preserved along the way.
Retention outlasts everything beneath it
Retention periods for regulated and institutional archives are commonly measured in decades. Medical images, engineering records, financial ledgers, research data and audiovisual collections are all kept far longer than any storage device is supported. Drives are typically replaced within five to seven years, servers on a similar schedule, and tape drives read only a limited range of older cartridge generations. An archive is therefore never finished being migrated. The only question is how each migration is carried out.
What has to survive a move
Copying the bytes is the easy part. An archive is useful only if several other things arrive intact as well.
| Element | Why it matters | How it is lost |
|---|---|---|
| Object data | The content itself | Silent corruption during copy, unread media errors |
| Checksums and fixity records | Proof the content is unchanged | Recalculated on arrival instead of carried over and compared |
| Metadata and tags | Context, ownership, classification | Tools that copy data and drop attributes |
| Original timestamps | Evidence of when a record was created | Replaced by the migration date |
| Retention and legal hold state | Compliance with retention rules | Locks not reapplied, or reapplied from the wrong date |
| Version history | Record of changes | Only the current version copied |
| Identifiers and paths | References held by applications and catalogs | New naming on the target breaks every pointer |
Retention state is the item most often mishandled. A record with seven years left on a ten-year retention should arrive with seven years left, in the same mode, and should not be deletable at any moment in between. The post on object storage integrity verification covers how checksums are carried and checked.
Migration as a project
The traditional approach treats each refresh as a discrete migration. A new system is installed, a tool copies everything across, the copy is verified and the old system is retired. For a small archive this is manageable. For hundreds of terabytes or more it becomes a long project, bounded by how fast the old system can be read. Tape archives are slowest, since every cartridge has to be mounted and read end to end.
The project approach carries predictable costs. Both systems run in parallel for the duration, applications must be repointed, and the archive's integrity depends on the migration tool handling every row in the table above. It is also repeated at each generation, so an archive kept thirty years pays for it four or five times. The general pattern is described in the post on S3 migration.
Refresh in place
Scale-out storage offers an alternative in which the archive never moves as a whole. New servers are added to the existing system. The software redistributes data onto them in the background, verifying checksums as it goes. Old servers are drained and removed when empty. From the application's point of view nothing has changed: the endpoint, the buckets, the object names, the metadata and the retention state are the same before and after.
This turns refresh from a periodic project into routine operations. There is no parallel system, no repointing and no step at which retention has to be reapplied, because the objects were never rewritten through the S3 interface. The mechanics are covered in the post on replacing aging nodes, and running old and new generations side by side in mixed hardware clusters.
Integrity between refreshes
Data in an archive is read rarely, which means corruption can go unnoticed for years. Media degrades, and a drive can return wrong data without reporting an error. An archive that only discovers damage when it tries to migrate has discovered it too late. Periodic scrubbing, in which the system reads data in the background, compares it with stored checksums and repairs from redundancy, keeps the archive healthy between hardware generations. The post on silent data corruption explains the failure modes.
The protection scheme also deserves a look at each refresh. Larger drives take longer to rebuild, which changes the durability arithmetic, and a scheme chosen ten years ago may not suit the hardware being installed now.
Formats and software age too
Hardware is one layer. An archive also depends on the storage software continuing to exist and on the access protocol continuing to be spoken. A proprietary archive format readable only by one product ties the archive's life to that product's. A widely implemented interface such as S3 reduces that dependency, since many applications and tools can read it, and if the storage software itself must one day be replaced, a standard interface is what makes the data extractable. File formats of the content are a separate preservation question for the archive's owners.
Where Scality RING fits
Scality RING is software-defined object and file storage that runs on standard x86 servers, and nodes and servers can be added to a live system without taking the namespace offline. That is what allows a single RING deployment to span several hardware generations and to retire old nodes while new ones take over, without a forklift migration, which is the refresh-in-place model described above.
RING supports versioning and S3 Object Lock, and because data is redistributed beneath the S3 interface, objects keep their names, metadata and retention state through a hardware change. Self-healing runs as a background task, erasure coding and replication are selectable per policy, and RING is built to scale from petabytes upward.
Questions for a long-lived archive
- How many hardware generations will the retention period span?
- At the last refresh, were checksums carried over and compared, or recalculated?
- Does retention state survive a hardware change without being reapplied?
- How often is archived data read and verified in the background?
- Could the data be read by another tool if the current software were retired?
An archive is a commitment that lasts longer than any of its components. Choosing a platform that can replace those components underneath the data, without the data noticing, is what keeps that commitment affordable.