A model that performed well in March cannot be rebuilt in September unless the data it learned from can be reconstructed exactly. Code and hyperparameters are usually held in version control. The dataset frequently is not. It sits on shared storage, gets corrected, extended and relabelled over months, and no record survives of which state of it produced any particular model.
For infrastructure teams this is a storage design question rather than a data science one. Reproducing a run means being able to present the same bytes, in the same grouping, at a later date, after the working copy has moved on. How that is achieved depends on what the storage layer can retain and prove, and it is easier to build in at the start than to retrofit after a regulator, a customer or an internal review asks how a model reached a decision.
What reproducing a training run actually requires
Reproducibility is often discussed as a single property, but it decomposes into several independent things that each have to be recorded. The training code and its dependencies need to be pinned. The configuration, including random seeds, needs to be stored alongside the run. The hardware and framework versions influence numerical results. And the data, which is usually the largest and least controlled of the four, needs to be identifiable after the fact.
Teams usually discover the gap in a specific way. A model behaves differently after a retrain and nobody can establish whether the cause was a code change, a framework upgrade or the 40,000 samples that were added in the interim. Without a dataset version, that question cannot be answered by inspection. It can only be answered by rebuilding the earlier run, which requires the earlier data.
The practical target is more modest than perfect bit-level determinism. Most organizations need to be able to state which samples a model saw, retrieve those samples unchanged, and show that the set has not been altered since. That is achievable with ordinary storage primitives if the design accounts for it.
Why the dataset is the hardest part to pin down
Training data changes for good reasons. Labels get corrected as annotation quality improves. Personal data gets deleted on request. New collection runs add samples. Corrupt files get removed. Each of these is a legitimate operation on the live dataset, and each one quietly invalidates the assumption that "the dataset" is a stable object.
Scale makes the usual answer unattractive. Copying a 400 TB dataset to freeze it costs 400 TB, and a team that retrains monthly will not keep twelve such copies. This is the same capacity planning problem that shapes how much capacity an AI environment needs, except that the multiplier comes from history rather than from concurrent activity.
There is also a subtler issue. A dataset is rarely just files. It is files plus the labels, plus the split between training and validation, plus the filter that excluded certain samples. A version that records the files but not the split does not reproduce the run, because a different split trains a different model on the same bytes.
Three ways teams version training data
Most approaches in production are variations on three patterns, and they differ mainly in what they cost and what they can prove.
Copy the dataset at each version
The simplest approach is to write a complete copy of the dataset whenever a version is declared, usually into a dated prefix. It is easy to reason about, easy to restore, and it needs no cooperation from applications. The cost is linear in the number of versions retained, which limits how many can be kept and how often versions can be cut.
Copying is most defensible when versions are infrequent and correspond to something meaningful, such as a model that was actually deployed. Keeping a full copy for every experimental run is rarely justifiable at scale.
Record the file list and hashes
A lighter approach stores a manifest for each version: the object keys, their sizes and a content hash for each one, plus the labels and splits. The manifest is small, often a few hundred megabytes for a very large dataset, and it can live in the same repository as the training code.
This only reproduces a run if the referenced objects still exist unchanged. The manifest is a claim about the data, not a copy of it. It works when it is paired with a storage layer that will not silently overwrite or delete the referenced objects, and it fails quietly when a cleanup job removes something the manifest still points to.
Make the storage layer keep old states
Object storage with versioning enabled retains previous states of an object when it is overwritten or deleted, so a manifest that records version identifiers rather than just keys can retrieve the exact bytes even after the live dataset has moved on. Only the changed objects consume additional capacity, which makes frequent versions affordable in a way that full copies are not.
The trade-off is that retained versions accumulate and need a retention policy, or capacity grows without anyone deciding that it should. That growth is usually invisible in a bucket listing, which is why it deserves explicit monitoring rather than an annual surprise.
What a dataset version needs to record
Whichever pattern is chosen, the record is only useful if it captures enough to rebuild the run. The items below are the ones most often missing when an old run cannot be reproduced.
| Element | Why it matters | Where it usually lives |
|---|---|---|
| Object keys and version identifiers | Identifies the exact bytes, not just the current contents of a path | Manifest stored with the run |
| Content hashes | Proves the retrieved data matches what was trained on | Manifest, verified at read time |
| Labels and annotations | A relabelled sample trains a different model from the same image | Label store or a versioned annotation file |
| Train, validation and test split | A different split produces different results from identical data | Manifest or run configuration |
| Exclusion and filter rules | Samples dropped for quality or consent reasons must stay dropped | Run configuration, applied at load time |
| Preprocessing version | Resizing, normalization and tokenization affect what the model sees | Code repository, referenced by the run |
| Deletion record | Explains legitimate gaps when a version cannot be fully rebuilt | Data governance log |
The last row matters more than it first appears. If personal data has been erased under a deletion request, an old dataset version genuinely cannot be reconstructed in full, and pretending otherwise creates a compliance problem rather than solving one. The workable answer is a record that states which samples were removed and when, so the difference between the original run and any rebuild is documented rather than mysterious.
Where preprocessing fits in the version boundary
Teams have to decide whether a version covers the raw collected data, the processed training-ready data, or both. Versioning raw data preserves the ability to reprocess with better methods later. Versioning processed data makes reproduction faster because the transformation does not have to be repeated, but it multiplies capacity if several processed variants exist.
The decision interacts with where preprocessing happens in the pipeline. If transformation runs at load time inside the data loader, the processed form never lands on storage and only the raw version can be retained, which makes the preprocessing code version part of the dataset record. If transformation produces a materialized copy, that copy is the thing worth versioning, and the raw data can often move to a colder tier.
A reasonable default for regulated work is to retain the raw data with versioning enabled, materialize processed datasets only for runs that reach production, and keep manifests for everything. That holds the capacity cost to something predictable while keeping the ability to answer questions about deployed models.
What versioning costs in practice
The capacity impact depends almost entirely on how much of the dataset changes between versions. A dataset that grows by appending new samples and never rewrites old ones costs very little to version, because the retained history is limited to deletions. A dataset that is relabelled in bulk, or reprocessed end to end, effectively creates a second copy each time.
That asymmetry is worth designing around. Storing labels separately from the sample payloads means a relabelling campaign rewrites megabytes rather than terabytes. Writing new collection runs into new prefixes rather than merging them into existing files keeps appends cheap. These choices cost nothing at design time and determine whether versioning is affordable at 10 PB.
Retention needs a stated policy in the same way large training datasets need a storage plan. A common structure keeps versions tied to deployed models for as long as those models are in service plus an audit margin, keeps experimental versions for a few months, and expires the rest automatically. Without a policy, retained versions become the fastest growing part of the environment and nobody notices until a capacity review.
Where Scality RING fits in the versioning design
Scality RING presents an S3 interface with object versioning, so a dataset that lives on RING can retain previous object states while the live prefix continues to change. A manifest that records version identifiers can then retrieve the exact objects a past run consumed, without a separate frozen copy of the dataset existing anywhere.
Because RING is designed for petabyte and exabyte scale on a single namespace, the retained history does not need to be split across systems as the dataset grows, and lifecycle rules can expire old versions on a schedule rather than through manual cleanup. Object lock can hold selected versions immutable for a defined period where a model needs to be defensible to an auditor, which keeps the evidence outside the reach of the pipeline that would otherwise overwrite it.
The design question for an infrastructure team is which prefixes carry versioning and object lock, and for how long. Applying both everywhere is expensive and usually unnecessary. Applying them to the prefixes that feed production models, and to the manifests themselves, covers most of what a reproducibility or audit request will ask for.
Judge the design by whether an old run can be rebuilt
The only reliable test of a versioning scheme is an exercise: pick a model that is currently in production, retrieve the dataset version it was trained on, and confirm the sample count, the hashes and the split all match what the run recorded. Teams that try this usually find one missing element, most often the label state or the exclusion list, and fixing it takes hours rather than the months it takes to rebuild the discipline after an audit finding.
Run the exercise on a real model rather than a small test case, because the failure modes appear at scale: manifests that take too long to verify, objects that were tiered and now take hours to recall, and cleanup jobs that removed data nobody realized was referenced. A scheme that survives that test will also survive the question of why a model made a particular decision, which is the question these records exist to answer.