A dataset that sits in one approved location at the start of an AI project rarely stays that way. Preparation, training and inference each produce their own copies, and most of them are created by tooling rather than by a person making a placement decision. By the time a model is serving requests, the same underlying records exist in a dozen forms, in several systems, reachable by people who were never granted access to the original.
The usual control is to locate the source data correctly and treat everything downstream as derived and therefore uninteresting. That works only if derived artifacts contain less than the source, and many do not. A shard file is a verbatim copy with a different name, a cache on a GPU node holds recent batches in the clear, and embeddings can be close enough to the original text that reconstruction is a practical concern.
The second issue is access rather than location. An artifact stored in the correct country but readable by a platform team, an experiment tracker and a shared notebook environment has a wider audience than the source system it came from. Sovereignty questions are answered by naming both the place and the people.
Ingest creates the first duplication. Raw records are pulled from production systems into a landing area, usually with fewer access controls than the system of record, because the pipeline needs to read everything and the team needs to inspect samples. That landing area is frequently the longest-lived copy in the project, since nobody deletes it while the pipeline is still being tuned.
Preparation multiplies it further. Cleaning, deduplication, tokenization and format conversion each write a new dataset, and the intermediate outputs tend to survive because reproducing them is expensive. Sharding then splits the curated set into files sized for the loader, which is where a single logical dataset becomes thousands of objects whose names no longer describe their contents. Decisions about where preprocessing runs determine where those intermediate copies land.
Training adds copies that were never meant to persist. Data is staged onto local NVMe on GPU nodes to keep the accelerators fed, and those caches hold recent batches in readable form. They are cleared when the job ends in theory, and in practice they survive crashed jobs, preempted nodes and machines reimaged on a slower schedule than anyone assumes.
Checkpoints are the largest derived artifact and the easiest to overlook, since they are treated as machine state rather than data. A checkpoint written every few minutes across a long run produces a substantial archive, and where it lands is usually decided by whichever storage was fastest rather than by policy. Retention for checkpoints is rarely defined, which is how they become the oldest surviving copy of a run.
Vector indexes deserve specific attention. Embeddings are derived, but they are derived in a way that preserves semantic content, and the index usually stores the source text or a pointer to it so that retrieval can return something readable. An index built for a retrieval system is therefore a copy of the corpus in a different shape, and it inherits whatever residency and access obligations the corpus carried.
Fine-tuned weights are the subtlest case. A model trained or adapted on a corpus encodes information from it, and memorization of rare or repeated sequences is a documented behavior rather than an edge case. Weights are also the artifact most likely to be copied elsewhere for evaluation or deployment, so the copy that leaves the boundary can be the one carrying the data with it.
| Artifact | Where it usually lives | What to record |
|---|---|---|
| Raw landing copy | Object storage bucket owned by the data team, separate from the source system | Retention date, who can list and read it, whether the source can be re-pulled |
| Curated dataset and shards | Object storage, often replicated to wherever training capacity sits | Every location holding a shard copy, and which job wrote it there |
| Node-local cache | NVMe on GPU hosts, outside the storage team's inventory | Cleanup after crashed jobs; reimage and disposal process for those hosts |
| Checkpoints | Whichever tier was fastest during the run, sometimes a different site | Retention policy, deletion owner, whether checkpoints leave for evaluation |
| Vector index | A database service, possibly managed and possibly hosted elsewhere | Whether source text is stored in the index; backup and snapshot locations |
| Prompt and output logs | Application logging stack and any hosted observability platform | Fields captured, retention, who reads them for quality review |
Inference generates a new stream of sensitive material that has nothing to do with the training corpus. Prompts contain whatever the user pasted, which in practice includes contract text, patient details, source code and internal documents. Model outputs quote from retrieved context. Both are logged by default in most application frameworks, because logging them is how teams debug quality problems.
These logs usually land in the general application logging stack rather than the governed data platform, so they inherit that stack's retention, export destinations and access list. Retrieval systems add a further layer, since retrieved chunks are often logged alongside the prompt, reproducing document content in a system nobody assessed for it. Prompt logging is part of the architecture of a retrieval system rather than an operational detail.
A useful map has one row per artifact and four columns: what it contains, where it is stored, which identities can read it, and when it is deleted. The access column changes the picture, because artifacts sit in shared buckets and shared clusters where access was granted to a team rather than to a dataset.
Service identities matter more than human ones here. Training jobs, orchestration frameworks, experiment trackers and notebook servers run under credentials that often have broad read access across a project namespace. Separating those credentials by dataset rather than by team is the mechanism that makes the map meaningful, which is the same argument as separating teams and projects at the storage layer.
Deletion closes the map. An artifact with no deletion owner and no expiry outlives the project, the model and the person who created it. Recording a deletion date when an artifact is first written is far easier than reconstructing intent later, and it turns a sprawling inventory into one that shrinks on its own.
RING runs on standard x86 servers in the customer's own facility, so the location of datasets, shards and checkpoints is a placement decision made by the operator rather than a property of a service. Multi-site deployments spread data across sites the operator has chosen, and the single namespace means copies stay addressable instead of scattered across unrelated systems.
Access is expressed through accounts, users, buckets and IAM-style policies with per-tenant isolation and quotas, which maps onto the artifact inventory directly. A curated dataset, its shards and its checkpoints can sit in separate buckets with separate policies, so the identity used by a training job is not the identity that can read the raw landing copy. Scality's ADI direction addresses AI data infrastructure more broadly.
Write the artifact map before the first training run rather than after, and keep it beside the pipeline definition so it changes when the pipeline does. Each new output path added to a job is a new row. Version identifiers help, since reproducing a run requires knowing exactly which copies were used.
Audit the places the map does not cover. List the GPU nodes and check what remains in local caches after a failed job. Check where the experiment tracker stores artifacts it was handed, and whether the vector database has snapshots in a managed backup. These copies appear in no architecture diagram.
Set a quarterly review that asks one question per row: is this copy still needed. Landing copies, intermediate datasets and old checkpoints accumulate because deletion is nobody's task, and deciding what to keep after training is the single change that most reduces the number of places the data exists.