Glossary
AI data management
AI data management is the ongoing practice of recording, governing and protecting the data used to build and run AI systems, from collection to deletion. Its distinctive burden is proof after the fact: which data trained or grounded a model, where it came from, and whether its use was permitted.
Governance stage by stage
Each stage of the lifecycle adds something that a later audit, deletion request or retraining run will need. Management is the discipline of capturing it at the moment it is created.
| Stage | What is recorded or enforced | What breaks without it |
|---|---|---|
| Collect | Source, licence or consent basis, owner | Use cannot be shown to be permitted |
| Ingest and prepare | Version, transformations applied, sensitivity tags | A training set cannot be rebuilt |
| Train or ground | Exact dataset versions each model or index used | Model behaviour cannot be traced to data |
| Serve | Permissions carried into retrieval indexes and caches | Restricted documents surface in answers |
| Retain | Retention period and storage tier | Data is kept forever or removed too early |
| Archive or delete | Every copy and derivative linked to the source | Deleted records survive in indexes and training sets |
What makes AI data harder to govern?
Classic data management keeps records accurate and available for reporting. AI adds four pressures. A model is a product of exact dataset versions, so a silent change to a dataset silently changes the model. Derived data multiplies: chunks, embeddings, labels and checkpoints all descend from sources and each needs a link back. Deletion is hard to complete once a record has spread into training sets and indexes. And permissions have to survive the move from source system to retrieval index, which in retrieval-augmented generation is where documents meet users directly.
Metadata defined before data lands
Nearly every one of those failures traces back to missing metadata. Tags for source, version, owner and permission are cheap to attach on write and slow to reconstruct across billions of objects years later. A data fabric or catalog can only report what sources recorded, so the schema of required tags is agreed before ingestion starts. Required tags also cover derived data, so an index or checkpoint carries the identity of the source versions it was built from. Retention follows the same records: reproducibility argues for keeping training data longer, and consent withdrawals argue for removing it sooner.
Neighbours in the AI data cluster
Management is continuous operation. AI data readiness is a point-in-time judgment on one dataset for one project, and it can only be answered where management has kept records. AI data infrastructure carries the data; management records what it is and who may use it. An AI data platform may supply the tooling, but the policies inside it are organizational decisions.
Frequently asked questions
What is the minimum metadata for AI data management?
Source, version or date, owner and access permissions answer most later questions. Sensitivity classification and the training run or index each object contributed to are common additions.
Who is accountable for AI data management?
Usually a data governance function, with storage, data engineering and AI teams implementing the controls. A working check is whether one person's data can be found and removed across training sets and indexes.