Glossary

AI data pipeline

An AI data pipeline is the ordered sequence of stages that takes raw data to a working model and back again: ingest, prepare, store, train, checkpoint, serve and monitor. Unlike a reporting pipeline that ends at a dashboard, it loops, because production data becomes the next round of training input.

The stages in order

  1. Ingest. Records, documents, images, logs and sensor data arrive from source systems.
  2. Prepare. Data is cleaned, deduplicated, labeled and converted into a form a model can read.
  3. Store. Prepared data lands in a shared layer that many readers can reach at once.
  4. Train. Workers read the prepared set in parallel, often many times over.
  5. Checkpoint. Training state is written out at intervals so a long run can resume after a failure.
  6. Serve. The trained model answers requests, often with retrieved context.
  7. Monitor. Accuracy and drift are tracked, and new data and feedback return to step 1.

A worked example: a team retraining a fraud model each month ingests the month's transactions, prepares them with confirmed fraud labels, trains on twelve months of history, checkpoints every few hours, deploys the model and then logs which alerts analysts overturned. Those overturned alerts become labeled input for next month's run, and the previous model and its training set are kept so results can be compared.

The retrieval variant

Applications built on retrieval-augmented generation skip training and checkpointing. Documents are ingested, split by document chunking, converted to embeddings and indexed, and the serve stage retrieves passages for each request. The loop still exists: edited and deleted source documents have to flow through to the index, or answers go stale.

Where pipelines slow down

Trouble tends to sit between stages rather than inside them. Each hand-off from one storage system to another adds transfer time, cost and one more copy to track. Stages built by different teams on different storage often exchange data by bulk copy, and training waits while it happens. A shared storage layer removes most hand-offs, but it has to serve every stage at once, so an overloaded layer slows several stages together.

Pipeline compared with the terms around it

The pipeline is the sequence of work. The AI data architecture decides where each stage's data lives and how many copies exist. A data lake is a common place for the store stage to land. AI storage is the storage underneath every stage, chosen to match the different load each one creates.