Glossary
AI data infrastructure
AI data infrastructure is the physical and software components that carry data through AI work: storage, networking, compute and the data services that run across them. It is judged by whether costly accelerators stay busy, and it sits below the models and applications that use it.
The four layers
| Layer | What it holds or does | Load it has to absorb |
|---|---|---|
| Storage | Source datasets, prepared data, checkpoints, model versions and retrieval corpora | Parallel reads in training, write bursts at checkpoints, many small reads in retrieval |
| Network | Moves data between storage and compute, and between compute nodes | Sustained bandwidth at peak, with every training worker reading at once |
| Compute | GPUs and CPUs that prepare data, train models and serve them | Utilization falls whenever data arrives late |
| Data services | Access control, protection, replication, tiering and movement | Policies applied consistently across the other three layers |
AI storage is the first row of this table seen on its own. Models, frameworks and applications sit above all four rows and are outside the term.
Why do idle accelerators point to a data problem?
AI budgets are often built around GPUs, with storage and network sized from what is left. The result shows up as accelerators that are billed, powered and depreciating while they wait for data. Adding more of them does nothing when the limit is the data path, and a cluster of 64 GPUs running at half utilization delivers roughly the work of 32 at the price of 64. The path from drives to GPU memory is covered under GPU storage.
Sizing starts with read rate, not capacity
Capacity is the easiest number to state and the least useful for sizing. The figure that decides most designs is aggregate read rate at peak, while other jobs share the same systems. Storage throughput governs the large reads of training. Latency governs the small lookups of inference and retrieval. The peak figure has to hold while data preparation, retrieval services and backup jobs share the same storage and links, since those workloads rarely pause for training. Two further questions tend to be deferred and then become costly: where checkpoints are written and for how long they are kept, and where datasets and model versions go once they stop being active, which is a question for storage tiering.
Where infrastructure stops
Infrastructure is the hardware and software that execute a plan. The plan itself is the AI data architecture. An AI data platform packages a share of these layers as a product with governance built in. A site can run fast infrastructure and still be unable to say which data trained a model, because that record belongs to data management, not to the hardware.