Glossary

GPU Storage

GPU storage is data storage infrastructure designed to supply graphics processing units (GPUs) with data at the throughput, latency and concurrency required for accelerated computing workloads.

It is commonly used for artificial intelligence (AI), machine learning (ML), high-performance computing (HPC), analytics and other applications that process large datasets in parallel.

As GPU processing performance increases, storage becomes an important part of overall system performance. A GPU can only process data as quickly as the surrounding infrastructure can deliver it. When storage, networking or data movement cannot keep pace, GPUs may spend processing cycles waiting for data rather than performing useful computation.

GPU storage addresses this challenge by combining high-performance storage media, scalable storage architectures, high-bandwidth networking and optimized data paths that keep data moving efficiently between storage systems and GPU memory.

How does GPU storage work?

GPU-based applications typically process large quantities of data in repeated stages. An AI training workload, for example, may continuously retrieve training samples, process them across many GPUs, write checkpoints and then read additional data for subsequent training iterations.

The storage system must support these operations without becoming a persistent bottleneck.

A simplified data path looks like this:

Storage → network → compute server → GPU memory → GPU processing

In conventional architectures, data retrieved from storage may first pass through CPU memory before being copied into GPU memory. These additional data movements consume CPU resources, memory bandwidth and time.

Technologies such as NVIDIA GPUDirect Storage (GDS) can create a more direct data path between local or remote storage and GPU memory. GDS uses direct memory access (DMA) transfers to avoid unnecessary copies through CPU memory, reducing CPU overhead while improving potential throughput and latency.

GPU storage is therefore broader than a particular storage device or protocol. It describes the infrastructure used to store, retrieve and move data efficiently enough to support GPU-accelerated applications.

Why is storage performance important for GPUs?

GPUs are designed for highly parallel computation. Modern accelerators can process enormous numbers of operations simultaneously, which creates substantial demand for data.

If the storage infrastructure cannot supply that data quickly enough, GPU utilization can fall. Expensive compute resources then spend more time waiting for I/O.

NVIDIA describes this challenge as GPU starvation: as GPU computing performance increases, applications require correspondingly higher I/O bandwidth to keep accelerators supplied with data.

Storage performance can affect several parts of an AI workload:

• Dataset loading: Training data must be transferred from storage to compute infrastructure.
• Model initialization: Large model weights may need to be loaded before processing begins.
• Training: GPUs continuously consume batches of training data.
• Checkpointing: Training systems periodically write model state to durable storage.
• Fine-tuning: Specialized datasets may be repeatedly accessed during model adaptation.
• Inference: Models, embeddings and related data may need low-latency access to serve requests.
• RAG: Retrieval-augmented generation systems access documents, chunks, embeddings and other knowledge data during retrieval workflows.

Storage requirements vary considerably across these stages. Training may emphasize sustained throughput, while inference and retrieval workloads can place greater emphasis on latency, concurrency and small-object performance.

What are the key requirements for GPU storage?

GPU storage architectures generally need to balance several performance and data management characteristics.

High throughput

Throughput measures how much data a storage system can transfer over a given period.

Large AI training environments can involve hundreds or thousands of GPUs simultaneously reading data. Storage must provide sufficient aggregate bandwidth to serve these accelerators without creating an I/O bottleneck.

Performance therefore needs to scale beyond the capabilities of an individual drive or storage server.

Low latency

Some GPU workloads depend on rapid access to relatively small pieces of data.

Inference, vector search, embeddings and metadata-intensive workloads can involve large numbers of smaller requests rather than continuous streams of large files. In these environments, storage latency can be as important as aggregate bandwidth.

High concurrency

GPU clusters distribute workloads across many accelerators and compute nodes. These systems can generate large numbers of simultaneous storage requests.

The storage architecture must maintain predictable performance as the number of clients, objects and operations increases.

Scalability

AI datasets can grow from terabytes to petabytes or exabytes. Storage infrastructure therefore needs to scale capacity without requiring frequent architectural changes.

Scale-out storage architectures distribute data and I/O across multiple storage nodes, allowing capacity and aggregate performance to increase as infrastructure expands.

Efficient data movement

Storage performance alone does not determine how quickly data reaches a GPU.

Network bandwidth, protocol overhead, CPU processing and memory copies can all affect the end-to-end path. Technologies such as RDMA and GPUDirect Storage can reduce some of these overheads by creating more efficient paths between storage, networking hardware and GPU memory.

Data durability and resilience

AI data often represents significant investment in collection, preparation and computation. Training datasets, checkpoints, model artifacts and enterprise knowledge bases therefore require protection against hardware failure, corruption, ransomware and operational errors.

GPU storage must balance performance requirements with the durability and resilience expected from enterprise data infrastructure.

What types of storage are used with GPUs?

There is no single storage technology required for GPU computing. Different media and architectures can serve different stages of an AI data lifecycle.

NVMe flash storage

NVMe SSDs provide high throughput, low latency and substantial parallel I/O performance.

They are frequently used for performance-sensitive AI workloads such as active training datasets, inference data, model loading and other workloads where storage response time directly affects GPU utilization.

Distributed file storage

Parallel and distributed file systems can provide multiple GPU servers with concurrent access to shared datasets.

File interfaces are common in HPC and AI environments because many existing frameworks and applications expect POSIX-style file access.

Object storage

Object storage provides a scalable way to manage the large volumes of unstructured data associated with AI.

Images, video, audio, documents, training datasets, model artifacts and checkpoints can all be stored as objects. S3-compatible APIs are also widely supported by cloud-native applications and AI data tools.

Enterprise adoption reflects this role. Independent research sponsored by Scality involving 504 enterprises running private AI found that more than 90% reported meaningful use of object storage in AI applications and pipelines.

High-performance object storage can also participate directly in performance-sensitive parts of AI infrastructure rather than serving only as a capacity or archive tier.

HDD-based storage

Not every byte associated with GPU workloads requires flash performance.

Large source datasets, historical training data, checkpoints and other capacity-intensive data can make all-flash infrastructure unnecessarily expensive. Scale-out HDD storage can provide high aggregate throughput when workloads involve large datasets and parallel access.

This creates an opportunity to match storage media to workload requirements instead of maintaining an entire AI dataset on the fastest available tier.

GPU storage vs. GPU memory

GPU storage and GPU memory serve different purposes.

GPU memory, commonly called VRAM or high-bandwidth memory (HBM), is memory located on or closely connected to the GPU. It holds the data, model parameters and intermediate results that the GPU is actively processing.

GPU storage provides persistent storage for datasets, models, checkpoints and other data used by GPU applications.

GPU memory is much faster but has far less capacity and is generally volatile. Persistent storage provides substantially greater capacity and retains data independently of the GPU.

AI infrastructure therefore continuously moves data between persistent storage and GPU memory. Improving this path is a central goal of high-performance GPU storage architecture.

What is GPUDirect Storage?

NVIDIA GPUDirect Storage is a technology that enables direct memory access transfers between GPU memory and storage.

Without GDS, storage data may travel through system memory before reaching GPU memory. This introduces an additional copy and consumes CPU and memory resources.

With GPUDirect Storage, supported storage and network infrastructure can use a more direct path to GPU memory. NVIDIA states that this architecture can reduce CPU utilization, decrease latency and increase system bandwidth.

GPUDirect Storage supports file-oriented environments through NVIDIA’s cuFile interfaces. NVIDIA also provides cuObject interfaces for object storage implementations using GDS data paths.

GDS does not replace the storage system itself. The underlying storage architecture must still provide sufficient performance, capacity, availability and scalability to support the workload.

What is the role of object storage in GPU infrastructure?

AI workloads generate and consume large amounts of unstructured data, making object storage a natural foundation for many GPU environments.

A single AI pipeline can include:

• Raw images, video, audio and sensor data
• Documents and text corpora
• Prepared training datasets
• Model weights
• Training checkpoints
• Embeddings
• RAG knowledge bases
• Generated outputs
• Archived datasets and previous model versions

Object storage can maintain these datasets in a shared namespace while exposing them through APIs that integrate with AI applications and data platforms.

The performance characteristics required from that object store depend on where data sits in the AI pipeline.

Capacity-heavy ingestion and archival workloads may prioritize scalability, durability and economics. Training and inference can require much higher throughput or lower latency.

This distinction is leading to architectures in which object storage spans multiple performance tiers rather than being treated exclusively as secondary storage.

How does GPU storage support AI training?

Training is one of the most storage-intensive GPU workloads.

Before training begins, potentially enormous datasets must be collected, cleaned, transformed and made accessible to GPU compute nodes. During training, multiple workers repeatedly retrieve batches of this data.

Storage infrastructure must maintain enough throughput across concurrent GPU nodes to prevent the data pipeline from constraining compute.

Checkpointing creates an additional requirement. AI systems periodically save the current state of a model so training can recover from failures or resume later. At scale, many GPU workers can write checkpoints simultaneously, creating bursts of storage traffic.

A GPU storage architecture therefore needs to support both sustained read throughput and substantial write activity while preserving checkpoint durability.

How does GPU storage support AI inference?

Inference has different storage characteristics from model training.

A production inference environment may need to load model weights quickly, access embeddings or vector data, retrieve contextual information and serve many requests concurrently.

The resulting storage workload can emphasize:

• Low latency
• Small-object performance
• High transaction rates
• Predictable response times
• High concurrency

These requirements become particularly important for real-time AI applications and RAG architectures, where storage and retrieval operations can occur as part of the request path.

How can organizations prevent GPUs from waiting for storage?

Improving GPU utilization requires looking at the complete data path rather than focusing only on the storage media.

Organizations can evaluate:

1. Storage throughput and latency relative to aggregate GPU demand.
2. Network bandwidth between storage and GPU compute infrastructure.
3. Concurrency under realistic numbers of GPU workers.
4. Dataset characteristics, including object or file sizes and access patterns.
5. CPU overhead associated with moving data through the system.
6. Data placement so frequently accessed datasets use appropriate storage media.
7. Scaling behavior as additional GPUs and storage capacity are added.

Benchmarking should represent actual application behavior. A system that performs well with large sequential reads may behave differently when thousands of clients generate small, concurrent requests.

Do all GPU workloads require all-flash storage?

No. GPU infrastructure does not automatically require every dataset to reside on flash.

Performance-sensitive working data may benefit substantially from NVMe or other flash media. At the same time, AI pipelines accumulate much larger pools of source data, checkpoints, previous model versions and archival information that may not require the same latency.

The appropriate architecture depends on how frequently data is accessed and how quickly it must reach compute.

A tiered approach can place performance-sensitive data on flash while using higher-capacity media for less latency-sensitive datasets. Scale-out architectures can also aggregate throughput across many HDDs, providing substantial bandwidth for large parallel workloads.

The goal is to deliver sufficient performance for each workload while avoiding the cost and power requirements of placing every AI dataset on the highest-performance media.

How does Scality support GPU storage?

Scality provides object storage infrastructure for both performance-intensive and capacity-intensive stages of AI data pipelines.

Scality RING XP is an all-flash configuration designed for high-performance AI workloads. It targets stages where GPU utilization and latency are especially important, including preparation, training, checkpointing and inference. Scality positions RING XP for high-throughput object access and low-latency access to small objects used in workloads such as inference, embeddings and vector databases.

Scality RING provides scale-out object storage for larger capacity-oriented AI datasets, including raw data, training corpora, RAG knowledge bases, checkpoints and archives. It can scale into petabyte and exabyte environments while providing S3-compatible access and cyber-resilience.

Scality ADI extends this model by aligning different storage media and performance levels with different stages of the data lifecycle. Its architecture spans GPU-oriented flash, hot and warm capacity tiers, and colder storage while maintaining a common data infrastructure.

This approach allows organizations to treat GPU storage as part of a broader AI data architecture: high-performance resources can serve workloads where latency and GPU utilization matter most, while larger datasets remain on storage designed for capacity, durability and long-term economics.

Frequently asked questions about GPU storage

What is GPU storage?

GPU storage is storage infrastructure optimized to deliver data at the throughput, latency and concurrency required by GPU-accelerated applications. It is commonly used for AI, machine learning, HPC and analytics workloads.

Why do GPUs need fast storage?

GPUs process data at very high rates. If storage and networking cannot deliver data quickly enough, GPUs can spend time waiting for I/O, reducing utilization and overall application performance.

Is GPU storage the same as GPU memory?

No. GPU memory, such as HBM or VRAM, holds data that a GPU is actively processing. GPU storage is persistent infrastructure that stores larger datasets, models, checkpoints and other information before and after processing.

What is GPUDirect Storage?

NVIDIA GPUDirect Storage enables direct DMA transfers between supported storage infrastructure and GPU memory, avoiding unnecessary data copies through CPU memory. This can reduce CPU overhead and improve data-transfer efficiency.

Can object storage be used for GPU workloads?

Yes. Object storage is widely used to hold AI datasets, model artifacts, checkpoints, embeddings and other unstructured data. High-performance object storage can also support performance-sensitive GPU workloads when it provides the required throughput, latency and concurrency.

Does GPU storage require NVMe?

No. NVMe is useful for workloads requiring very low latency and high I/O performance, but GPU environments can use multiple storage media. HDDs, SSDs, NVMe and archival technologies can each serve different stages of an AI data lifecycle.

What storage is best for AI GPUs?

The appropriate storage depends on the workload. Training commonly requires high aggregate throughput, while inference and retrieval workloads may place greater emphasis on low latency and high concurrency. Large AI environments often combine multiple storage tiers so data is placed according to its performance, capacity and durability requirements.