GPU starvation occurs when an accelerator waits for the data or work it needs to continue processing. Storage can cause that wait, but so can image decoding, CPU preprocessing, network congestion or a data loader that cannot prepare batches quickly enough. Low GPU utilization tells you to investigate; it does not identify the component responsible.
To identify a storage bottleneck, establish whether missing input data causes the idle periods, then trace the delay back through the input pipeline. The strongest evidence combines a workload trace, measurements from the storage and network paths, and a controlled test that changes where the data comes from. That gives infrastructure and AI teams a basis for deciding what to fix before adding capacity or buying faster hardware.
In a training workload, data moves through several stages before the GPU can use it. The application retrieves records, decodes or transforms them, assembles batches and transfers those batches into device memory. Prefetching allows some of this work to happen while the GPU processes the previous batch.
Starvation happens when the next batch is not ready when needed. A job may alternate between short bursts of computation and idle gaps, with samples processed per second falling below its demonstrated potential. The question is which stage failed to deliver on time.
Distinguish steady training from startup and scheduled pauses. Loading model weights, discovering a dataset, running validation and writing checkpoints can all produce periods of low GPU activity. These events deserve attention when they delay useful work, but diagnosing them as slow training reads can send the investigation in the wrong direction.
Inference also needs context. A lightly used service may have idle GPUs because requests are infrequent, while a busy service may wait on retrieval, preprocessing or model loading. The diagnostic sequence below focuses on training and batch processing, where input demand is sustained and easier to compare.
Capture a representative section of the workload after initialization. Record batch availability, device transfers, GPU computation and any distributed synchronization, then align those events with infrastructure metrics. A framework profiler can show CPU and device activity, although data-loader subprocesses or remote reads may need separate instrumentation.
The useful observation is whether a missing batch delays the next computation. Time spent retrieving data is not necessarily lost GPU time: prefetching may hide that work behind computation already in progress. Concentrate on the waits that remain visible to the training loop.
For distributed training, inspect individual workers as well as the aggregate. One worker with slow input can arrive late at a synchronization point and hold up the others. A cluster average can obscure that imbalance and make an input problem resemble a communication problem.
| Observed pattern | What to investigate next |
|---|---|
| GPU idle gaps coincide with waits for the next batch | Reads, preprocessing and data-loader scheduling |
| CPU decoding stays busy while reads complete promptly | Preprocessing cost and available CPU resources |
| Read latency rises as more jobs start | Shared storage, network or gateway contention |
| One worker repeatedly reaches synchronization late | That worker’s input path, data assignment and host |
| Pauses match checkpoint saves | Serialization, write completion and read/write contention |
| Slow startup is followed by steady processing | Dataset discovery, model loading and cache population |
These patterns narrow the investigation; none proves a storage bottleneck on its own. Keep the job’s throughput alongside utilization throughout the process. A smoother utilization chart matters only if it corresponds to useful work completing faster or more predictably.
Break input preparation into retrieval, decoding, transformation and batch assembly where the framework permits. A long wait for the next batch can include all of these stages, plus time waiting for a worker to run. Calling the entire interval “storage latency” overstates what the measurement establishes.
Check CPU allocation at the workload level, particularly in containers. A server may have spare cores while a training job hits its CPU quota or its loader relies on one busy process. Host memory pressure and swapping can also interfere with input preparation without indicating a storage-system performance limit.
In PyTorch, the data loader’s worker count controls whether and how much loading happens in subprocesses. Increasing workers can overlap input preparation with training, but the useful setting depends on the workload and available resources. Test a small range while tracking batch wait time, CPU use and job throughput.
If more workers reduce waits without increasing read latency substantially, insufficient concurrency was likely part of the problem. If throughput stops improving while request latency climbs, additional workers may be adding contention. Memory settings and host-to-device transfers deserve separate measurement; faster transfers cannot correct slow reads upstream.
The most useful comparison changes one part of the pipeline while keeping the model, batch size, sample shapes and processing steps consistent. Use a representative dataset subset and repeat each run long enough to get beyond startup effects. Record cache state so a warm run is not mistaken for a faster storage configuration.
Run a diagnostic using prepared inputs that removes normal data fetching and preprocessing. This estimates how the workload behaves when input preparation is no longer the limiting factor. Preserve representative input dimensions and execution paths, especially for workloads with variable sequence lengths or content-dependent processing.
A large throughput improvement places the input pipeline under suspicion. It does not isolate storage because several stages were removed together. Little improvement suggests that compute, communication or another part of execution deserves attention first.
Next, place the same raw dataset subset on local storage and retain the normal preprocessing and batching steps. Compare this with the shared-storage run using the same sample order where practical. Keep the loader configuration unchanged so the data source is the principal difference.
If local reads improve performance, the shared read path is implicated. That path can include the client library, connection pool, gateway, network and storage service. The result supports investigating those components; it does not establish that the storage media need replacing.
A dataset that fits in cache may run much faster after its first pass. That can be acceptable for a repeatedly reused working set, but it may conceal problems with new datasets or concurrent jobs. Document which caches are involved rather than simply labeling a run “cached.”
Use an isolated test to evaluate cold access instead of clearing caches on a shared production service. Compare first-pass performance with later passes and with the expected working-set size. If only warm runs meet the target, staging time and cache capacity become part of the operational requirement.
Estimate demand from the target processing rate and the average bytes fetched per sample. For example, an illustrative workload processing 2,000 samples per second and fetching 1 MB per sample requires about 2 GB/s of input reads before accounting for extra reads or retries. Use the stored bytes actually fetched, which may differ substantially from the size of decoded tensors.
Compare that estimate with sustained client throughput during the job. Then account for concurrent jobs and uneven demand rather than treating a single workload’s average as the platform requirement. This calculation is a starting point for measurement, not a universal bandwidth target per GPU.
Enough average bandwidth does not guarantee that every batch arrives on time. Look at the distribution of read latency, including the slower requests, and whether those requests coincide with batch stalls. A p99 value describes the latency at or below which 99% of measured requests complete; it helps expose delays that an average can hide.
Small-object workloads can spend substantial time on requests and per-object processing while transferring relatively few bytes. A storage platform may therefore show modest bandwidth even when the access pattern is limiting the job. Measure request rate, object size and outstanding reads together.
Compare performance when the job runs alone and alongside representative activity. Training reads may share resources with ingestion, checkpoint writes, replication or other users. A configuration that meets the target in isolation may miss it during normal operation.
Collect client and server measurements over the same time window. Rising server-side latency and queueing can support a storage-service diagnosis, while a constrained host network link points elsewhere. Errors, retries and client connection limits also belong in the investigation because they can delay reads without exhausting backend bandwidth.
Consider an illustrative training job with these results. These numbers demonstrate the method; they are not Scality benchmark results.
| Test | Processing rate | Interpretation |
|---|---|---|
| Raw inputs from shared storage | 800 samples/s | Baseline with the full production input path |
| Same raw inputs from local storage | 1,600 samples/s | Shared read path is contributing to the slowdown |
| Prepared inputs with normal fetching and preprocessing removed | 1,800 samples/s | Additional input-pipeline overhead remains |
The local-storage result doubles throughput, making the shared path the first place to investigate. However, the comparison does not distinguish network congestion from request serialization or slow storage responses. That requires the aligned measurements collected during the runs.
Suppose client traces then show that reads are issued almost serially and storage-side latency remains low. Increasing bounded read concurrency would be a more targeted experiment than replacing drives. If the traces instead show rising storage queues and latency as concurrency grows, the evidence supports investigating backend contention or capacity to serve the workload.
The prepared-input result also sets expectations. Even after improving the shared path, preprocessing may leave a remaining gap. Treat each result as evidence about the next constraint rather than a promise that one change will reproduce the fastest diagnostic run.
When per-object overhead dominates, packaging records into larger shards can reduce the number of individual reads. Test the trade-off: larger shards affect shuffle behavior, parallelism and how much unnecessary data must be fetched. The right layout depends on how the application samples and reuses the dataset.
When CPU preparation dominates, consider reusing preprocessed data, adjusting worker resources or simplifying repeated transformations. When the network limits reads, investigate the specific constrained link or shared route. Storage changes should follow evidence of a storage constraint, such as sustained throughput limits or request latency under representative load.
A performance tier can help when the active dataset needs faster access than the durable capacity tier economically provides. Scality’s validated architecture with WEKA pairs NeuralMesh for active AI and HPC data with Scality RING as the scalable object tier. That makes data movement between tiers part of the performance design, including how quickly a new working set becomes available.
For that architecture, test both an already active dataset and one that must be retrieved from the object tier. Measure time to the first useful batch alongside sustained training throughput. A fast warm run alone does not show whether the system can meet the schedule when teams switch datasets or start several jobs together.
Repeat the original workload after the change, with the same dataset, configuration and comparable cache state. Compare processing rate, exposed batch waits, slow read requests and total completion time. Include the concurrent activity that originally caused trouble and the checkpoint behavior expected in production.
The investigation is complete when the evidence explains the delay and the targeted change reduces it under realistic conditions. Keep the test configuration and results with the workload so future scaling decisions start from a measured requirement. GPU utilization is a useful signal, but the outcome that matters is getting more useful processing from the infrastructure already deployed.