An AI training job can wait for data while the storage network has plenty of unused bandwidth. This happens when the job needs thousands of small files and spends more time requesting, locating and preparing them than transferring their contents. The storage system may move large files quickly, yet struggle to deliver enough individual training samples each second.
For infrastructure teams, that distinction changes the investigation. Adding network bandwidth will not necessarily help a workload that is waiting for file lookups. Buying faster storage will not fix a loader that spends most of its time decoding images. Before changing the infrastructure, establish which part of getting a sample into the model is taking the time.
Why small files create a different storage workload
Consider a manufacturing team training a model to identify defects in product photographs. Its dataset contains millions of separate images, each with a label describing what the image shows. Keeping those images as individual files makes them straightforward to collect, review and correct. Training introduces a different requirement: retrieve enough of them, in the required order, to keep the model working continuously.
Reading a file involves more than moving its bytes. A file-based system must resolve the path and handle the operations needed to open and read it. An object-based system must process a request for the object's key and return the data. Caches and implementation details change the work involved, but each access still has overhead.
With a large file, that overhead can be spread across a long transfer. With a small image, the transfer may finish quickly, leaving the request and lookup work as a substantial share of the total access time. Repeating that process across millions of samples makes the overhead consequential. A separate label file or an unnecessary existence check can add further operations for each sample.
There is no universal size below which a file becomes a performance problem. The useful question is whether the time spent arranging an access is large relative to the time spent transferring its contents. File size, access latency, caching and the number of simultaneous requests all influence that relationship.
A simple example: modest bandwidth, many requests
Suppose a training workload needs 5,000 images per second, with an average stored size of 100 KB. Using decimal units, that is about 500 MB of image data per second. If every image requires a separate read, the system must also complete 5,000 image reads per second. Labels and other supporting data may increase the request count.
Now assume, for illustration, that retrieving one image takes an average of two milliseconds. A reader that waits for each retrieval to finish before starting another can complete only about 500 reads per second. At this file size, that reader would move roughly 50 MB per second, even if the network could carry much more. These are illustrative calculations, not measured Scality performance figures.
Parallel readers can overlap those waits and raise throughput. However, the storage service, client and network must sustain the combined request rate, and the application must have enough work in progress. This is why a large-file bandwidth result alone cannot establish whether storage will meet a small-file training requirement.
What the data loader does before the model sees a batch
A data loader retrieves training samples and prepares them for the model. For the defect-detection example, it may select the next images, read their labels, decode the image files, resize them and assemble them into a batch. That batch then has to reach the accelerator's memory. Reading the stored bytes is only one stage.
An image that occupies little space on storage can expand substantially when decoded. Resizing, cropping and other transformations also consume processor time and memory. As a result, a slow loader does not automatically mean slow storage. The same visible symptom, a model waiting for its next batch, can originate in several places.
This distinction matters when infrastructure and machine-learning teams investigate together. The storage team may see low bandwidth and conclude that storage is lightly loaded. The model team may see idle accelerators and conclude that storage is too slow. Both observations can be accurate without identifying the cause: the missing measurement is where time is spent between selecting a sample and delivering a usable batch.
Why performance can deteriorate as training scales
A small experiment often reads from a limited dataset on one machine. Recently accessed files may remain in memory, and a few readers may generate little contention. A larger job can lose both advantages. More training processes read more samples, while the active dataset becomes too large for the available caches.
Randomized access adds another complication. Training commonly changes sample order to avoid repeatedly presenting the same sequence to the model. When those samples are scattered across separate files, reading ahead becomes less useful and cache reuse can fall. The storage workload becomes a large number of independent accesses instead of a predictable stream.
Adding accelerators also increases the rate at which batches must arrive. A loader that comfortably supplies one training process may be unable to supply several at once. In synchronized distributed training, one process delayed by a slow batch can hold up others at a synchronization point. Occasional long waits can therefore matter even when average read latency appears acceptable.
Look at the conditions under which performance changes. A fast second pass suggests caching is helping. A sharp decline when more readers start suggests a shared resource is becoming constrained. A pause before any training begins points toward dataset discovery, initialization or staging, which requires a different fix from slow reads during training.
How to find the delay without replacing the storage first
Start with a representative portion of the actual dataset. Preserve its mix of file sizes, labels and transformations, rather than choosing only convenient samples. Record how long it takes to produce usable batches, alongside storage latency, request rate, network throughput and host CPU use. Separate startup time from sustained operation so one does not hide the other.
The most useful tests change one part of the workload at a time. They should help both teams answer a specific question about the delay, rather than produce another headline throughput number.
| Test | What to keep consistent | What the result helps establish |
|---|---|---|
| Compare shared storage with a local copy | Samples, transformations and batch size | Whether changing the storage access path materially improves batch delivery |
| Reduce transformations temporarily | Storage location and sample selection | Whether decoding or preprocessing is consuming much of the time |
| Increase readers gradually | Dataset and batch preparation | Whether overlapping reads helps, and where contention begins |
| Compare first and repeated passes | Reader count and access pattern | How much performance depends on cached data |
| Compare individual files with packed samples | Sample contents and equivalent processing | Whether reducing separate accesses improves delivery |
A local-copy test needs careful interpretation. If it is faster, the shared access path deserves investigation, but the result does not identify the storage media as the cause. Network latency, client behavior, metadata operations and differences in cache state may all contribute. Record those conditions before drawing a purchasing conclusion.
Similarly, a test with preprocessing disabled is diagnostic rather than representative of production training. It can reveal that image decoding is expensive, but the real job still needs usable inputs. Restore the normal processing after isolating its contribution, then judge any proposed change by the complete workload.
Match the fix to the work causing the delay
Reduce repeated discovery and unnecessary accesses
Some jobs spend considerable time building a list of samples before they start. Repeated directory scans or object listings can become expensive as the dataset grows, especially when each training process repeats the same discovery work. A versioned manifest, a file listing the samples and their associated labels, can let jobs open a known dataset directly. It must stay aligned with the dataset version so performance improvements do not introduce missing or incorrect samples.
Next, inspect how much work happens for each sample. Does the loader check whether a file exists immediately before opening it? Does it fetch a tiny label separately when the training format could keep the pair together? Removing redundant operations can reduce request pressure without changing the underlying storage platform.
Pack samples when training can read them together
Packing combines multiple samples into larger files, often called shards. A loader can then read a shard sequentially and extract samples from it, spreading access overhead across more useful data. For a workload that repeatedly scans a large image collection, this can be more efficient than retrieving every image independently. The benefit depends on the loader actually using the packed layout efficiently.
Simply putting everything into one enormous archive is not enough. A loader that repeatedly seeks to unrelated locations may retain much of the original access problem, while a compressed archive may require extra decompression to reach a sample. Very large shards can also complicate distribution across workers and make small updates expensive. Very small shards may preserve too much of the request overhead.
Sample order needs attention as well. If images are packed by product type or capture date, reading each shard in sequence could present strongly grouped examples to the model. The training pipeline needs an appropriate shuffling strategy, and the team should check that the change preserves the intended sampling behavior. Faster delivery is useful only if the training data remains correct.
Overlap reads without overwhelming the system
When readers spend much of their time waiting for accesses to complete, additional readers can help keep work moving. Increase concurrency in steps and watch whether usable samples per second improve. Stop treating a higher reader count as progress once it produces longer waits, more memory pressure or a flat throughput curve. At that point, the extra work is competing for a constrained resource.
Prefetching prepares upcoming batches while the model processes the current one. It can absorb short interruptions when the loader is fast enough on average. It cannot sustain a job whose loader consistently produces batches more slowly than the model consumes them. A deeper queue delays that shortage and uses more memory; it does not remove the underlying deficit.
Cache the data the job will actually reuse
Local flash or a shared performance tier can reduce repeated access to a larger repository. This is particularly useful when several training passes reuse the same working dataset. The capacity decision should follow the active data and competing jobs, not only the total size of the organization's archive. A cache that constantly evicts samples just before they are needed provides little benefit.
Include cache population in the comparison. Staging data for an hour to save a few minutes on a single run may be a poor trade, while the same staging cost may be worthwhile across many runs. Measure the time from starting the job to finishing useful work, as well as the faster training phase after data is already nearby.
Where Scality RING fits in the storage design
An organization can retain its full AI dataset on a capacity tier while using a performance tier for active training data. Scality's validated integration with WEKA illustrates this approach: active file data and file metadata reside on WEKA's flash tier, while colder file payloads can move to Scality RING. Keeping metadata on the performance tier lets file browsing and lookup stay there even when the underlying payload has been tiered.
That separation matters for small-file workloads because locating a file and retrieving its contents are distinct operations. A design needs to account for both, including what happens when training requests data that is no longer on flash. Fast access to a warm working set does not establish how quickly a new dataset will become ready. Test initial access and recall behavior alongside repeated training passes.
The practical decision is how much of the dataset needs performance-tier access, how often that working set changes and whether the loader can read the stored format efficiently. A repository may hold years of images while an individual project uses only a subset. Conversely, a job that continually traverses most of the collection may need a much larger active tier or an efficient streaming layout.
Judge the result by reliable batch delivery
A useful acceptance test follows the training job through startup, its first pass and subsequent passes, with the expected number of concurrent readers. Record the dataset version, access pattern, preprocessing and cache conditions so the result is repeatable. Include slow batches as well as average throughput, because intermittent delays can disrupt an otherwise fast run.
The first improvement should follow the evidence. Reduce lookup overhead when repeated discovery is the problem, pack samples when separate accesses dominate, and address preprocessing when the CPU cannot prepare data quickly enough. Small-file performance improves when the storage layout and loader match the way training consumes samples. The result to buy and measure is a dependable supply of usable batches.