Storage vendors publish impressive throughput figures for AI workloads, and those figures are usually accurate for the configuration that produced them. The difficulty is that the configuration rarely resembles the buyer's environment. A number produced by large sequential reads from a tuned client fleet says very little about how a system will behave when a real data loader asks for millions of small samples in random order while a checkpoint is being written.
Benchmarking before a purchase is worth the effort, but only if the tests are chosen to answer questions the organization actually has. The goal is not to produce a number. It is to establish whether the system will keep the accelerators busy under the conditions the team will run it in, and to find out where it stops doing so.
Three differences account for most of the gap. The first is the object size distribution. A benchmark using large files measures bandwidth; a workload using small samples measures request handling. The second is concurrency. Results from a small number of high performance clients do not tell you how the system behaves with hundreds of workers issuing requests at once. The third is mixed activity. Benchmarks usually run one operation at a time, while production runs training reads, checkpoint writes and ingestion of new data simultaneously.
There is also a caching effect that flatters repeated tests. A benchmark that runs the same dataset several times will often show results from cache rather than from storage. That number is real, but it describes the second epoch of training rather than the first, and it says nothing about how a newly arrived dataset will perform.
The useful mental model is that a benchmark answers a narrow question. Deciding which narrow questions matter is the part that requires the buyer's judgement.
Before running anything, write down what the environment does. How many samples per second does the pipeline need to keep the accelerators busy, and what is the average sample size? How large is a checkpoint and how long may it stall the job? How much new data arrives per day and during which hours? How many independent teams share the system?
Those four answers determine the tests. A team whose pipeline needs 40,000 small samples per second has a request rate problem and should test that directly. A team training on video has a bandwidth and range request problem. A team with a shared cluster has a contention problem that only appears with concurrent tenants. Running a generic suite without this step usually produces numbers that nobody can act on.
If the pipeline already exists, the fastest calibration is to measure it on current storage first. That establishes the baseline the new system has to beat and often reveals that the current constraint is somewhere other than storage, which changes the procurement question entirely.
The tests below cover the behaviors that most often determine whether an AI environment performs as expected. Each one should be run at the concurrency the production environment will use, not at the concurrency that produces the best figure.
| Test | What it establishes | Condition that makes it honest |
|---|---|---|
| Small object read rate | Samples per second the system can deliver | Random order, cold cache, real object size distribution |
| Sustained sequential read | Bandwidth for large media and packed formats | Dataset larger than any cache in the path |
| Concurrent write burst | Checkpoint stall duration | All ranks writing at once, while reads continue |
| First pass versus repeat pass | How much of the result depends on caching | Compare epoch one against epoch three |
| Mixed read and write | Behavior under realistic simultaneous load | Ingestion and training running together |
| Metadata operations | Listing, stat and lookup cost at scale | Buckets with the object counts you will really have |
| Degraded mode | Performance while a node or drive is rebuilding | Fail a component deliberately during the run |
| Restore after failure | Time to reload checkpoint state and resume | Cold start, no warm cache |
The degraded mode row is the one most often skipped and the one most likely to matter. Hardware fails during long training campaigns, and a system that delivers full performance only while completely healthy will disappoint at exactly the wrong time. Ask to run this test, and treat reluctance as information.
Several environmental factors can dominate the result and should be held constant or documented. Network path is the most common: a storage system capable of far more than the client network can carry will produce a benchmark that measures the network. Client count and client tuning matter similarly, and results from a vendor's tuned client stack may not transfer to the organization's standard build.
Dataset size relative to cache is the factor that quietly invalidates the most benchmarks. If the test dataset fits in memory or on a local cache tier anywhere in the path, the result describes the cache. Use a dataset several times larger than the combined cache, or the number will not survive contact with production.
Finally, note whether the test ran on an empty system. Object storage behavior can change as capacity fills and as object counts grow, so a result from a system at 5 percent utilization is a weak predictor of behavior at 70 percent. Ask what happens at the fill level you expect to operate at.
Two additional exercises tell buyers a great deal. The first is a full ingestion of a realistic dataset, timed end to end, including whatever validation the team performs. This reveals how long it will take to make new data usable, which is a recurring operational cost rather than a one-time setup.
The second is an expansion. Add capacity to the system during a running workload and observe what happens to performance while data redistributes. Growth is certain in an AI environment, and the cost of growing is part of the total cost of the platform. A system that requires a quiet period to expand imposes a scheduling constraint that should be known before purchase, not after.
It is also worth testing the operations a team performs under pressure: deleting a large prefix, recalling data from a colder tier, and restoring after a failure. These rarely appear in performance material and they shape how the system feels to operate.
Scality RING is built to scale aggregate throughput by adding nodes, so the figures that matter in an evaluation are the ones taken under parallel load from many clients rather than from a single stream. For AI environments the relevant tests are usually aggregate small object read rate at production concurrency, sustained write during a simulated checkpoint, and behavior while a node is rebuilding, since those correspond to the three moments when a training cluster is most exposed.
Because RING supports very large namespaces on a single system, the metadata test should use object counts in the range the organization expects to reach rather than a convenient sample, and the expansion test should be included, since adding nodes is the normal growth path rather than an exceptional event. A validated design conversation is more useful than a peak number, and it should cover what the system does during recovery as well as what it does when healthy.
A benchmark that confirms what everyone already believed has not earned its cost. The valuable outcome is a number that changes something: a different network design, a different tier boundary, a revised checkpoint interval, or a clear answer that the storage system was never the constraint.
Keep the test definitions and the results, and rerun them after the system is in production. Environments drift, datasets grow and pipelines change, and the same tests run a year later will show whether the design still holds. That repeatability is worth more than any single figure produced during the buying process, and it connects directly to how the environment should be sized as it grows.