Accelerator hours are measured carefully because they are billed carefully. Storage usually appears as a single line in an infrastructure budget with no connection to any particular model, which makes it impossible to say whether a training campaign was expensive because of the compute, the data volume, or the time the cluster spent waiting. Building a per-run figure is not difficult, and it changes which optimizations get funded.
The exercise is worth doing at the point a programme moves from experiments to repeated production training, because that is when the same costs start recurring monthly and when small structural choices compound.
Storage cost for a run has four components, and only the first is usually counted. Capacity is the cost of holding the dataset, the checkpoints and the derived copies for the duration they exist. Throughput is the cost of the infrastructure sized to deliver data fast enough, which is often a larger driver than raw capacity in AI environments. Retention is the cost of everything the run leaves behind afterwards. And idle compute is the cost of accelerators waiting on storage, which is usually the largest of the four and almost never attributed to storage at all.
That last component is what makes the exercise worth doing. If a cluster costs a known amount per hour and spends 12 percent of a campaign waiting on data, that percentage is a storage cost regardless of which budget line it lands in. Presenting it as such is often the only way a storage investment gets compared fairly against a compute investment.
A workable model starts from a cost per terabyte per month for each tier, a cost per accelerator hour, and the measured behavior of a real run. The arithmetic is straightforward once those are in place.
| Component | How to calculate it | Common omission |
|---|---|---|
| Dataset capacity | Size multiplied by tier cost multiplied by months held | Counting only one copy when three exist |
| Derived and staged copies | Processed and packed variants, plus local staging | Treating derivatives as free because they are temporary |
| Checkpoint capacity | Checkpoint size multiplied by count retained | Retaining every checkpoint with no expiry |
| Performance tier premium | Difference between fast and capacity tier pricing | Keeping cold data on the fast tier by default |
| Idle accelerator time | Stall hours multiplied by cluster cost per hour | Not measuring stalls at all |
| Post-run retention | What remains, multiplied by how long it is kept | Assuming the run ends when the model ships |
The derived copies row is worth attention because it is the one that surprises people. A dataset that exists as source, as a processed variant, as a packed training format and as a staged local copy occupies several times its nominal size, and each copy has its own retention behavior. Counting them individually often reveals that the storage bill is mostly duplicates.
In most environments, three patterns account for the majority of avoidable spend. The first is retention with no expiry: intermediate checkpoints and old processed datasets that nobody decided to keep and nobody decided to remove. The second is tier misplacement: completed campaign data sitting on performance storage because moving it was never scheduled. The third is stall time, which costs more than both but is invisible without instrumentation.
These are worth ranking before any of them is addressed, because the effort differs sharply. Expiring checkpoints is a lifecycle rule and takes an afternoon. Moving completed datasets to a capacity tier is a scheduled job. Reducing stall time may require a network change or a pipeline rewrite, and it is only worth starting once the measurement shows the stall is large enough to justify it.
The instrumentation needed for the third is modest: record the time each run spends waiting for data and waiting on checkpoint writes. Once that exists, the cost of a storage constraint can be stated in the same units as the cost of fixing it, which is the conversation that idle accelerator investigations usually lack.
Both views are useful and they answer different questions. Cost per run supports decisions about a specific model: whether retraining monthly is justified, whether a larger dataset earns its keep, whether a sweep of 40 configurations is affordable. Cost per month supports capacity planning and budgeting.
Converting between them requires an allocation rule for shared capacity, and the simplest defensible rule is to attribute dataset capacity to the projects that read it, split by usage, and to attribute checkpoints and derived copies wholly to the run that created them. Precision is not the point. Consistency is, because the value of the figure is in comparing runs against each other over time.
It is also worth separating the recurring cost of holding data from the one-off cost of a run. A campaign that trains for two weeks may leave behind data that costs more over the following year than the training itself did, and that is a retention decision rather than a training decision.
The structural lever in most AI storage bills is the split between a performance tier and a capacity tier. Scality RING is typically the capacity tier, holding full datasets, completed campaign data, milestone checkpoints and final model states, while faster storage serves the active working set. The cost saving comes from keeping the expensive tier sized for what is actively being read rather than for everything that exists.
Because RING scales by adding nodes on standard servers, capacity growth is incremental rather than stepwise, which makes the per-terabyte figure in the model stable enough to plan against. Lifecycle rules handle the expiry of derived and intermediate data automatically, which addresses the retention component without depending on anyone remembering, and consolidating datasets on one namespace removes the duplicate copies that separate systems tend to accumulate. Sizing that tier accurately is the same exercise as establishing how much capacity the environment needs, with the retention tail included.
A cost model earns its keep when it settles an argument. Useful outcomes look like a decision to expire checkpoints after seven days, to move completed datasets on a schedule, to buy network capacity instead of storage capacity, or to stop retaining processed variants that can be regenerated in an hour.
Rebuild the figure once a quarter rather than once. Datasets grow, retention accumulates and pipelines change, and a model that is refreshed regularly will show which of those is moving the total. A model built once and referenced for a year describes an environment that no longer exists, and the cost that matters most, the time a cluster spends waiting, is the one most likely to have changed.