Most storage sizing for AI training starts with the dataset. Checkpointing is what breaks that plan. A training job pauses at intervals to write out the full state of the model so the run can be resumed after a failure, and that write is large, sudden, and issued by every worker at once. A cluster that reads training data at a comfortable rate can still stall for minutes on each checkpoint.
The sizing question is therefore separate from capacity. It is a write throughput question, driven by the size of the model state and by how long the team is willing to have expensive accelerators sitting idle while that state lands on storage. Working it out takes arithmetic rather than benchmarking, and the arithmetic is worth doing before hardware is chosen.
Reading training data is steady and can be overlapped with computation. Data loaders prefetch, cache and run ahead, so the storage system sees a continuous stream shaped by the pipeline rather than by the model. Checkpointing has the opposite profile. Nothing happens for several minutes, then the entire job writes simultaneously, then nothing happens again.
That burst is the number that matters. Average throughput across an hour can look modest while each individual checkpoint saturates the write path and stops training for the duration. Infrastructure teams investigating why accelerators sit idle often find the answer in checkpoint windows rather than in the read path, because idle time during a checkpoint does not look like a storage problem in aggregate utilization graphs.
The write is also concentrated in a small number of very large objects rather than spread across many small ones, which means it stresses sustained bandwidth and the parallelism of the write path rather than metadata or request rates.
Checkpoint size is set by the model and the optimizer, not by the training dataset. For a model trained with a common adaptive optimizer in mixed precision, the state usually includes the parameters, a higher precision master copy of the parameters, and two optimizer moments per parameter. In practice this lands somewhere around 12 to 16 bytes for every parameter in the model, and the exact figure depends on the optimizer and precision the team has chosen.
That produces numbers worth knowing before a procurement conversation. A 7 billion parameter model produces roughly 100 GB of state. A 70 billion parameter model produces roughly 1 TB. A model an order of magnitude larger produces state measured in tens of terabytes. None of these figures depend on whether the dataset is 50 TB or 5 PB.
Two things modify the raw number. Sharded checkpointing splits the state across workers so each one writes a fraction of the total, which turns a single large stream into many parallel streams of the same aggregate size. And some teams save only the parameters for inference use while keeping full optimizer state only for the most recent checkpoint, which reduces what has to be retained without reducing the size of the write itself.
The calculation is short. Take the checkpoint size, decide the longest acceptable pause, and divide. A 1 TB checkpoint that must complete in 60 seconds needs roughly 17 GB/s of sustained write throughput to the storage system. The same checkpoint with a 5 minute tolerance needs closer to 3.5 GB/s.
The acceptable pause is a cost decision rather than a technical one. If a cluster costs a known amount per hour and checkpoints every 30 minutes, a 3 minute checkpoint consumes 10 percent of the run. Reducing that to 30 seconds recovers most of it, and whether that is worth the storage investment depends on the size of the cluster and how long the training campaign runs.
| Input | How to establish it | Common mistake |
|---|---|---|
| Checkpoint size | Parameter count multiplied by bytes of state per parameter | Sizing from dataset volume instead of model state |
| Acceptable stall | Accelerator cost per hour against percentage of run time lost | Treating any pause as acceptable because training is long |
| Checkpoint interval | Expected time between failures, balanced against work lost | Picking a round number with no failure data behind it |
| Concurrent writers | Number of ranks writing shards simultaneously | Testing with one stream and assuming it scales |
| Retained checkpoints | How far back a run may need to be rewound | Keeping every checkpoint because nobody set a policy |
| Restore throughput | Read bandwidth needed to reload state after a failure | Sizing writes only and discovering restore is slower |
Restore deserves particular attention because it is the operation that happens under pressure. A checkpoint that writes in 60 seconds but takes 20 minutes to read back turns every hardware failure into a much longer interruption than the checkpoint schedule suggests.
Checkpoint frequency trades two costs against each other. Checkpointing often means more time spent writing and less work lost when something fails. Checkpointing rarely means the opposite. The balance point depends on how frequently the cluster actually fails, which most organizations can measure from their own job history rather than estimate.
Large clusters fail more often than teams expect, because the failure rate scales with the number of components. A job spread across hundreds of accelerators for several weeks will be interrupted, and the interval should be chosen so that a single interruption costs an acceptable amount of recomputation. Half an hour of lost work is usually tolerable. Half a day rarely is.
Asynchronous checkpointing changes the arithmetic significantly. If the framework can copy model state to host memory quickly and then stream it to storage in the background, training resumes after the copy rather than after the write completes. The storage system still has to absorb the same volume, but it can do so over a longer window, which lowers the peak throughput requirement. Teams should confirm their framework supports this before sizing for a worst case that does not apply.
Checkpoints accumulate quickly. A 1 TB checkpoint written every 30 minutes produces 48 TB a day if nothing is removed. Very few teams need that history, but the default behavior of most training scripts is to keep writing without cleaning up, and the capacity shows up on a storage review months later as unexplained growth.
A workable policy keeps the last few checkpoints for restart purposes, keeps periodic milestones for the length of the training campaign, and keeps the final state indefinitely alongside the record of which data produced it. That last pairing matters: a retained checkpoint with no matching record of the dataset version cannot be explained later, and a dataset version with no checkpoint cannot be compared against the model it produced.
Because checkpoint retention is predictable, it is a good candidate for automated lifecycle rules rather than manual cleanup. Expiring intermediate checkpoints on a schedule keeps the capacity bounded without anyone having to decide, each week, which files are still needed.
Scality RING scales write throughput by adding nodes, so the aggregate bandwidth available to a checkpoint grows with the size of the system rather than being fixed by a controller pair. That matters for sharded checkpointing, where many workers write concurrently and the useful measure is aggregate throughput under parallel load rather than the speed of a single stream.
A common design keeps the most recent checkpoints on a performance tier close to the cluster for fast restart, and writes milestone checkpoints and final model states to RING for retention, where they sit alongside the training data and the dataset manifests. Lifecycle rules can then expire intermediate checkpoints automatically, and the checkpoints that need to be defensible can be held immutable for a defined period.
The sizing conversation with any storage system should be about sustained aggregate write throughput with the real number of concurrent writers, not peak figures from a single client. Ask for the write rate the system sustains while it is also serving training reads, because that is the condition a checkpoint actually occurs in.
The measurement that matters is how long training is paused, end to end, for a real checkpoint on the real cluster. Instrument the job to record it, then compare it against what the arithmetic predicted. A gap usually points at something specific: too few parallel writers, a network path that is narrower than the storage, or a framework that serializes the write when it could overlap it.
Repeat the measurement for restore as well, from a cold start, with nothing cached. That number determines how long a failure costs, and it is the one most likely to be wrong in a design that was sized on capacity alone. A checkpoint strategy is working when a failed job resumes in minutes and the storage system was never the reason the accelerators were idle.