AI storage sizing starts with everything your workloads need to retain: source datasets, prepared training data, saved dataset versions, checkpoints and model artifacts. Add expected growth and temporary working space, and you have the amount of data the system has to hold. The original dataset is only one part of that calculation.
A 200 TB source collection, for example, reaches roughly 455 TB of expected peak occupancy under the assumptions worked through below. Data protection and the free space the platform needs to operate sit on top of that figure. A different retention policy or preprocessing workflow could change the result substantially.
For infrastructure teams, the useful question is how much data will coexist at the busiest point in the planning period. That gives you a capacity requirement you can explain, test and update as the AI program grows.
What counts toward AI storage capacity?
An AI project can create several stored representations of the same information. A video collection might exist as original recordings, extracted frames, labeled samples and packaged training files. Some remain valuable for years; others only need to survive until a processing job finishes.
Build the inventory around actual stored bytes and retention periods. Record where each copy lives, who owns it and when it can be deleted. That prevents both undercounting hidden copies and charging the same shared dataset to every project that reads it.
Source data and prepared datasets
Source data is the retained collection of documents, images, audio, video or other records feeding your AI workflows. Use its stored size as the starting point. A file inventory is more useful here than the amount of data an application has read over its lifetime.
Prepared datasets need their own measurement. Tokenization, filtering, decompression, image conversion and packaging can make the output smaller or larger than the input. There is no universal preprocessing multiplier that works across these workflows.
Process a representative sample before committing to a capacity estimate. Measure what your pipeline actually retains, apply that ratio only to comparable data, and measure temporary files separately because they may disappear once preprocessing completes.
Dataset versions and experiment outputs
Reproducing a training run requires knowing which data it used. Depending on the tooling, preserving that state may involve a manifest referencing existing objects, additional changed objects or an entirely separate dataset copy. Three named dataset versions do not necessarily consume three times the original capacity.
Count the incremental bytes retained by your implementation. Include evaluation outputs, experiment logs and released model artifacts if they occupy the storage pool you are sizing. Treat promised compression or deduplication savings as unproven until a representative test demonstrates them.
Training checkpoints
A checkpoint used to resume training can contain more than model weights. It may also contain optimizer state and other information needed to continue the run. An inference model export therefore is not a reliable substitute for measuring the saved training checkpoint.
Estimate checkpoint capacity from the total size of each complete checkpoint, the number retained per run and the number of runs whose checkpoints coexist. For distributed training, include all files or shards belonging to the saved state. Avoid multiplying by the GPU count again if that total already includes every shard.
A practical AI storage sizing formula
Calculate the requirement for each storage pool separately. A durable dataset repository, a shared performance tier and local worker storage have different contents and retention rules. Combining them into one number too early makes it difficult to see where capacity is actually needed.
For one pool, use:
Expected peak occupancy = retained data + forecast growth + peak temporary data
Retained data includes every dataset, version, checkpoint and artifact that must remain available in that pool. Forecast growth is the additional retained volume expected by the end of the planning period, after planned deletions. Peak temporary data is the extra space in use while active operations overlap.
That total is the amount of data the system has to accommodate. It is not the capacity you buy. Every platform also needs room for data protection and an operating reserve, so that rebuilds, background tasks and failure handling have space to work. How much depends on the platform, the protection scheme and the failure domains in the design, so confirm those figures for the proposed configuration rather than applying a generic percentage.
Worked example: Sizing a 200 TB AI dataset
Consider an enterprise team building a shared repository for several training projects. It retains 200 TB of source data and has measured the outputs of its preparation pipeline. The team wants enough capacity for the next 12 months, with a defined allowance for temporary work.
| Capacity component | Occupied capacity |
|---|---|
| Source datasets | 200 TB |
| Prepared training datasets | 60 TB |
| Dataset versions | 40 TB |
| Training checkpoints | 10 TB |
| Model artifacts and experiment outputs | 10 TB |
| Current retained data | 320 TB |
| Growth over 12 months | 80 TB |
| Peak temporary working data | 55 TB |
| Expected peak occupancy | 455 TB |
The team starts with 200 TB of source data. Prepared datasets, retained versions, checkpoints and experiment outputs bring its current storage use to 320 TB. Allowing for a year of growth and temporary processing files raises expected peak occupancy to 455 TB.
That is the amount of data the system needs to accommodate. The final storage configuration also needs room for data protection and an operating reserve, which should be confirmed with the storage team.
The assumptions behind each line:
- Prepared training datasets. This pipeline was measured, and its retained output came to 30% of source volume. That ratio is specific to the workflow, not a general rule.
- Dataset versions. Incremental bytes retained beyond the current datasets, rather than full additional copies.
- Training checkpoints. Ten retained runs, five checkpoints each, roughly 0.2 TB per complete checkpoint.
- Growth. A planning assumption of 25% net growth across retained data over 12 months, after expected deletions.
- Peak temporary working data. The additional space in use while preparation and training jobs overlap.
- Units. All values are decimal TB, with no reduction assumed from compression or deduplication.
These are illustrative planning assumptions, not Scality benchmark results or a customer deployment. Your own pipeline should supply the equivalent figures.
Checkpoint retention deserves a conversation of its own, because it scales that line directly: keeping several times as many checkpoints per run multiplies the checkpoint figure by the same factor, without a single byte of new source data arriving. It is also one of the easiest settings to accept as a default and never revisit. Before approving hardware, ask whether the extra checkpoints are needed for recovery, evaluation or reproducibility, and who decides when they expire.
Usable capacity is not the same as raw capacity
Occupancy figures describe data. Storage platforms describe themselves in two other ways, and confusing them is a common source of procurement error.
Usable capacity is the space available for your data once the platform’s data protection and internal overheads are accounted for. Raw capacity is the total physical capacity of the installed drives. Raw is always the larger of the two, because protecting data means storing more than the data itself: whether a platform replicates or uses erasure coding, some proportion of the hardware carries redundancy rather than payload.
Comparisons go wrong when one proposal quotes raw capacity and another quotes usable. Ask each vendor which of the two their figure represents, and which overheads they have already subtracted, before comparing prices or drawing conclusions about density.
A separately maintained recovery copy is sized on its own terms. Its retention and protection policies may differ from the primary repository, so automatically doubling the primary figure can misstate the requirement. Data protection within a single storage system is not an independent recovery copy.
Keep units consistent throughout the proposal. Decimal TB and binary TiB describe different quantities, which can make a correctly provisioned system appear smaller when its management interface uses a different unit.
Size the performance tier around active work
The full retained collection does not automatically belong on the fastest storage tier. A repository may hold years of source material while current jobs use a much smaller subset. Measure that active working set and the time available to stage it before deciding how much flash capacity to provision.
Suppose three concurrent jobs each need a different 12 TB dataset staged locally. That creates a 36 TB requirement before allowing for checkpoints, preprocessing or replacement of cached files. If all three use the same dataset through a shared cache, the requirement may be lower; if each worker stores a full local copy, it may be considerably higher.
Placement matters as much as the total. Free space on one worker cannot necessarily satisfy a staging requirement on another. Check capacity per node as well as across the cluster, and account for overlapping jobs rather than only the largest individual job.
Scality RING provides distributed object storage with S3 access and scale-out capacity. In an AI storage design, that creates an option for the durable repository holding retained datasets and artifacts. Whether a workload should read directly from it or use a separate staging tier depends on the application’s access pattern and measured performance.
The sizing consequence is practical: estimate the retained collection and the active working set separately. That lets the design expand long-term capacity without assuming every additional terabyte needs identical placement or performance.
Enough capacity does not guarantee enough throughput
A system can have room for every dataset and still leave training jobs waiting for data. Capacity and throughput are separate questions, and a design that satisfies the capacity calculation can still miss the window a job has to stage its data in.
Test the operations that will overlap in production. Reading training data while another team prepares a dataset and a third writes checkpoints is a different workload from a single sequential transfer. Use the result to validate the storage and network design, rather than inferring performance from installed terabytes.
Plan growth around retention and purchase lead times
A flat annual growth percentage is useful for an initial estimate, but it can hide important changes. Adding a video project may introduce a much larger source collection, while changing checkpoint retention can increase occupancy without adding any new source data. Build a forecast that separates those drivers as the program becomes established.
For continuously ingested data with a fixed retention period, a useful starting estimate is retained ingest per day multiplied by retention days, before derived datasets and protection overhead. The estimate only holds if expired data is actually removed.
Set the expansion trigger early enough to cover procurement, deployment and validation. If installation takes several months, reaching the operating limit is too late to begin ordering capacity. Track net growth and the estimated date that limit will be reached.
What to measure before approving the design
A representative pilot should establish the relationship between source bytes, prepared data, saved checkpoints and temporary space. It should also confirm that retention cleanup works and that a saved checkpoint can be restored. A capacity forecast is much stronger when its largest multipliers come from your pipeline rather than from assumptions.
For the worked example, planning starts at approximately 455 TB of expected peak occupancy, with data protection, operating reserve and any separate recovery and performance tiers sized on top of it. Keep the workings visible so changes in retention, concurrency or dataset growth can be traced to their capacity impact.
The most useful AI storage plan is one the infrastructure and AI teams can maintain together. Each retained copy has an owner, each growth assumption has a basis, and each temporary allocation has an expiry condition. That makes the next expansion a forecasted decision instead of a surprise during a training run.