AI Infrastructure
AI storage networking: Where bandwidth gets lost
A storage system rated at 200 GB/s and a cluster that can consume it will still frequently meet somewhere around 40. The shortfall is almost never in the ...
Scality Blog
Practical insight on exabyte-scale object storage, AI data infrastructure, and cyber resilience — from the team at Scality.
AI Infrastructure
Vector database backups: What else must you protect?
Vector databases are often treated as derived data, on the reasoning that if the index is lost it can simply be rebuilt from the source documents. That ...
AI Infrastructure
AI data retention: What to keep after model training
When a model goes into production, the environment that produced it does not clean itself up. The training data stays where it was, the checkpoints accumulate, ...
AI Infrastructure
AI dataset integrity: Detecting missing or changed data
A training job that quietly reads 4,000 fewer samples than it did last month will still finish, still produce a model, and still report a plausible accuracy ...
AI Infrastructure
AI storage benchmarks: What to test before buying
Storage vendors publish impressive throughput figures for AI workloads, and those figures are usually accurate for the configuration that produced them. The ...
AI Infrastructure
Multimodal AI storage: Managing images, audio and video
A storage design that works well for a dataset of photographs can behave very differently once audio and video are added to it. The three media types differ by ...
AI Infrastructure
AI checkpoint storage: How to size write throughput
Most storage sizing for AI training starts with the dataset. Checkpointing is what breaks that plan. A training job pauses at intervals to write out the full ...
AI Infrastructure
Training data versioning: How to reproduce an AI run
A model that performed well in March cannot be rebuilt in September unless the data it learned from can be reconstructed exactly. Code and hyperparameters are ...
AI Infrastructure
Small-file performance: Why AI data loaders slow down
An AI training job can wait for data while the storage network has plenty of unused bandwidth. This happens when the job needs thousands of small files and ...
Object storage
Erasure coding vs. replication: Capacity and recovery
Erasure coding and replication both help storage survive hardware failures, but they consume capacity differently. Replication keeps complete copies of data. ...