Small object storage: Where overhead comes from

Object storage is built around large objects. Its protection schemes, its throughput figures and its cost per terabyte all look best when files are measured in megabytes or gigabytes. Many real workloads are not shaped that way. Log archives, sensor readings, thumbnails, email stores, document repositories and machine learning datasets can hold billions of objects of a few kilobytes each, and for those workloads the cost of storing each object starts to matter as much as the bytes inside it.

Small object storage is not a problem to avoid so much as a workload to plan for. The overhead comes from a handful of predictable places, and once they are understood, both the capacity estimate and the performance expectations become far more accurate. The effect of small files on AI training pipelines specifically is covered in the post on why AI data loaders slow down. This post looks at the storage side.

Where the overhead comes from

Every object, however small, incurs costs that do not shrink with its size:

• A metadata record. Key, timestamps, checksum, version, permissions and data location, stored and protected separately from the data.
• A minimum allocation. File systems and storage layers allocate space in blocks or chunks. An object smaller than the allocation unit still consumes a full unit.
• Protection overhead. Replication or erasure coding applies per object or per chunk, with fixed costs that dominate when the payload is tiny.
• Per-request work. Authentication, metadata lookup and network round trips happen for every request, whether it moves 4 KB or 4 GB.

For a 1 GB object these costs are invisible. For a 4 KB object they can exceed the size of the data itself.

Capacity overhead in practice

The table below illustrates how fixed costs change effective overhead as average object size falls. The figures are illustrative, assuming a metadata record of around 1 KB per object, a 4 KB minimum allocation and triple replication for small objects. Actual values depend on the system and its configuration.

The practical lesson is that a capacity estimate built from total bytes and the headline protection ratio will understate raw capacity for small-object workloads. The estimate should be built from the size distribution, as described in the post on raw versus usable capacity.

Why erasure coding and small objects do not mix well

Erasure coding splits data into fragments and adds parity, spreading the pieces across many devices. For a large object that saves a great deal of capacity compared with replication. For a small one it multiplies the number of writes and metadata entries without saving much space, a point covered in more detail in the post on erasure coding layouts. Many systems therefore replicate objects below a size threshold and erasure code above it, or pack small objects into larger containers before coding them. Either way, the blended overhead depends on where the threshold sits relative to the workload's size distribution.

Request rates matter more than throughput

A cluster rated at many gigabytes per second of throughput may still struggle with a small-object workload, because the limiting factor is operations per second, not bandwidth. Writing ten million 10 KB objects moves only 100 GB, but it requires ten million metadata writes, ten million protection operations and ten million client round trips.

Performance planning for small objects should therefore start from requests per second at the expected object size, measured for reads, writes and listings separately. Benchmarks run with large objects say little about how the same cluster behaves under a small-object load, a gap discussed in the post on why throughput is not enough.

Reducing small-object overhead

Some overhead is unavoidable, but several design choices reduce it:

• Batch at the application. Where the application allows it, grouping many small records into one larger object, such as a daily archive or a columnar file, cuts object count dramatically.
• Use tiering by size. Placing small objects under a replication policy and large ones under erasure coding matches protection to object size.
• Design keys for listing. Prefixes that reflect how data is read keep listings and lifecycle scans narrow.
• Expire aggressively. Small objects often have short useful lives. Lifecycle rules keep object counts from growing without bound.
• Size metadata resources separately. Memory and fast storage for metadata should follow object count, not capacity, a theme expanded in the post on object storage metadata.

Where Scality RING fits

Scality RING supports both replication and erasure coding, selectable per policy, so protection can be matched to object size rather than applied uniformly. It runs on standard servers chosen by the customer, which allows the ratio of fast resources for metadata to bulk capacity to be set for the object counts a workload actually produces.

RING distributes data across the nodes in the system, spreading request load across many devices, and it emits operational metrics and logs the operator keeps locally, so request rates and per-bucket growth can be watched alongside capacity.

Questions to answer before storing billions of small objects

• What is the object size distribution, and what share of objects falls below the replication threshold?
• What raw capacity does the workload need once per-object overhead is included?
• How many requests per second will reads, writes and listings generate?
• Can the application batch records into larger objects without losing what it needs?
• Which lifecycle rules will keep the object count in check?

Planned around object count and request rate rather than bytes alone, small-object workloads run predictably on object storage. Planned around bytes alone, they are the most common source of capacity and performance surprises.