Storage fundamentals
Object metadata
Object metadata is the information an object storage system keeps about each object in addition to its contents: its name, size, creation time, checksum, storage settings and any descriptive attributes the application attached. Metadata is what allows an object to be found, validated and managed without reading the data itself.
What kinds of object metadata are there?
S3-compatible storage distinguishes three kinds.
| Kind | Set by | Examples | Can change after write? |
|---|---|---|---|
| System metadata | The storage system and request headers | Size, last-modified time, entity tag, content type, storage class, version ID, retention | Mostly no |
| User-defined metadata | The application, at upload | Project code, source system, patient or case reference | Only by rewriting the object |
| Object tags | The application, at any time | Classification, cost center, lifecycle group | Yes, without rewriting |
What is metadata used for?
Every operation other than reading the object's bytes is a metadata operation. Listing a bucket walks the index. Lifecycle rules select objects by age, prefix or tag. Access policies can match on tags. Replication and inventory compare metadata between locations. Applications use custom metadata to carry context with the data, so that an image, a log file or a backup block remains self-describing wherever it is copied. Unlike a file system, where such attributes are limited, object storage treats rich metadata as a core feature.
Why does metadata performance matter?
A store holding billions of objects has billions of index entries, and the index is consulted on every request. Workloads with many small objects, frequent listings or heavy deletion stress the metadata layer far more than the data layer. This is why object storage designs place metadata on fast media, typically flash, even when the data sits on hard drives, and why object count is a sizing input alongside capacity. Slow metadata shows up as high storage latency on small requests and long-running listings. Object storage metadata and small object storage go into the mechanics.
What are the governance considerations?
Metadata can itself be sensitive. Object names, tags and custom fields may contain personal or confidential information, and they are often exposed more widely than the data, in logs, inventories and monitoring systems. Rules about where data may be stored apply to its metadata too. Keys and tags should therefore avoid embedding personal data where possible, and telemetry sent outside the organization should be checked for metadata content. Metadata residency covers the location question.
Frequently asked questions
How much user metadata can an object carry?
The S3 specification limits user-defined metadata to about 2 KB per object, and tags to a small fixed number of key-value pairs.
Can metadata be searched?
The S3 API lists by key prefix. Searching by metadata or tag usually requires an inventory, an external index or a platform-specific search feature.
Is metadata encrypted?
Object contents are covered by encryption at rest. Whether metadata is encrypted depends on the platform, and object keys in particular are often visible in logs.
Related terms
- Object storage: the model built around objects and their metadata.
- S3 bucket: the namespace metadata is indexed within.
- Storage latency: where slow metadata is felt.
- Object versioning: version IDs recorded in metadata.
- Checksum: integrity information kept as metadata.