Storage fundamentals
Multipart upload
Multipart upload is the S3 method for writing a large object as a series of separate parts that the storage system assembles into one object at the end. Parts can be sent in parallel and retried individually, which makes large transfers faster and more tolerant of network errors than a single long upload.
How does a multipart upload work?
The process has three steps, each a call in the S3 API.
| Step | What happens |
|---|---|
| Initiate | The client asks to start an upload and receives an upload ID |
| Upload parts | The client sends numbered parts, in any order and in parallel, each acknowledged with a checksum tag |
| Complete | The client sends the list of parts and the storage system assembles the object |
| Abort (alternative) | The client cancels and the stored parts are discarded |
Why use it?
A single upload of a large object has to restart from the beginning if the connection fails. With parts, only the failed part is resent. Several parts can travel at once over separate connections, which raises storage throughput on fast networks and makes better use of a storage system that spreads writes across many nodes. It also allows an upload to begin before the final size is known, which suits streaming sources such as database dumps and backup software.
What are the rules for parts?
In the S3 specification, an upload can have up to 10,000 parts, and every part except the last must be at least 5 MB. Clients pick a part size that keeps the count within the limit, often tens of megabytes for very large objects. The object's entity tag after a multipart upload is not a simple checksum of the whole content, which matters for tools that try to verify integrity by comparing tags. Most development kits and transfer tools handle part sizing, parallelism and retries automatically.
What happens to incomplete uploads?
If a client starts an upload and never completes or aborts it, the parts already sent stay in the bucket. They are invisible to a normal object listing and still occupy capacity. On a busy system with unreliable clients, abandoned parts can add up to a meaningful amount. A lifecycle rule that aborts incomplete multipart uploads after a set number of days is the standard remedy and belongs in every bucket's baseline configuration. S3 multipart uploads covers tuning and cleanup in more depth.
Frequently asked questions
When should multipart upload be used?
For objects from around 100 MB upward, and for any upload over an unreliable link. Tools usually switch to it automatically above a threshold.
Can parts be uploaded out of order?
Yes. Each part has a number, and the storage system assembles them in numerical order on completion.
Is there a multipart equivalent for downloads?
Range requests serve the same purpose, letting a client fetch sections of an object in parallel.
Related terms
- S3 API: the interface that defines multipart upload.
- Object storage: the model in which objects are written whole.
- Storage throughput: what parallel parts improve.
- S3 bucket: where incomplete parts accumulate.
- Checksum: how each part is verified.