A bucket listing shows completed objects. It does not show the parts of multipart uploads that were started and never finished, and those parts consume capacity on the same disks as everything else. On a cluster that has run for a few years, the gap between what the listing accounts for and what the hardware has actually filled can be large, and nothing in the ordinary monitoring path points at it.
The usual first move is to compare raw used capacity against the sum of object sizes and call the difference protection overhead. That mixes several causes together. Erasure coding and replication inflate raw usage by a known factor, non-current versions accumulate quietly, and delete markers hide objects that still occupy space. Where hidden capacity goes is rarely explained by one of them alone.
Incomplete multipart uploads are the component most often missed, because they are the only one a standard object listing cannot show at all. A ListObjectsV2 call returns keys that exist. An upload that was initiated and never completed has no key. It has an upload ID, a set of parts, and a footprint on disk.
Why unfinished uploads accumulate
Multipart upload is not an unusual path. Most S3 clients switch to it automatically above a configured threshold, and some libraries use it for anything larger than a few megabytes. Any workload writing large objects therefore runs a three-step protocol: initiate, upload parts, complete. An upload that fails between the first step and the third leaves its parts in place indefinitely, because the server cannot know whether the client still intends to finish.
The causes are ordinary. A client process is killed mid-transfer. A network interruption ends an upload and the retry starts a fresh one rather than resuming the old. A batch job is canceled and its cleanup path never runs. Backup software abandons an upload when the job window closes. A migration tool is restarted after a failure and begins again from the source, orphaning the first attempt. None of these produce an error anyone investigates afterwards, since from the application's point of view the retry succeeded.
Accumulation tracks failure frequency rather than data volume, so one unstable network segment or one client with aggressive timeouts can leave far more orphaned parts than throughput suggests. Diagnosing application failures at the request level usually surfaces the same instability.
Finding the parts that do not appear in a listing
The call is ListMultipartUploads, issued per bucket. It returns the upload ID, the destination key, the initiator identity and the initiation timestamp. That timestamp is the most useful field, since an upload initiated months ago is abandoned rather than in progress, whatever the application team believes.
Listing the uploads establishes their existence, not their size. For size, each upload ID has to be passed to ListParts, which returns part numbers, sizes and per-part ETags. Summing part sizes across every open upload in every bucket produces the real figure. On a cluster with many buckets that is a scripted operation, and it is worth writing once and keeping, since the number is only meaningful as a trend.
Two details make this awkward. Both calls are paginated, so a script that reads only the first page understates the total, sometimes badly. And uploads are enumerated per bucket, so the inventory has to start from a complete bucket list, including buckets nobody has opened since the workload that created them was retired.
What the orphaned parts actually cost
Parts are stored under the same durability policy as completed objects, so raw capacity consumed exceeds the sum of the part sizes. The choice described in erasure coding vs replication sets that multiplier, and it applies just as fully to data no application will read again.
There is a second cost capacity dashboards do not show. Every part is a metadata entry, and abandoned parts add work to background scanning and repair, which competes with live requests for the same resources.
| Signal in the upload inventory | What it usually means | What to check |
|---|---|---|
| Timestamps spread evenly across months | A steady low rate of client failures, not one incident | Which initiator and prefixes dominate, and the client's retry settings |
| Many uploads initiated within the same hour | A single job failed or was canceled partway | Job logs for that window, and whether the job has any abort path at all |
| Open uploads on a bucket with no recent writes | A decommissioned workload left residue behind | Whether the bucket itself should be emptied and removed |
| Repeated upload IDs against the same key | A scheduled job failing at the same point each run | The job's timeout against the observed duration of a successful transfer |
| Parts smaller than the client's usual part size | Uploads cut off early, often on a congested path | Connection resets on that client's route to the endpoint |
Setting an abort rule without breaking live jobs
The durable fix is a lifecycle rule with an AbortIncompleteMultipartUpload action and a DaysAfterInitiation value. Once an upload has been open longer than that, the system aborts it and reclaims the parts without operator involvement. The rule can be scoped to a prefix, so a bucket holding mixed workloads does not need one window for all of them.
The window is not a matter of taste. It should be derived from the longest legitimate upload duration on that bucket, meaning the slowest link, the largest object and the most patient client, then padded. Seven days is a common starting point and is too short for some archive ingests. Choosing it blindly converts a capacity problem into a transfer problem, because an upload aborted mid-flight fails at the complete step and the application sees an error it has probably never handled.
Check three things first: whether any client performs resumable uploads that legitimately span days, whether a migration or seeding job is in flight, and whether the bucket already has a lifecycle configuration, since putting one replaces the existing configuration rather than adding to it. The discipline described in testing before deleting data applies here too, even though an abort rule never touches a completed object.
Where Scality RING fits
RING implements the S3 multipart protocol, including ListMultipartUploads and ListParts for inspection and the AbortIncompleteMultipartUpload lifecycle action for reclamation, so both the audit and the remedy use standard S3 calls rather than a vendor interface. The same inventory script can therefore run against RING and against any other S3 endpoint in the estate.
Because RING is deployed on the operator's own hardware, reclaimed capacity returns to the operator's own pool rather than reducing a bill. The effect is on timing, since no monthly invoice line makes the waste visible and the inventory has to be run deliberately. RING's multi-tenant model, with accounts, buckets and per-tenant quotas, gives a natural unit of attribution, because open uploads trace back to the account that initiated them.
RING applies its configured durability policy to multipart parts as it does to completed objects, and it emits operational metrics the operator keeps locally. Pairing a periodic upload inventory with those metrics gives the comparison that matters, which is the trend in reclaimable space rather than a snapshot.
What to put on the schedule
Run a cluster-wide multipart inventory quarterly and record two numbers per bucket, the count of open uploads and the summed part bytes, then keep the history. A stable count is normal residue, a climbing count points at a client or job that needs attention, and a sudden jump carries a date that maps to a specific failure.
Make an abort rule part of the bucket creation checklist rather than a remediation task, and document the DaysAfterInitiation value alongside the longest legitimate upload the bucket is expected to carry. Review it whenever a new workload is pointed at the bucket, particularly one writing very large objects over slow links.
Finally, record which buckets have no rule and why. A bucket exempted because a long seeding job is in progress is a manageable exception when it is written down with an end date. An exemption nobody remembers making is how the next several terabytes disappear.