An application fails against object storage and the ticket says the upload did not work. The response that caused it carried an HTTP status, an S3 error code, a message and a request ID, and in most cases none of that reached the log line the operator is now reading. Error codes are the most precise diagnostic signal the S3 API produces, and they are routinely discarded before anyone can use them.
The usual response is to turn up logging. That produces volume without precision, since the default failure path in most application wrappers catches an exception, logs its message, and drops the structured fields carrying the code and the request ID.
Codes also get lost because several are deliberately vague. S3 semantics include responses chosen to avoid leaking information, so the code reported is not always the one describing what happened. Reading them well means knowing which are precise and which are a category.
What the status class narrows down
The split between 4xx and 5xx allocates the investigation. A 4xx says the request, as received, was not one the endpoint would accept: wrong credentials, wrong path, insufficient permissions, a malformed body, a signature that did not verify. A 5xx says the endpoint accepted it and then failed to complete it. The first is usually an application or configuration question, the second a storage or infrastructure one.
That split is a starting point rather than a rule. A 403 can come from a load balancer rewriting a header. A 500 can come from an application sending a request the server mishandles. What the split reliably does is decide who looks first, and where storage and application teams report failures to each other, that is most of the value.
The status alone is not enough, since one status carries very different codes. A 403 might be AccessDenied, SignatureDoesNotMatch, RequestTimeTooSkewed or InvalidAccessKeyId, and those four share almost nothing as causes. Logging the status without the code discards most of what the response contained.
SlowDown, 503 and the errors that describe capacity
SlowDown, and 503 responses generally, are the only S3 errors that describe load rather than the request. They mean the endpoint declined to serve something it would otherwise have served, and they signal that request rate has exceeded what some part of the system sustains.
The complication is that SDK retry logic handles them transparently. An application retrying with exponential backoff absorbs a moderate rate of 503 responses and reports success, so the first visible symptom is latency rather than errors. By the time SlowDown reaches application logs, retries are no longer covering it and are themselves adding to the load.
Counting them at the endpoint rather than waiting for application logs is the difference between a capacity signal and an incident. A rising 503 rate against a flat request rate means capacity is degrading; against a rising request rate it means the workload changed. Telling the two apart needs both series, one reason request rate and response code distribution belong together in the metrics collected continuously. How concurrency and request mix translate into latency is covered in the case for looking past throughput.
Errors that hide more than they reveal
AccessDenied is the most common and least specific error in S3. It can mean the identity is not permitted the action, that a bucket policy denies it, that the object is owned by another account, or that the object does not exist and the requester may not be told so. That last case consumes the most time, since the investigation starts from a false assumption.
NoSuchKey and NoSuchBucket are precise and worth separating in logs. NoSuchKey means the bucket was found and the key was not, which points at key construction, a lifecycle rule, or a delete marker in a versioned bucket. NoSuchBucket means the name itself did not resolve, usually a region or endpoint misconfiguration rather than anything about the data. Code that treats both as "not found" erases the distinction that decides where to look next.
SignatureDoesNotMatch is a request integrity failure and not necessarily a credential failure. The signature covers the method, the path, a set of headers and the payload hash, so anything modifying the request in transit breaks it. A proxy that normalizes the path, a balancer that rewrites the Host header, or a gateway that injects a header inside the signed set all produce it with valid credentials. RequestTimeTooSkewed is its relative, caused by the client clock drifting outside the permitted window, and it appears most often on container hosts where time was never synchronized.
| Error code | What it usually means | What to check first |
|---|---|---|
| SlowDown or 503 | Request rate above what the endpoint sustains | Rate per prefix, client concurrency, retry amplification |
| AccessDenied | Policy denial, ownership mismatch, or a hidden missing object | Which identity signed, and whether the exact action is granted |
| SignatureDoesNotMatch | The request changed between signing and arrival | Proxies and balancers on the path, and recent key rotation timing |
| RequestTimeTooSkewed | Client clock outside the allowed window | Time synchronization on the signing host, container clocks especially |
| NoSuchKey | Bucket resolved, key absent at that exact string | Prefix and case in the key, lifecycle rules, delete markers |
| InternalError or 500 | Server-side fault, frequently transient | Whether it is one node or all, correlated by request ID |
Why the code rarely reaches the log
SDKs raise typed exceptions carrying the code, the status, the request ID and often an extended request ID. Application error handling usually catches a broad exception class and logs its string form, which renders the message but not the structured fields. The information exists in memory at the moment of failure and is discarded one line later.
The request ID is the field whose loss hurts most. It is the only value correlating a client-side failure with a specific server-side log entry, and without it an investigation becomes a search over time ranges. Recording it costs one field.
Permission failures illustrate the cost. Diagnosing an AccessDenied means knowing which identity signed the request, which action was attempted and against which resource, none of which is in the message string. Policies are easier to operate when the failures they produce are legible, an argument for the structured approach in least-privilege policy design.
Where Scality RING fits
RING presents an S3-compatible API with accounts, users, buckets and IAM-style policies, so error codes follow S3 semantics rather than a vendor-specific scheme. That matters for diagnosis, since the codes then mean what the S3 documentation says, and the difference between a native implementation and a partial one shows up precisely in edge-case error behavior, as set out in the comparison of the two.
Because RING runs in the operator's own facility and emits metrics and logs kept locally, the server side of a request ID correlation is available without involving a third party. A failed request in an application log can be matched against the endpoint's own record of it.
Multi-tenancy in RING uses per-tenant isolation and quotas, so capacity and permission errors can be attributed to a tenant rather than appearing as an undifferentiated cluster symptom.
What to log and what to check
A logging standard for S3 failures needs five fields: the HTTP status, the error code, the request ID, the bucket and key, and the signing identity. Any handler catching an S3 exception should record all five, and a code review rule enforces that better than documentation.
On the operations side, the response code distribution belongs in a standing dashboard panel rather than a query someone writes mid-incident, broken out by code and not only by status class. A weekly look at the 4xx mix catches credential rotations that half-succeeded and policy changes that quietly broke one application.
Two checks belong on a recurring schedule. Clock synchronization on every host that signs requests, since RequestTimeTooSkewed is silent until it is total. And a review of what sits between clients and the endpoint, since every proxy and gateway on that path is another chance for a signature to break in a way that looks like a credential problem.