An S3 endpoint is a single hostname standing in front of a set of nodes that fail individually. Most deployments put a load balancer in that position, point a TCP health check at the service port, and treat the design as finished. The gap appears during a real failure, when a node accepts connections and answers on the port while returning errors to every S3 request, and the balancer keeps sending it traffic.
The usual answer is more redundancy: more nodes, a second balancer, a shorter health check interval. That covers the machine that stops responding entirely, which is the least common failure and the least damaging. The harder cases are partial, where a node serves some requests and fails others, or answers slowly enough that clients time out and retry, multiplying load on the nodes still healthy.
What separates endpoint failure from most storage failures is that the client participates in recovery. SDKs cache DNS answers, hold connections open, retry on their own schedule, and carry server-side state across a multipart upload. A failover that looks clean in the balancer's logs can stay visible to applications for minutes.
A TCP health check confirms that something is listening on the port. It does not confirm that the S3 service behind it can verify a signature, read bucket metadata, or reach the storage layer. An HTTP check against a static path is better, but that path is often answered by the web tier in front of the S3 service, which is not the component that fails first.
A check worth relying on performs a real S3 operation with real credentials against a dedicated bucket, such as a signed HEAD of a known object, since that exercises signature verification, the metadata path and a read from persistent storage. Thresholds matter as much as content. Ejecting a node on one failed check turns a transient error into an availability event, while three consecutive failures at five second intervals removes it inside twenty seconds.
Response time belongs in the check as well. A node answering correctly but slowly costs more aggregate throughput than one out of rotation, since clients waiting on it consume connections and retry budget while producing nothing. A health check timeout well below the client timeout lets the balancer notice degradation first, the same asymmetry that makes throughput a poor single measure of endpoint behavior.
DNS time to live governs failover speed only in theory. Resolver libraries, container runtimes and language standard libraries all cache answers, and several cache past the published TTL. Connection pools are worse, resolving the endpoint once at creation and reusing that address for the life of the process.
Long-lived SDK connections are stickier than DNS. An S3 SDK opens a pool of HTTPS connections and keeps them alive, so a client can issue thousands of requests over the same connection, and therefore against the same node, without another lookup. Removing a node from a DNS record does nothing to established connections, which persist until one side closes them or an idle timeout expires.
The number that matters is the interval between removing a node and the last request it serves. Measuring that needs per-node request counts, one reason endpoint counters belong in the metrics collected continuously rather than only during incidents. Two settings bound that interval: a maximum connection age at the balancer, which forces pools to rebuild against current membership, and client-side connection TTL where the SDK exposes it.
Planned removal should drain rather than cut. Draining stops new connections arriving while letting requests in progress finish, then closes the connection once it is idle. The drain window has to exceed the slowest single request the endpoint serves, usually a large PUT or GET over a constrained link rather than a metadata operation.
Multipart uploads complicate this, since they are not single requests. A multipart upload is a sequence of independent HTTP requests sharing server-side state keyed by an upload ID, and it can run for hours. Parts retry against any node as long as that state is visible clusterwide. If it is local to the node that issued the upload ID, a failover leaves the upload unable to complete and its parts consuming capacity, one of the ways unfinished uploads accumulate cost without appearing in bucket listings.
Unclean failure is often less damaging than it looks, since connections reset and the SDK retries against a healthy node. The case that is not routine is a CompleteMultipartUpload arriving at a node with no record of the upload, which returns an error most applications cannot recover from without restarting the transfer.
| Failure mode | What a TCP check reports | What the client experiences |
|---|---|---|
| S3 process alive, storage layer unreachable | Healthy | 500 or 503 on every request that node takes |
| Certificate expired on one node | Healthy | TLS handshake failures on a fraction of new connections |
| Node under memory pressure, responses slow | Healthy | Timeouts and retries, throughput collapsing |
| Authentication path unreachable from one node | Healthy | 403 responses on correctly signed requests |
| Clock drift on one node | Healthy | RequestTimeTooSkewed on everything that node answers |
| Node powered off abruptly | Unhealthy at next interval | Connection resets mid-transfer, then clean retries |
Most of this can be tested during normal operation, and an idle cluster is the wrong place to try, since connection reuse only appears when real clients hold real pools open. The safe starting point is an administrative drain of one node during ordinary traffic. If it produces client-visible errors, either the window is too short or the thresholds are wrong.
The more informative test reproduces partial failure. Dropping packets to one node with a firewall rule, while the S3 service keeps running, creates the state a TCP check cannot see and reveals whether the check tests the API or the socket.
Four numbers make the test worth running: time from removal to the first client error, time to zero errors, requests still served by the removed node, and whether any multipart upload failed. Errors should be broken out by code, since a burst of 503 responses means something different from a burst of 500 responses, as the codes themselves make clear.
RING presents an S3-compatible API across nodes sharing a single namespace, including across sites. Because the namespace is not partitioned by node, a retried request can be served by a different node without the application knowing which one answered first. Multi-site deployments spread data so that an entire site can be lost without data loss, with geographic placement configured by the operator.
Nodes and servers can be added to a live RING system, and older hardware retired, without taking the namespace offline. That is the property an endpoint failover depends on, since membership changes without the S3 service surface changing underneath clients.
RING emits operational metrics and logs the operator keeps locally, which is what makes the per-node request counts described above measurable. Without them a failover test confirms that errors stopped but not that traffic moved.
A drain test on one node, run quarterly during ordinary traffic, catches most of what drifts over time: a new application whose connection pool never recycles, a health check that stopped exercising the storage path, a drain window that no longer covers the largest object being written.
The record kept alongside it should be short and specific. The endpoint hostname and every address it resolves to. The health check definition, including the S3 operation, the credentials, the interval and both thresholds. The drain timeout and the longest request seen in the previous quarter. The maximum connection age at the balancer. And for each application, whether its SDK caches DNS and what its retry policy does on a 503.
The final entry is the measured failover time rather than the designed one, dated and attached to the test that produced it. A designed failover time is a configuration value. A measured one is evidence.