A storage system rated at 200 GB/s and a cluster that can consume it will still frequently meet somewhere around 40. The shortfall is almost never in the drives. It accumulates in the path between them: in uplinks that are narrower than the sum of what they serve, in clients that cannot drive their own network ports, in single connections that cannot fill a link, and in devices in the middle that were sized for a different kind of traffic.
Finding where it goes is a matter of measuring at each boundary rather than at the ends. Infrastructure teams who only compare the storage figure against the observed figure have one number and no explanation. Measuring per hop turns it into a short list of specific limits, most of which have straightforward fixes.
Start by calculating the ceiling at every layer
Before measuring anything, work out what each layer can carry in theory. Storage node ports multiplied by node count gives the aggregate the storage side can offer. Client ports multiplied by client count gives what the compute side can absorb. Leaf to spine uplinks give what the fabric can carry between them. The smallest of those three is the ceiling, and no amount of tuning will exceed it.
This calculation frequently ends the investigation on its own. A cluster of 32 servers with a single 100 GbE port each cannot receive more than about 400 GB/s in theory and considerably less in practice, regardless of what the storage system is capable of. If the design expected more, the constraint was built in at purchase.
Write the numbers down and compare them against the requirement the pipeline actually has. Teams investigating idle accelerators often discover the answer at this stage, before any packet has been captured.
The places bandwidth usually disappears
The list below covers the causes that account for most of the gap in real deployments, roughly in the order they are worth checking.
| Location | Symptom | How to confirm it |
|---|---|---|
| Leaf to spine oversubscription | Throughput falls as more clients join | Compare access port total against uplink capacity |
| Single client port limit | Per-client rate plateaus at a round number | Measure one client in isolation at full rate |
| Single stream throughput | One connection cannot fill the link | Compare one connection against many in parallel |
| Client processor and interrupt load | High system time while throughput is flat | Watch processor utilization on the client during transfer |
| Encryption overhead | Plain transfers are faster than encrypted ones | Compare with and without transport encryption |
| Load balancer or proxy in the path | Aggregate ceiling unrelated to storage capacity | Test against endpoints directly, bypassing it |
| Uneven endpoint distribution | Some storage nodes busy, others idle | Compare per-node throughput during a run |
| Packet loss and retransmission | Throughput collapses intermittently | Check interface error and retransmit counters |
Uneven distribution is worth particular attention in object storage environments, because clients often resolve a single endpoint name and a naive resolution can send a disproportionate share of requests to a subset of nodes. The storage system reports plenty of spare capacity while individual nodes are saturated, which makes the aggregate figures misleading.
Why one connection is rarely enough
A single connection between two machines is limited by more than link speed. Round trip time, receive window sizing and the processing cost of a single flow all cap what one stream can achieve, and on a high speed link the cap is often well below the port rate. This is why a data loader configured with a small number of workers can leave most of the network idle.
The fix is parallelism, both in the number of connections and in the number of requests in flight on each one. Object storage clients usually expose these as configuration: concurrency, connection pool size and multipart part size. Raising them is often the single most effective change available, and it costs nothing.
There is a limit. Beyond a certain concurrency, additional requests add queuing rather than throughput, and on small object workloads the constraint moves from bandwidth to request handling entirely. That transition is a useful thing to locate deliberately, because it tells the team which resource to buy more of.
The client side is part of the storage design
Bandwidth is frequently lost inside the client rather than on the wire. Moving tens of gigabytes per second through a host consumes processor time for interrupts, memory copies and encryption, and a host whose processors are already busy preparing training batches may not have enough left to drive its network port.
Symptoms are recognizable: high system time, a throughput plateau that does not respond to more connections, and a rate that improves when preprocessing is reduced. Where this is the constraint, the effective remedies are on the client, through offload features, larger transfer sizes, fewer copies in the data path, or moving preprocessing work elsewhere so the host has capacity to receive data.
Transport encryption deserves a specific decision rather than a default. It has a real cost at these rates, and whether to accept it depends on whether the path between compute and storage is already a controlled network. That is a security judgement, but it should be made knowingly rather than discovered during a performance investigation.
Where Scality RING fits in the network design
Scality RING scales by adding nodes, so its aggregate throughput is the sum of what its nodes can serve and the network design has to be able to carry that sum. The practical implication is that client requests need to spread across storage endpoints, which usually means a load balancing approach that distributes connections evenly and that is itself sized for the aggregate rather than for a typical share of it.
Because a RING deployment grows over time, the network sizing should be revisited when nodes are added. Storage capacity that arrives without matching fabric capacity produces a system that is larger but not faster, which is a common and avoidable outcome. Reviewing uplink ratios and endpoint distribution as part of an expansion is cheaper than discovering the shortfall during a training campaign.
Measure at each boundary, not just at the ends
The investigation that works is mechanical. Measure one client against one storage node to establish the per-pair rate. Add clients until the aggregate stops rising and note where it stopped. Compare per-node throughput to find uneven distribution. Then compare those observed numbers against the theoretical ceilings calculated earlier. The layer where observation first falls well short of theory is where the bandwidth is going.
Keep those measurements, because they become the baseline for every future change. When throughput drops after a cluster expansion or a client image update, having last quarter's numbers per hop turns a long investigation into a short comparison, which is the same discipline that makes pre-purchase testing useful rather than decorative.