Glossary

Storage bandwidth

Storage bandwidth refers to the maximum rate at which a link or component in the storage path can carry data, fixed by its signalling rate and width.

Bandwidth belongs to the hardware. The rate a workload reaches beneath that ceiling is storage throughput.

Why bandwidth matters in multi-site and scale-out storage

In a scale-out cluster, data moves constantly between components: clients to servers, servers to drives, servers to each other for protection and rebuild, and sites to sites for replication. Each of those paths has a fixed ceiling, and the narrowest one sets what the whole system can do. For teams running storage across several sites, inter-site bandwidth is often the most expensive and least elastic link in the design, and it decides how fast data can be protected, moved or recovered.

Units, encoding and overhead

Network rates are quoted in bits per second and storage rates in bytes, so 100 Gb/s is 12.5 GB/s before any overhead. Decimal and binary units also differ: 1 GB/s is about 7 percent less than 1 GiB/s.

Links spend part of their raw rate on encoding and headers. SATA III signals at 6 Gb/s but carries 600 MB/s of data after 8b/10b encoding. PCIe 3.0 and later lose under 2 percent to encoding. On Ethernet, a standard 1,500-byte frame delivers 1,460 bytes of TCP payload out of 1,538 bytes on the wire, about 94.9 percent, so a 100 Gb/s link carries at most about 11.9 GB/s of TCP payload; jumbo frames raise that to about 99 percent. Storage protocols, TLS and replication traffic take further shares.

Bandwidth by component

PCIe and full-duplex Ethernet carry these rates in both directions at once. Figures quoted as the sum of both directions double the number without changing what one direction can carry.

Oversubscription and the bandwidth-delay product

Shared links are usually provisioned below the sum of the links that feed them. A leaf switch with 48 server ports at 25 Gb/s (1,200 Gb/s) and four 100 Gb/s uplinks (400 Gb/s) is oversubscribed 3:1. A PCIe switch joining 24 NVMe drives to one ×16 upstream port is oversubscribed about 6:1. Under oversubscription, the bandwidth any one device gets depends on what its neighbours are doing.

Distance adds a second constraint. A link is only full when enough data is in flight to cover the round trip, an amount equal to bandwidth × round-trip time. At 10 Gb/s and 5 ms that is 6.25 MB; at 100 Gb/s and 5 ms, 62.5 MB. If TCP windows or request concurrency cannot keep that much outstanding, achieved throughput falls short of the link however fast it is. Fibre adds about 1 ms of round trip per 100 km, as covered under storage latency.

What bandwidth means for multi-site operations

For multi-site operations, bandwidth turns into days. A 10 Gb/s link running flat out moves 1.25 GB/s × 86,400 s = 108 TB a day; at a realistic 50 percent average, 54 TB. A site that creates or changes more data than that each day builds a replication backlog, and the gap between sites widens until the change rate falls.

Seeding and recovery make the numbers larger. Copying 1 PB across a fully used 10 Gb/s link takes about 800,000 seconds, more than nine days, before overhead. Resynchronising a site after a long outage, or restoring a large dataset from a remote copy, runs on the same arithmetic, so the inter-site link sets the realistic recovery time as firmly as any software feature does. Moving data to and from public cloud adds egress fees to the bandwidth bill.

Inside a site, the same reasoning applies to rebuilds. Recovering a failed server's data uses the network between the remaining servers, and the time a cluster spends with reduced protection depends on how much of that bandwidth rebuild traffic can take without starving production.

Bandwidth upgrades also tend to arrive in steps. Inter-site circuits are bought in fixed increments and often take months to provision, so headroom for growth in daily change rate has to be planned well ahead of the point where replication lag becomes visible.

Bandwidth in Scality RING deployments

For RING clusters stretched across sites, Scality publishes an envelope of "10Gb/s or greater bandwidth and <5ms latency" between them (Solved by Scality). At the floor of that envelope, filling the link takes about 6.25 MB in flight.

Within a site, bandwidth planning comes down to drives, server ports and fabric uplinks, and the published RING XP build used one 100GbE port per server (Solved by Scality). Each of those nodes can therefore exchange at most 12.5 GB/s with the rest of the cluster in each direction.