Glossary
NVMe over Fabrics
NVMe over Fabrics (NVMe-oF) extends the NVMe command set across a network, so a server can use flash in another machine almost as if it were installed locally. It lets block volumes grow separately from compute and removes the stranded capacity of drives stuck inside individual servers.
What problem does NVMe-oF solve?
Flash inside a server is usable only by that server. One rack runs out while the next has plenty idle, and the only fix is to move data or buy more. NVMe-oF puts the flash in shared enclosures. Servers connect across the network and see raw block volumes, with the parallel queues of NVMe carried end to end, so latency stays close to local.
Compute and storage can then be expanded on different schedules, which is the basis of disaggregated designs.
Which transport carries it?
| Transport | What it needs | Character |
|---|---|---|
| RDMA (RoCE, InfiniBand) | A tuned, lossless fabric | Lowest latency, hardest to operate |
| Fibre Channel | An existing SAN | Familiar to storage teams, reuses cabling |
| NVMe/TCP | Ordinary Ethernet | Slightly higher latency, easiest to deploy |
The choice follows the network the organization already runs. RDMA over Ethernet works only when the fabric is configured not to drop packets, which takes deliberate design and ongoing tuning. A shop with a storage area network can carry NVMe over its existing Fibre Channel. Plain Ethernet suits NVMe/TCP, accepting a small latency penalty for simpler operations. Relative ordering is consistent, but absolute latency numbers depend on the hardware.
What does sharing flash put at risk?
One shared enclosure serves many servers, so its failure reaches all of them. The blast radius of a controller fault, a firmware bug or a network event is larger than for a local drive. Designs add redundant enclosures, multiple network paths and data protection across enclosures. Shared flash also puts the network in the critical path of every read, so congestion that once delayed a backup now delays a database.
Where does it sit among storage models?
NVMe-oF presents block volumes, so it competes with the SAN and with hyper-converged infrastructure, where each server contributes its own drives. It targets latency-sensitive workloads. Large unstructured data usually lives in object storage, reached over S3 rather than block commands.
Frequently asked questions
Is NVMe-oF the same as NVMe/TCP?
No. NVMe/TCP is one transport. NVMe-oF is the general approach, which also runs over RDMA and Fibre Channel.
Does NVMe-oF replace the SAN?
It can run on one. Many SANs adopt it as a faster protocol over the same Fibre Channel network.
Why does RDMA need a lossless network?
RDMA assumes packets are not dropped and recovers poorly when they are, so the fabric has to be configured to prevent loss.
Related terms
- NVMe: the protocol being extended.
- Storage area network: the established shared block model.
- Block storage: what NVMe-oF presents.
- Hyper-converged infrastructure: local drives pooled across servers.
- Storage latency: what the transport choice changes.