NVMe over Fabrics is a protocol that extends the high-performance NVMe interface over network connections, enabling multiple systems to access remote high-performance flash storage with microsecond-level latency characteristics similar to directly attached storage.
Traditional network storage protocols like iSCSI and Fibre Channel were designed around mechanical disk drive latency characteristics—milliseconds. When you introduced these protocols over networks, the additional network latency was negligible compared to the millisecond-level latency disk drives provided. But flash storage operates at microsecond-level latency. Introducing traditional network storage protocols creates bottlenecks that obscure the performance advantages flash provides. NVMe over Fabrics eliminates this bottleneck by using the efficient NVMe protocol over network connections, enabling organizations to build high-performance shared storage architectures with all-flash arrays accessible to multiple systems.
Why NVMe over Fabrics Matters for Enterprise Architecture
NVMe over Fabrics enables architectural flexibility that traditional storage protocols can’t provide. Rather than dedicating storage infrastructure to individual servers, organizations can deploy shared all-flash arrays accessible over networks. This enables dynamic resource allocation where storage capacity can be allocated to systems that need it without requiring physical storage array reconfiguration.
The latency characteristics of NVMe over Fabrics make it practical for workloads that previously required directly attached storage. Databases, real-time analytics platforms, and other latency-sensitive applications can now access shared storage over networks while maintaining the microsecond-level latency they require. This flexibility enables better infrastructure utilization compared to dedicated storage per system.
For large enterprises managing complex infrastructure, NVMe over Fabrics enables new storage deployment models. Data centers can maintain centralized shared enterprise flash storage accessible from multiple systems. This centralization simplifies disaster recovery by enabling geographic redundancy where all systems access the same replicated data, rather than managing individual storage replication for each system.
How NVMe over Fabrics Works
NVMe over Fabrics uses several transport mechanisms to carry NVMe protocol across networks. NVMe over Ethernet (RoCE) runs NVMe over Ethernet networks using RDMA (Remote Direct Memory Access). RDMA enables computers to access each other’s memory directly without CPU involvement, allowing low-latency communication. NVMe over Fibre Channel (FC-NVMe) carries NVMe over existing Fibre Channel networks. NVMe over InfiniBand extends NVMe over high-performance InfiniBand networks.
Each transport mechanism offers different tradeoffs between cost, latency, and complexity. NVMe over Ethernet (RoCE) is most cost-effective because it uses standard Ethernet infrastructure. Fibre Channel and InfiniBand typically provide lower latency but require specialized network infrastructure. Organizations typically choose transport mechanisms based on existing network infrastructure and specific latency requirements.
The architecture of NVMe over Fabrics systems involves one or more systems acting as storage controllers exposing flash storage over networks, while multiple client systems access the storage over network connections. This enables one physical storage system to serve multiple hosts simultaneously, more efficiently utilizing expensive flash storage infrastructure.
Key Considerations for NVMe over Fabrics Deployment
Network latency becomes critical in NVMe over Fabrics deployments. While NVMe directly attached to local PCIe buses achieves microsecond-level latency, NVMe over Fabrics latency includes network traversal time. Well-optimized NVMe over Ethernet (RoCE) implementations achieve single-digit millisecond latency, which is still dramatically better than traditional storage protocols but notably higher than local NVMe. Organizations should understand latency requirements for their workloads before choosing NVMe over Fabrics.
Network bandwidth also impacts deployment success. Flash storage can provide enormous IOPS and throughput. The network connecting systems to NVMe over Fabrics storage must have sufficient bandwidth to not become a bottleneck. A single system accessing all-flash storage over a limited network might saturate the network and not achieve the IOPS the storage can provide.
Organizations deploying NVMe over Fabrics must also consider protocol overhead and CPU utilization. RDMA and other optimizations reduce overhead, but NVMe over Fabrics still consumes more CPU resources than direct-attached storage. System sizing should account for the CPU overhead of network-based storage access.
NVMe over Fabrics in Enterprise Deployments
Many modern all-flash arrays include NVMe over Fabrics capabilities, enabling them to be accessed over networks while maintaining flash-level performance. This allows organizations to consolidate storage infrastructure and enable flexible resource allocation. Systems can dynamically request storage capacity rather than having static storage allocations.
Organizations should understand how NVMe over Fabrics relates to disaster recovery architectures. Shared NVMe over Fabrics storage enables simpler disaster recovery—multiple systems accessing the same storage can all be failed over to secondary locations together, maintaining their shared storage relationships. This simplifies orchestration compared to systems with individual storage that must be replicated separately.
The relationship between NVMe over Fabrics and flash storage latency is important. While local NVMe achieves microsecond latency, NVMe over Fabrics adds network latency. Understanding the specific latency characteristics of your network and transport mechanism is essential for NVMe over Fabrics deployments.
Advanced Topics
Some organizations implement NVMe over Fabrics across multiple data centers for disaster recovery purposes. This enables multiple facilities to access the same shared storage, with replication at the storage layer ensuring data availability. This approach simplifies disaster recovery compared to application-level replication.
Organizations should also consider storage tiering approaches where NVMe over Fabrics storage serves as the performance tier while less expensive storage handles capacity needs. This hybrid approach can provide cost-effective solutions balancing performance requirements with budget constraints.