As data center architectures evolve to support real-time analytics, high online transaction processing (OLTP) databases, and AI inference workloads, storage architects face a classic dilemma: how to scale block storage performance without inducing massive infrastructure costs or operational complexity.
Traditional storage protocols like iSCSI introduce significant translation overhead, creating severe CPU bottlenecks. While NVMe over Fabrics (NVMe-oF®) solves this by extending native NVMe® parallelism across the network, early iterations required expensive, specialized hardware. But that’s not the case any longer.
Enter NVMe over TCP (NVMe/TCP), invented by Lightbits Labs. A modern protocol for high-performance data access over low-latency networks. It encapsulates NVMe commands directly within standard TCP/IP packets, enabling enterprise storage systems to achieve near-local NVMe performance over standard Ethernet infrastructure.
What Network Requirements Are Needed for NVMe over TCP?
One of the primary economic and operational drivers of NVMe over TCP adoption is its compatibility with standard, off-the-shelf networking components. Unlike other fabric transports, it does not require proprietary components. However, delivering deterministic performance for workloads like AI inference requires specific optimizations:
- Bandwidth & Throughput: While NVMe/TCP can operate over 10GbE, high-performance workloads (such as real-time analytics and AI Inference) are better suited for 25GbE, 50GbE, or 100GbE+ networks to prevent serialization bottlenecks.
- Standard NICs and Switches: It utilizes commodity NICs and standard Ethernet switches. There is no requirement for specialized hardware.
How does NVMe/TCP compare to NVMe over RoCE?
When choosing an NVMe-oF transport mechanism, architects typically evaluate NVMe/TCP against NVMe/RoCE (RDMA over Converged Ethernet). Both unlock NVMe parallelism, but they target fundamentally different architectural priorities.
Architectural Trade-offs
| Feature / Metric | NVMe/TCP | NVMe/RoCE |
|---|---|---|
| Network Infrastructure | Standard commodity Ethernet | Lossless Ethernet |
| Hardware Requirements | Standard NICs & Switches | RDMA-capable NICs & Specialized Switches |
| Average Latency | 100 – 300 microseconds | 10 – 150 microseconds |
| Host CPU Utilization | Moderate (Handled by software TCP stack) | Near-Zero (Kernel bypass via hardware offload) |
| Deployment Complexity | Low (Plug-and-play with existing gear) | High (Proprietary coponents & requires advanced fabric tuning) |
| Best Used For | Cloud-native apps, Kubernetes, real-time analytics, high transactional databases, and distributed AI inference scaling | High-Frequency Trading, AI/ML LLM training at scale, 8K uncompressed video editing |
If your primary constraint is achieving the absolute lowest latency (sub-50 microseconds) and you have the staff to manage a complex, lossless network, NVMe/RoCE is a good choice. However, if you require a highly scalable, cost-effective shared pool of storage that integrates seamlessly into a modern Kubernetes pipeline without a hardware forklift upgrade, NVMe/TCP is the best choice.
What are the Latency and Throughput Advantages of NVMe/TCP?
Evaluating NVMe/TCP against legacy protocols like iSCSI reveals stark performance deltas. NVMe/TCP eliminates the legacy SCSI command emulation layer, mapping application I/O requests directly to the NVMe controller.
- Massive Parallelism: Legacy iSCSI relies on a single, sequential command queue. NVMe/TCP preserves the native NVMe queue architecture, supporting up to 65,535 I/O parallel queue pairs, with each queue accommodating up to 65,535 concurrent commands.
- Low Latency: NVMe/TCP delivers latency in the 100-300 microsecond range. Compared to traditional iSCSI deployments, this represents an average 25-40% reduction in storage latency, delivering consistent tail latency for high-transaction databases.
- Throughput and IOPS Efficiency: By streamlining the data path, NVMe/TCP often yields up to a 35% increase in IOPS over iSCSI on identical network hardware. Furthermore, benchmarks indicate that NVMe/TCP can deliver up to 4x the IOPS per CPU core compared to iSCSI, reclaiming critical compute cycles for application workloads such as AI inference engines.
How Do You Deploy NVMe/TCP in a Data Center?
Deploying NVMe/TCP follows a standard fabric storage lifecycle consisting of target configuration, network segregation, and initiator connection.
Deploying Lightbits software-defined storage, designed with VMe/TCP, in a data center is straightforward because it maps directly onto standard Ethernet infrastructure without requiring proprietary hardware or complex RDMA fabrics.
- Step 1: Prepare the Network Fabric. Because Lightbits runs on standard TCP/IP, you do not need to rewrite your network stack.
- Step 2: Set Up Lightbits Storage Nodes. Lightbits runs on standard x86 commodity servers equipped with local PCIe NVMe SSDs.
- Step 3: Provision Storage Volumes. Once the cluster is online, you create volumes to expose to your compute nodes.
- Step 4: Connect the Compute Hosts. Your compute servers require no proprietary drivers—they use the native Linux kernel or standard virtualization hooks.
- For Kubernetes environments: Simply install the open-source Lightbits CSI driver in your cluster. This automates the discovery, attachment, and formatting of volumes dynamically as Pods spin up.
- For VMware Environments: Use the native NVMe/TCP adapter inside vSphere (7.0U3+) to automatically map Lightbits storage arrays directly into your ESXi storage pool via vCenter.
What Software Platforms Support NVMe/TCP?
There is broad, native support for NVMe/TCP across operating systems, virtualization hypervisors, and container orchestration layers.
Operating Systems & Hypervisors
Linux (Kernel 5.0+): Features mature, production-grade native drivers for both the NVMe/TCP initiator and target.
- Linux: Features mature, production-grade native drivers for both the NVMe/TCP initiator and target. Download the free NVMe/TCP Drivers.
- VMware vSphere (7.0 Update 3 and later): Offers built-in support for NVMe/TCP initiators, allowing virtualization admins to connect ESXi hosts to shared NVMe-oF arrays using standard VMware storage workflows. Download the solutions brief Lightbits for VMware.
- Windows Server: Native NVMe-oF initiators that support both TCP and RDMA transports are integrated into modern server builds. Learn more about Lightbits’ Early, Initial Interoperability with Microsoft Windows Server NVMe-oF Initiator Preview
Container & Cloud-Native Platforms
- Red Hat OpenShift Virtualization: Fully supports NVMe/TCP, providing significantly faster VM provisioning times and minimizing operational overhead. Lightbits provides a fast and cost-efficient platform for modern application development, delivering the speed of direct-attached SSDs with the manageability of networked storage. Tailored for organizations seeking to modernize their data centers for efficiency, flexibility, and performance by combining software-defined storage with open-source solutions like OpenShift-V to run both containerized and virtualized workloads side by side. Learn more about NVMe over TCP storage for OpenShift-V.
- Kubernetes: Leading Software-Defined Storage platforms—such as Lightbits Labs—provide a native CSI driver that leverages NVMe/TCP. This allows DevOps teams to dynamically provision high-IOPS, persistent block storage directly to stateful containers that execute real-time streaming analytics or high-transactional workloads.