An abstract purple background featuring white light rays
An abstract purple background featuring white light rays

NVMe over TCP

Invented by Lightbits Labs, it’s the protocol of the future for high-performance data access on low latency networking.

Performance and Cost-Efficiency on Standard Ethernet

“By 2027, 25 percent of enterprise organizations will deploy NVMe-oF as a storage network protocol, up from less than 10 percent in mid-2023.”

“NVMe-over-TCP makes networked NVMe storage easy to deploy at scale, while delivering performance similar to local flash.”

“NVMe/TCP allows organizations to provision scalable storage without having to change their network architecture fundamentally, and provides latencies akin to that provided from conventional direct-attached storage.”

“As network bandwidth advances and Ethernet technology innovations continue, organizations will move away from the constrained limitations of traditional fibre channel SAN to more flexible and scalable network based storage, with NVMe/TCP a clear frontrunner.”

How Does Lightbits Deliver High-Performance SDS Using NVMe/TCP?

Lightbits delivers high-performance software-defined storage (SDS) by using a NVMe/TCP-direct path that eliminates the translation overhead found in legacy protocols. By disaggregating storage from compute over standard TCP/IP, Lightbits achieves up to 75M IOPS with sub-millisecond consistent latency, requiring no specialized hardware or proprietary networking.

  • Protocol Native Efficiency: As the inventors of NVMe/TCP, Lightbits uses a clean-sheet software design that avoids the iSCSI or RDMA translation layers, reducing CPU overhead and latency.
  • Storage Disaggregation: By separating compute and storage resources, Lightbits allows each to scale independently, ensuring storage I/O never bottlenecks application performance.
  • Intelligent Flash Management (IFM): Optimizes data placement and reduces garbage collection, extending SSD endurance by 20x while maintaining high throughput.
  • Clustered Architecture: A self-healing, scale-out design distributes workloads across multiple nodes, providing linear performance scaling and HA without the “vSAN-like tax” of high host CPU consumption.
NVMe/TCP on Wikipedia

How Does NVMe/TCP Work?

NVMe/TCP is typically implemented to modernize legacy SANs, iSCSI, and Fiber Channel though it’s specifically adapted for NVMe-based storage performance. It is a ubiquitous standard for storage disaggregation and a component of the broader NVMe-oF standard. It extends the NVMe standard over TCP networks. The industry has widely accepted that this new NVMe-oF model will replace Direct-Attached Storage (DAS) and become the default protocol for disaggregated storage in cloud infrastructure.

Lightbits NVMe/TCP Legacy SDS (e.g., Ceph/iSCSI)
Performance Up to 16x faster Significant translation overhead
Latency Consistent sub-millisecond Variable/Higher tail latency
Hardware Standard Ethernet/x86 Often requires specialized NICs
Download Free NVMe/TCP Drivers

Parallelism of NVMe over TCP

Disaggregation for tens of thousands of nodes across entire data center

First flexible connection setup ideal for containers and serverless cloud-native applications

Maximized SSD utilization in the data center

Disaggregated storage with the performance and latency of direct-attached SSDs

Parallelism of NVMe over TCP

Built on top of the TCP/IP software stack, NVMe®/TCP enables efficient and streamlined block storage, optimized for today’s multi-core application servers.

Diagram illustrates how Lightbits utilizes the NVMe/TCP protocol

Server Parallelism

Multi- Core CPUs and Multiple NVMe I/O Queues

Network Parallelism

Multiple NVMe I/O Queues mapped to Multiple TCP connections

Lightbits Parallelism

Lightbits NVMe/TCP Open Storage Platform