The Power of NVMe Over TCP
What is NVMe over TCP? NVMe over TCP—an evolutionary networking protocol that extends the ultra-fast, parallel performance of local NVMe storage across standard Ethernet fabrics using standard TCP/IP.
How does NVMe/TCP work compared to iSCSI? For years, iSCSI served as the enterprise go-to for IP-based storage, but its legacy serial architecture often introduces significant CPU overhead and latency bottlenecks in modern data infrastructure. In contrast, NVMe over TCP bypasses these inefficiencies, delivering near-local flash performance over your existing network infrastructure without requiring costly, specialized hardware.
As data centers rapidly shift toward microservices and containerized application deployment, NVMe over TCP has become foundational for cloud-native infrastructure. By matching the elastic scalability and high throughput of modern distributed environments, NVMe over TCP ensures that your storage layer never becomes the bottleneck for next-generation cloud applications.
Why is NVMe/TCP important for cloud-native infrastructure?
“By 2027, 25 percent of enterprise organizations will deploy NVMe-oF as a storage network protocol, up from less than 10 percent in mid-2023.”
“NVMe-over-TCP makes networked NVMe storage easy to deploy at scale, while delivering performance similar to local flash.”
“NVMe/TCP allows organizations to provision scalable storage without having to change their network architecture fundamentally, and provides latencies akin to that provided from conventional direct-attached storage.”
“As network bandwidth advances and Ethernet technology innovations continue, organizations will move away from the constrained limitations of traditional fibre channel SAN to more flexible and scalable network based storage, with NVMe/TCP a clear frontrunner.”
What are common use cases for NVMe/TCP?
Large-Scale Kubernetes & Cloud-Native Environments
NVMe/TCP is ideal for Kubernetes because it provides the massive parallel queue depths needed to handle simultaneous I/O requests from thousands of containers. It allows teams to build high-performance, scale-out storage fabrics using the standard IP networks already running their cloud-native infrastructure.
High-Performance Databases
Traditional relational databases (like PostgreSQL and MySQL) and NoSQL databases require low latency and high IOPS. NVMe/TCP eliminates the CPU serialization bottlenecks inherent in older protocols like iSCSI. This allows transactional databases to process more queries per second and enables analytics platforms to ingest and parse massive datasets much faster.
AI Data Pipelines
AI workloads require streaming massive datasets to GPU-accelerated servers. If the storage layer can’t feed data fast enough, expensive GPUs sit idle. NVMe/TCP provides the high-throughput, low-latency data pipeline required for AI inference, allowing you to maximize your investments in compute hardware without investing in complex, proprietary storage networking.
SAN Modernization
For those with a datacenter modernization initiative, NVMe/TCP allows you to transition to a modern NVMe SAN using your Ethernet. This avoids the massive CapEx of buying specialized FC HBAs, while still delivering a massive leap in storage performance for hypervisors like VMware vSphere or KVM.
Edge Computing and CDNs
At the edge, physical space, power, and specialized IT expertise are often limited. Deploying complex storage networks like FC or RDMA-enabled Ethernet (RoCE) at dozens of remote edge sites is impractical. Because NVMe/TCP operates over standard, ubiquitous TCP/IP, it enables you to deploy ultra-fast block storage at edge data centers, regional hubs, and CDN nodes using standard, off-the-shelf networking hardware.
How Does Lightbits Deliver High-Performance SDS Using NVMe/TCP?
Lightbits LightOS delivers high-performance software-defined storage (SDS) by using a NVMe/TCP-direct path that eliminates the translation overhead found in legacy protocols. By disaggregating storage from compute over standard TCP/IP, LightOS achieves up to 75M IOPS with sub-millisecond consistent latency, requiring no specialized hardware or proprietary networking.
- Protocol Native Efficiency: As the inventors of NVMe/TCP, Lightbits uses a clean-sheet software design that avoids the iSCSI or RDMA translation layers, reducing CPU overhead and latency.
- Storage Disaggregation: By separating compute and storage resources, Lightbits allows each to scale independently, ensuring storage I/O never bottlenecks application performance.
- Intelligent Flash Management (IFM): Optimizes data placement and reduces garbage collection, extending SSD endurance by 20x while maintaining high throughput.
- Clustered Architecture: A self-healing, scale-out design distributes workloads across multiple nodes, providing linear performance scaling and HA without the “vSAN-like tax” of high host CPU consumption.
How Does NVMe/TCP Work?
NVMe/TCP is typically implemented to modernize legacy SANs, iSCSI, and Fiber Channel though it’s specifically adapted for NVMe-based storage performance. It is a ubiquitous standard for storage disaggregation and a component of the broader NVMe-oF standard. It extends the NVMe standard over TCP networks. The industry has widely accepted that this new NVMe-oF model will replace Direct-Attached Storage (DAS) and become the default protocol for disaggregated storage in cloud infrastructure.
| Lightbits LightOS with NVMe/TCP | Legacy SDS (e.g., Ceph/iSCSI) | |
| Performance | Up to 16x faster | Significant translation overhead |
| Latency | Consistent sub-millisecond | Variable/Higher tail latency |
| Hardware | Standard Ethernet/x86 | Often requires specialized NICs |
What are the benefits of NVMe/TCP for modern storage?
Parallelism of NVMe over TCP
Built on top of the TCP/IP software stack, NVMe®/TCP enables efficient and streamlined block storage, optimized for today’s multi-core application servers.
Server Parallelism
Multi- Core CPUs and Multiple NVMe I/O Queues
Network Parallelism
Multiple NVMe I/O Queues mapped to Multiple TCP connections
Lightbits Parallelism
Lightbits NVMe/TCP Open Storage Platform