The Power of NVMe Over TCP
What is NVMe over TCP? NVMe over TCP—an evolutionary networking protocol that extends the ultra-fast, parallel performance of local NVMe storage across standard Ethernet fabrics using standard TCP/IP.
How does NVMe/TCP work compared to iSCSI? For years, iSCSI served as the enterprise go-to for IP-based storage, but its legacy serial architecture often introduces significant CPU overhead and latency bottlenecks in modern data infrastructure. In contrast, NVMe over TCP bypasses these inefficiencies, delivering near-local flash performance over your existing network infrastructure without requiring costly, specialized hardware.
As data centers rapidly shift toward microservices and containerized application deployment, NVMe over TCP has become foundational for cloud-native infrastructure. By matching the elastic scalability and high throughput of modern distributed environments, NVMe over TCP ensures that your storage layer never becomes the bottleneck for next-generation cloud applications.
Why is NVMe/TCP important for cloud-native infrastructure?
“By 2027, 25 percent of enterprise organizations will deploy NVMe-oF as a storage network protocol, up from less than 10 percent in mid-2023.”
“NVMe-over-TCP makes networked NVMe storage easy to deploy at scale, while delivering performance similar to local flash.”
“NVMe/TCP allows organizations to provision scalable storage without having to change their network architecture fundamentally, and provides latencies akin to that provided from conventional direct-attached storage.”
“As network bandwidth advances and Ethernet technology innovations continue, organizations will move away from the constrained limitations of traditional fibre channel SAN to more flexible and scalable network based storage, with NVMe/TCP a clear frontrunner.”
What are common use cases for NVMe/TCP?
Large-Scale Kubernetes & Cloud-Native Environments
NVMe/TCP is ideal for Kubernetes because it provides the massive parallel queue depths needed to handle simultaneous I/O requests from thousands of containers. It allows teams to build high-performance, scale-out storage fabrics using the standard IP networks already running their cloud-native infrastructure.
High-Performance Databases
Traditional relational databases (like PostgreSQL and MySQL) and NoSQL databases require low latency and high IOPS. NVMe/TCP eliminates the CPU serialization bottlenecks inherent in older protocols like iSCSI. This allows transactional databases to process more queries per second and enables analytics platforms to ingest and parse massive datasets much faster.
AI Data Pipelines
AI workloads require streaming massive datasets to GPU-accelerated servers. If the storage layer can’t feed data fast enough, expensive GPUs sit idle. NVMe/TCP provides the high-throughput, low-latency data pipeline required for AI inference, allowing you to maximize your investments in compute hardware without investing in complex, proprietary storage networking.
SAN Modernization
For those with a datacenter modernization initiative, NVMe/TCP allows you to transition to a modern NVMe SAN using your Ethernet. This avoids the massive CapEx of buying specialized FC HBAs, while still delivering a massive leap in storage performance for hypervisors like VMware vSphere or KVM.
Edge Computing and CDNs
At the edge, physical space, power, and specialized IT expertise are often limited. Deploying complex storage networks like FC or RDMA-enabled Ethernet (RoCE) at dozens of remote edge sites is impractical. Because NVMe/TCP operates over standard, ubiquitous TCP/IP, it enables you to deploy ultra-fast block storage at edge data centers, regional hubs, and CDN nodes using standard, off-the-shelf networking hardware.
How Does Lightbits Deliver High-Performance SDS Using NVMe/TCP?
Lightbits SDS delivers high-performance software-defined storage (SDS) by using a NVMe/TCP-direct path that eliminates the translation overhead found in legacy protocols. By disaggregating storage from compute over standard TCP/IP, Lightbits SDS achieves up to 75M IOPS with sub-millisecond consistent latency, requiring no specialized hardware or proprietary networking.
- Protocol Native Efficiency: As the inventors of NVMe/TCP, Lightbits uses a clean-sheet software design that avoids the iSCSI or RDMA translation layers, reducing CPU overhead and latency.
- Storage Disaggregation: By separating compute and storage resources, Lightbits allows each to scale independently, ensuring storage I/O never bottlenecks application performance.
- Intelligent Flash Management (IFM): Optimizes data placement and reduces garbage collection, extending SSD endurance by 20x while maintaining high throughput.
- Clustered Architecture: A self-healing, scale-out design distributes workloads across multiple nodes, providing linear performance scaling and HA without the “vSAN-like tax” of high host CPU consumption.
How Does NVMe/TCP Work?
NVMe/TCP is typically implemented to modernize legacy SANs, iSCSI, and Fiber Channel though it’s specifically adapted for NVMe-based storage performance. It is a ubiquitous standard for storage disaggregation and a component of the broader NVMe-oF standard. It extends the NVMe standard over TCP networks. The industry has widely accepted that this new NVMe-oF model will replace Direct-Attached Storage (DAS) and become the default protocol for disaggregated storage in cloud infrastructure.
| Lightbits SDS with NVMe/TCP | Legacy SDS (e.g., Ceph/iSCSI) | |
| Performance | Up to 16x faster | Significant translation overhead |
| Latency | Consistent sub-millisecond | Variable/Higher tail latency |
| Hardware | Standard Ethernet/x86 | Often requires specialized NICs |
What are the Benefits of NVMe/TCP?
NVMe/TCP for High-Performance Database Workloads
Traditional relational databases such as PostgreSQL, MySQL, and Oracle, and high-concurrency NoSQL systems such as MongoDB, demand ultra-low latency and massive I/O IOPS to maintain strict transactional integrity and rapid query response times. Legacy storage protocols like iSCSI were designed for spinning disks, introducing severe CPU serialization bottlenecks and SCSI-translation overhead that choke database performance during peak concurrent loads.
NVMe over TCP (NVMe/TCP) eliminates these legacy constraints by leveraging native, end-to-end parallelism. By mapping up to 64,000 I/O queues directly to multi-core CPUs over standard Ethernet connections, NVMe/TCP prevents queue lock contention and allows database servers to process significantly more transactions and queries per second.
Additionally, for real-time analytics platforms, data warehouses, and vector databases used in AI, NVMe/TCP enables the rapid ingestion, indexing, and parsing of massive datasets without compute starvation. Because it allows for complete storage disaggregation, database administrators can scale storage capacity dynamically and independently of database compute nodes. This ensures that mission-critical databases benefit from robust HA, instant failover routing, and consistent, sub-millisecond tail latencies that rival direct-attached local flash—all running on standard TCP/IP network infrastructure.
Parallelism of NVMe over TCP
Built on top of the TCP/IP software stack, NVMe/TCP enables efficient and streamlined block storage, optimized for today’s multi-core application servers.
Server Parallelism
Multi- Core CPUs and Multiple NVMe I/O Queues
Network Parallelism
Multiple NVMe I/O Queues mapped to Multiple TCP connections
Lightbits Parallelism
Lightbits NVMe/TCP Open Storage Platform
Enterprise Adoption Scenarios
SAN Modernization and iSCSI Replacement
SAN Modernization and iSCSI Replacement
Organizations replace legacy FC fabrics and aging iSCSI storage arrays with NVMe/TCP to cut costs and simplify data center operations. Running directly over standard 25/100Gb Ethernet switches and commodity NICs, NVMe/TCP eliminates the need for expensive HBAs or dedicated optical cabling. It drastically reduces latency and reclaims CPU cycles previously wasted on legacy SCSI translation.
Kubernetes and Cloud-Native Infrastructure
Kubernetes and Cloud-Native Infrastructure
DevOps and platform teams deploy NVMe/TCP via CSI drivers to provide high-performance block storage for stateful container workloads. By disaggregating storage from compute, Kubernetes environments can scale compute nodes and storage arrays independently across routable IP networks. This supplies persistent volumes with local-NVMe speeds while preserving container agility and multi-tenant scaling.
AI/ML Data Pipelines and High-Performance Databases
AI/ML Data Pipelines and High-Performance Databases
Organizations leverage NVMe/TCP to feed high-throughput workloads like GPU-driven AI training, real-time analytics, and high-concurrency transactional databases. NVMe/TCP’s native support for up to 64,000 parallel I/O queues prevents I/O bottlenecks and GPU starvation, delivering the sub-millisecond tail latencies required for large-scale data ingestion and rapid checkpointing.
NVMe over TCP – Frequently Asked Questions
Is NVMe over TCP faster than iSCSI?
Yes, NVMe over TCP is substantially faster, delivering lower latency and higher IOPS than iSCSI. Because iSCSI relies on legacy SCSI protocols engineered for spinning disks, it suffers from command queue bottlenecks and high CPU translation overhead. In contrast, NVMe over TCP was designed natively for SSDs, supporting up to 64,000 parallel queues. This architecture drastically reduces protocol overhead and eliminates host-side processing bottlenecks, allowing NVMe over TCP to achieve near-line-rate throughput across the standard Ethernet networks that you already have.
What is the difference between NVMe/TCP and FC?
NVMe/TCP runs native NVMe commands over standard, ubiquitous Ethernet networks, whereas Fibre Channel (FC) relies on specialized, high-cost SAN hardware and dedicated protocols.
-
Choose NVMe/TCP to achieve high-performance, modern flash storage performance using existing Ethernet infrastructure without the capital expense of a dedicated SAN.
-
Choose FC if you have existing, legacy FC SAN investments or require strictly deterministic, hardware-enforced zero-packet-loss storage networks.
Why are enterprises adopting NVMe over TCP?
Enterprises are adopting Lightbits’ NVMe over TCP (NVMe/TCP) SDS to deliver local flash performance and sub-millisecond latency over standard Ethernet without requiring expensive FC hardware or complex RDMA networking. As inventors of the NVMe/TCP standard, Lightbits couples software-based storage disaggregation with proprietary Intelligent Flash Management—which extends SSD endurance up to 20x—allowing organizations to slash CapEx and OpEx, reclaim host CPU cycles, and seamlessly scale high-throughput block storage for Kubernetes high-performance databases.
Is NVMe/TCP suitable for Kubernetes environments?
NVMe/TCP is exceptionally well-suited for Kubernetes environments, delivering low-latency, disaggregated block storage over standard Ethernet infrastructure without requiring specialized hardware. Integrated natively via CSI drivers, it enables rapid, automated provisioning of high-performance PVs for demanding stateful workloads like databases. This allows containerized applications to achieve local-flash speeds and seamless multi-tenant scaling while fully preserving the operational flexibility, resiliency, and elasticity required by modern cloud-native architectures.
Can NVMe over TCP support AI workloads?
NVMe over TCP easily supports demanding AI workloads by delivering the ultra-high throughput, massive parallel I/O, and sub-millisecond tail latencies required to prevent GPU starvation during deep learning training and inference. Running over standard 100Gb+ Ethernet infrastructure, it allows organizations to scale disaggregated pools of fast flash storage to seamlessly feed massive training datasets, vector databases, and high-frequency pipeline checkpoints to distributed AI compute nodes without the operational complexity or high cost of dedicated RDMA network fabrics.
What are the latency advantages of NVMe/TCP?
NVMe over TCP dramatically reduces I/O latency compared to legacy storage protocols like iSCSI by eliminating SCSI-to-NVMe translation overhead and leveraging native, end-to-end NVMe processing. By supporting up to 64,000 parallel I/O queues mapped directly to CPU cores, it eliminates command queue lock contention during heavy, concurrent operations. This streamlined architecture delivers consistent, sub-millisecond tail latencies—adding as little as tens of microseconds of network overhead compared to local NVMe SSDs—while running entirely over standard Ethernet infrastructure.
Is NVMe over TCP cost-effective for enterprises?
NVMe over TCP is highly cost-effective for enterprises because it delivers enterprise-grade NVMe flash performance over existing, standard Ethernet infrastructure. By running on commodity NICs and standard IP switches, it eliminates the need for expensive FC HBAs, proprietary SAN hardware, or the complex, specialized switches required by RDMA-based protocols like RoCE. This dramatically slashes CapEx while lowering OpEx, as storage teams can manage traffic using familiar TCP/IP tools and skill sets. Additionally, disaggregating flash storage into scalable, centralized pools prevents stranded drive capacity across servers, maximizing hardware utilization and ROI.
How scalable is NVMe over TCP storage?
NVMe over TCP is exceptionally scalable, enabling organizations to scale storage capacity and compute resources independently from terabytes to exabytes across standard Ethernet networks. Unlike loss-sensitive protocols like NVMe/RoCE that are bounded by single lossless network domains, NVMe/TCP leverages standard TCP/IP routing to seamlessly span across subnets, racks, and geographically distributed data centers without distance limitations. Supported by massive native parallelism—up to 64,000 I/O queues with 64,000 commands each—it effortlessly supports thousands of concurrent compute nodes and multi-tenant workloads without suffering performance bottlenecks.
Does NVMe/TCP require specialized hardware?
No, NVMe over TCP does not require specialized hardware, operating entirely on standard NICs and off-the-shelf Ethernet switches. Unlike FC, which demands dedicated HBAs and specialized SAN switches, or NVMe over RoCE, which requires specialized RNICs and complex, lossless network configurations (such as PFC and ECN), NVMe/TCP leverages universal TCP/IP networking. This allows organizations to deploy high-performance, disaggregated block storage across existing network infrastructure using native drivers built directly into modern Linux kernels, hypervisors, and operating systems.
What workloads benefit most from NVMe over TCP?
Workloads that benefit most from NVMe over TCP are I/O-intensive, performance-critical applications requiring sub-millisecond latency and high IOPS over scalable Ethernet infrastructure. Key beneficiaries include enterprise databases (both transactional like PostgreSQL and NoSQL like MongoDB or Cassandra) and real-time analytics engines. Additionally, containerized stateful applications in Kubernetes, large-scale virtualized server environments, and high-frequency financial platforms gain disaggregated, local-flash performance and flexible multi-tenant scaling without the high costs or complexity of specialized SAN hardware.
How secure is NVMe over TCP?
NVMe over TCP provides robust enterprise security by combining native NVMe-oF authentication and encryption standards with established IP network defense mechanisms. It supports DH-HMAC-CHAP for mutual host-target verification and TLS 1.3 for end-to-end data-in-transit encryption. Because it operates on standard TCP/IP infrastructure, cybersecurity teams can easily restrict access using NVMe Qualified Name (NQN) mapping while isolating storage traffic with familiar tools like VLANs, IPsec, and firewall policies.
What are best practices for deploying NVMe/TCP?
Deploying NVMe/TCP successfully requires isolating storage traffic on dedicated high-bandwidth VLANs, enabling Jumbo Frames (MTU 9000), and leveraging TCP Segmentation Offload on modern NICs to minimize CPU overhead. Hosts should implement NVMe Asymmetric Namespace Access for failover multipathing and pin I/O queues to CPU cores to eliminate lock contention during peak traffic. Finally, security and operational stability should be enforced by configuring DH-HMAC-CHAP host-target authentication, applying TLS 1.3 encryption for data in transit, and restricting access using precise NVMe Qualified Name zoning.