It’s no secret that the storage industry underwent significant transformation in the last decade. But do you know what to attribute it to? NVMe over TCP. If digital transformation is your data center initiative, holding on to legacy iSCSI and its limitations is counterproductive.
To understand why, consider its predecessor: SCSI (Small Computer Systems Interface). For decades, SCSI was the standard protocol for computer and server access to storage. The SCSI standard was used to transfer data blocks over short distances using various serial and parallel buses within a computer. The development and widespread adoption of the Transmission Control Protocol/Internet Protocol (TCP/IP), which enables computer systems to connect to TCP/IP networks like the Internet, led to the creation of iSCSI, effectively merging SCSI with network capabilities.
Despite the rise of flash storage and the eventual decline of spinning disk drives, SCSI and iSCSI initially maintained their position for data transfer. However, it quickly became apparent that iSCSI was becoming a bottleneck. The SCSI protocol involves shuffling data between the client (initiator) and a drive using a single queue of commands. This single-queue system was adequate when computers typically had only one CPU core. Because iSCSI is essentially SCSI over TCP/IP, it inherits this single queue limitation, corresponding to the single queue in the initiator-logical unit connection. Modern computers, both clients and servers, now commonly feature dozens of CPU cores, rendering the single queue model insufficient to handle the demands of high-performance storage.
NVMe Storage Revolution
SCSI’s dominance occurred during an era focused on traditional spinning magnetic disk drives, allowing applications to access these slower drives within computer systems. However, the emergence of flash storage, with solid-state drives (SSDs) replacing mechanical drives, brought about a need for high-performance, low-latency parallel data access. This demand became a primary driver for developing the Non-Volatile Memory Express (NVMe®) specification. Flash storage offers numerous advantages, including significantly lower access latency, higher bandwidth, accelerated application performance, and better overall value due to its smaller footprint and lower energy consumption than traditional drives.
Organizations soon recognized that NVMe storage offered a superior solution to the data storage challenges by providing a way to overcome the limitations of iSCSI. NVMe offers flexible and scalable expansion, essential to meet today’s ever-increasing storage demands. Considering the rapid rise of online retail and the needs of e-commerce storage, the massive amounts of data that need to be analyzed illustrate the critical importance of high-speed access to stored data. NVMe enables the rapid retrieval of data, delivering the necessary information precisely when needed.
NVMe over Fabrics (NVMe-oF) and the NVMe over TCP
NVMe has become the recognized protocol for accessing high-performance solid-state drives (SSDs). While initially designed for interfacing with storage devices via PCI Express (PCIe), the NVMe industry group developed the NVMe over Fabrics (NVMe-oF™) specification to extend NVMe across network fabrics. The initial NVMe-oF specification defined Fibre Channel and Ethernet using RDMA as network transports. However, these transports were often limited to smaller deployments and required specialized data center equipment, hindering widespread adoption.
This is where NVMe over TCP (NVMe/TCP) comes in. Like how iSCSI enabled SCSI over standard Ethernet and TCP/IP, NVMe/TCP provides the same capability for NVMe. NVMe/TCP is now positioned to replace iSCSI as the primary communication standard between compute and storage servers, becoming the preferred protocol for disaggregated storage and compute architectures.
NVMe/TCP: A Game-Changer for [Cloud, Platform] Service Providers
NVMe/TCP is game-changing for cloud (CSPs) and platform service provider [a service provider that offers a technology-based platform that others can use to build upon, conduct business, or access specific functionalities]. For example, RPC Node providers [services that offer access to blockchain nodes via RPC] would be categorized as such. Its advantages align perfectly with the scalability, performance, and efficiency demands of hyperscale cloud environments, enabling a cloud operations model for on-premises implementations.
- Scalability and Flexibility: Cloud environments are designed to scale rapidly and efficiently. NVMe/TCP’s ability to leverage ubiquitous TCP/IP networks allows service providers to easily expand their storage infrastructure without requiring specialized hardware. This agility is crucial for meeting the fluctuating demands of cloud users.
- Performance: NVMe/TCP delivers high performance and low latency, which is essential for cloud applications that demand rapid data access. This translates to improved application responsiveness and a better user experience, a critical factor for service providers to maintain competitiveness.
- Cost-Effectiveness: Using standard Ethernet networks, NVMe/TCP helps service providers reduce CapEx and OpEx. It eliminates the need for expensive, specialized network infrastructure, simplifies management and lowers the total cost of ownership. These cost savings translate to higher margins for the service providers and cloud cost savings that they can pass on to their customers.
- Disaggregation: NVMe/TCP facilitates the disaggregation of compute and storage resources, enabling service providers to scale and manage these resources independently. This leads to greater resource utilization, increased efficiency, and more flexible service offerings.
- Efficiency: NVMe/TCP’s efficient block storage software stack, optimized for NVMe, maximizes performance within existing data center infrastructures. This efficiency translates to higher storage density and reduced power consumption, contributing to a greener more sustainable cloud infrastructure.
The Future is NVMe/TCP
NVMe/TCP is ready to address the evolving challenges of modern storage. Building a large-scale cloud storage infrastructure with NVMe/TCP is efficient and straightforward. TCP is ubiquitous, scalable, reliable, and well-suited for short-lived connections and containerized applications, which are fundamental to cloud-native architectures. Migrating to disaggregated flash storage with NVMe/TCP is seamless because it doesn’t require any changes to the existing data center network infrastructure, as most data center networks are already designed to support TCP/IP.
In conclusion, iSCSI has reached the limits of its utility in the data center. NVMe/TCP is the ideal solution for today’s modern infrastructure and cloud environments. By extending NVMe across the data center using a standard and efficient TCP/IP fabric, users, especially cloud service providers, can realize a multitude of benefits:
- Low average and tail latencies
- Utilization of existing networking infrastructure and application servers, eliminating the need for costly rip-and-replace upgrades
- Performance and latency comparable to Direct-Attached SSDs (DAS)
- Efficient block storage software stack optimized for NVMe and existing data centers
- Parallel access to storage optimized for today’s multi-core applications and client servers
- Standards-based solution with broad industry support
- Disaggregation of compute and storage across a data center’s availability zones and regions
If you are interested to learn more about NVMe storage, read the blog: NVMe Storage Explained: NVMe Shared Storage, NVMe-oF, and More
