The advent of all-flash Solid-State Drives (SSDs) has transformed the storage function in enterprise and cloud computing. SSDs offer greatly improved storage performance compared to traditional spinning hard disk drives (HDDs). It is therefore not surprising that 18% of businesses are now using all-flash storage, with industry research predicting that a further 14% will have adopted it by 2022. The all-flash storage market is growing briskly at 11% per year. The adoption of compute acceleration technologies, more modern applications, and faster storage media is leading IT infrastructure architects to confront the challenge of networking–getting the data from the storage media to the application that needs it.
Most infrastructure now in place relies on the Internet Small Computer Systems Interface (iSCSI) standard–the networked version of the bus SCSI protocol–to link storage with servers. iSCSI carries SCSI commands over a TCP/IP network in order to deliver block-level access to storage devices. The difficulty is that iSCSI is a serial technology. It is therefore far too slow to accommodate the massively parallel nature of solid-state storage drives. Non-Volatile Memory (NVM) Express, or NVMe, is a much better solution for this workload.
However, NVMe, too, has drawbacks. For example, NVMe devices are great in that they deliver performance, but can suffer from the local “puddling” of storage resources. The innovative NVMe over Fabrics (NVME-oF) protocol will work well in theory, but in reality, the various implementations: Fiber Channel (FC), RDMA over Converged Ethernet (RoCE), InfiniBand, iWARP, TCP/IP are not all created equal in terms of ease of use and cost. Thus it makes it a difficult option to select. In contrast, NVMe over TCP/IP is superior in all respects: performance, ease of use, and cost-efficiency because it utilizes standard TCP/IP networking.
iSCSI Overview
The IT hardware industry has been addressing the requirement to connect storage devices with computers for many decades. Starting in the 1990s, companies like Apple and Microsoft worked together to publish the original SCSI X3.131-1986 standard, known as “SCSI-1” with the X3T9 technical committee of the American National Standards Institute (ANSI). In this effort, the industry benefited from the American slang term “scuzzy,” which connotes sloppy, unattractive behavior as an ironic counterpoint to the technology’s slick capabilities.
Based on a shared bus transport model, SCSI comprises a set of standards for the physical connection and transfer of data between peripheral devices and computers. It defines protocols, commands, and interfaces for almost any kind of peripheral, but in practice, it was mostly used for hard disk drives.
Computers and storage technologies proceeded to grow larger and faster over time. With the resulting increases in bandwidth and distance requirements, the industry developed several new serial SCSI transports, including:
- Fibre Channel (FC)—a high-speed data transfer protocol that provides lossless delivery of raw block data. FC is mostly used in the connection of storage to servers in storage area networks (SANs). An FC network is a “switched fabric,” so-named due to its having switches in a network operate in unison, effectively as “one big switch.” FC networks usually run on fiber optic cables.
- Serial Attached SCSI (SAS)—a point-to-point serial protocol that moves data in and out of storage devices like HDDs or tape drives. It uses the standard SCSI command set.
- SCSI over TCP/IP (iSCSI)—a transport layer protocol that works over the Transport Control Protocol (TCP). iSCSI makes it possible to transport block-level SCSI data between an iSCSI “initiator” and a storage device using common TCP/IP network connections.
What is iSCSI?
SCSI – literally Small Computer Systems Interface. This protocol has been the de facto standard for computer and server access to storage for decades. The SCSI set of standards is used to carry blocks of data over short distances using a variety of serial and parallel busses internal to a computer. The widespread use of the Transmission Control Protocol/Internet Protocol (TCP/IP), which connects computer systems to networks such as the Internet, brought SCSI and the Internet together, creating iSCSI.
To understand iSCSI, you first need to understand the basic SCSI protocol. SCSI is a block-based set of commands designed for Direct-Attached Storage (DAS) use cases. Through SCSI commands, a computer can instruct storage media to spin up and execute data reads/writes. With SCSI, the client is known as the “initiator.” The storage volume it accesses is called the “target.”
Access between an initiator and target using the SCSI protocol comprises a queue of commands. The initiator sends commands to be performed by the target in a queue. The initiator then receives data and acknowledgments for each of its queued commands. There is a single queue for each connection between an initiator and target.
SCSI vs. iSCSI
iSCSI is SCSI transport over TCP/IP and designed for Network Attached Storage (NAS) use cases. With iSCSI, therefore, a connection between the initiator and target consists of a single TCP/IP socket opened by a single thread. Because iSCSI is an update of the old SCSI, the ubiquitous storage protocol, computer operating systems (OSs) implemented the same model in their storage stacks. iSCSI represents a situation where access to a storage volume is always mediated by a single queue, which corresponds to the single queue in the connection between the initiator and target.
iSCSI Performance
iSCSI has become the common standard for block storage over standard networks because TCP/IP is widely adopted in those networks. This is not optimal from a performance perspective, however. The issue is that applications that need low latency now use SSDs, and iSCSI is not designed to work well with them. Rather, TCP/IP protocols are simply transports for the same original SCSI protocol developed for spinning hard drives.
This single queue model was not a problem for many years. HDDs were well suited to the single queue because most hard disk drives had a single read/write head. Flash memory behaves quite differently. It does not utilize a “single head” that serializes data access. Instead, flash memory is arranged as banks of storage elements, similar to how dynamic random access memory (DRAM) works.
Servers have also changed, further affecting the efficacy of the single-queue iSCSI model. A typical server today may have dozens of CPU cores. As a result, storing data from several cores in parallel requires locks on the single storage queue. This is not a big problem for spinning HDDs, which perform at only about 100 IOPs. A typical enterprise storage system built with HDDs might perform at several thousand IOPs per volume. When storage migrates to flash, this all changes dramatically.
The single queue model is problematic with flash SSDs. A single SSD is capable of hundreds of thousands of IOPs. The resulting queue contention translates into a serious performance issue for a server with numerous cores. The iSCSI single-queue model creates further difficulties with the standard implementation of a single thread and single TCP socket.
It is possible to perform multipath I/O by surfacing a logical unit via several target ports and load balancing I/O over multiple threads and sockets. But this strategy is complicated to manage on the target side. It invariably suffers from queueing issues on the initiator side. Multiple paths are still treated as a single device by the operating system. Performance will inevitably lag.
The NVMe solution and its limitations
Getting better performance out of SSDs required a new communication protocol. Over ninety companies collaborated through the NVM Express Workgroup. Intel was one of the leading firms represented in the effort to develop the resulting open, logical-device interface specification that enables a computer to access non-volatile (i.e., flash) storage media. The typical implementation involved a machine’s PCI Express (PCIe) bus.
NVM Express®, or NVMe®, is designed to take advantage of SSDs’ internal parallelism and inherent low latency. NVMe lets its host hardware and software exploit parallelism to a great degree. As a result, NVMe reduces I/O overhead, improving performance compared to iSCSI.
NVMe has been transformative, emerging as the new standard protocol for accessing high-performing SSDs. The initial limitation, however, was that NVMe was designed for direct-attached PCIe SSDs. The specification could not work for networked storage. To address this limitation, the NVMe working group developed NVMe-oF®, which supports a rack-scale remote pool of SSDs over network fabrics.
The IT industry has generally accepted that NVMe-oF will replace iSCSI as the preferred communication method between compute servers and storage servers. It will become the default protocol for disaggregating storage and compute servers.
However, NVMe-oF has limitations. Initial deployment options for NVMe-oF were limited to Ethernet over Remote Direct Memory Access (RDMA) and Fibre Channel fabrics. These are only suitable for small-scale deployments. Indeed, most data centers do not currently support NVMe over Fabrics using technologies such as RDMA or Fibre Channel.
Issues with RDMA or Fibre Channel include:
- Requiring specialized, not commonly used equipment for data centers
- Being generally out of the mainstream
- Requiring significant changes in the data center’s network infrastructure
- Locking the customer to a single vendor
Introducing NVMe/TCP
Given the potential, but also the limitations of NVMe-oF, it made sense for the industry to find a way to take advantage of the performance characteristics of NVMe, but do so in a way that leveraged the predominant TCP/IP network transport protocol. This led to NVMe/TCP. Lightbits Labs, working in collaboration with Facebook, Intel, and other industry leaders, is extending the NVMe-oF standard to support TCP/IP.
This approach complements RDMA fabrics, with NVMe/TCP disaggregation offering the advantage of being simple and highly efficient. TCP is ubiquitous, scalable, reliable—ideal for short-lived connections and container-based applications. Additionally, migrating to shared flash storage with NVMe/TCP does not require changes to the data center network infrastructure. No infrastructure changes should make deployment across the data center relatively easy. After all, nearly all data center networks are designed to carry TCP/IP.
Furthermore, the broad industry collaboration now taking place on the NVMe/TCP protocol makes it likely that NVMe/TCP will become a broad ecosystem that supports a wide range of operating systems and NICs. For instance, the NVMe/TCP Linux drivers are a natural match for the Linux kernel. They use the standard Linux networking stack and NICs without any modifications. As a result, NVMe is showing great promise as a new protocol. It is suitable for hyperscale data centers, among other use cases, because it is easy to deploy with no changes to the underlying network infrastructure.
The NVMe specification supports highly parallel access to solid-state storage. There is no single bottleneck. Thus, I/O to an NVMe “namespace,” the equivalent of the SCSI logical unit, can be performed via 64,000 queues with up to 64,000 commands per queue. This parallel access is a critical architectural development because multiple queues eliminate the contention inherent in the SCSI model.
iSCSI vs. NVMe/TCP
Comparing NVMe/TCP to iSCSI, the following benefits emerge:
- Extends NVMe across the entire data center using a simple and efficient TCP/IP fabric
- Enables disaggregation across a data center’s availability zones and regions
- Leverages TCP/IP transport to bring low average and tail latencies to a highly parallel NVMe software stack
- Requires no changes to the networking infrastructure or application servers
- Delivers a high-performance NVMe-oF solution that has the same performance and latency as Direct Attached SSDs (DAS)
- Uses an efficient and streamlined block storage software stack optimized for NVMe and existing data centers
- Provides parallel access to storage optimized for today’s multi-core application/client servers
- Supports a standards-based solution that will have industry-wide support
Comparing performance of NVMe/TCP with iSCSI
How does the NVMe/TCP stack compare to iSCSI in terms of performance? Lightbits conducted a side-by-side comparison of the two technologies to arrive at benchmark results. The test used the industry-standard Flexible I/O Tester (FIO) benchmark tool with identical setups for iSCSI and NVMe/TCP. Specifically, the test used an initiator with a dual-socket Intel® Xeon® E5-2648L v4 processor running at 1.8 GHz, with 14 cores per socket and two hyperthreads per core. The target machine had the same socket and core count. It ran a Xeon E5-2650L v4 processors running at 1.7 GHz. Each target used Intel DC P3520 SSDs (450 GB).
The test performed all I/O via Mellanox ConnectX-4 100GbE NICs on both machines. Both initiator and target machines ran Linux 4.13.0-rc1 kernel release. For iSCSI, we used the stock iSCSI initiator in the kernel. For NVMe/TCP, the test used an open-source initiator contributed by Lightbits. For iSCSI storage, we configured a 12-drive RAID-0 group; for Lightbits NVMe/TCP, the exported namespace used the same hardware and the same 12 SSDs.
The benchmark demonstrated a typical multithreaded I/O-bound application. Such applications often open a thread per task, with each thread performing a single I/O (i.e., multiple threads, each with I/O queue depth of 1) at a time. Then, the number of threads was varied to see the I/O scalability as the number of threads increased. Varying the number of threads was a critical test for applications that run on today’s modern servers, which have dozens of CPU cores. The I/O mix was 70% reads and 30% writes.
Before running the benchmarks, we fully overwrote all SSDs to precondition the tests. For iSCSI, the maximum commands per session were set to 2048, and the logical unit queue depth at 1024. Both parameters were the implementation maximums. With these settings, the test reflected a reasonable comparison of the two TCP-based protocols with identical hardware and software configurations.
The results: iSCSI reached a peak of 89K IOPS when running 128 threads over two multipathed target connections in active-active mode. This IOPS amount was the maximum I/O performance the application could achieve. Multipathing over two target connections provided minimal benefit compared to a single connection.
It is also quite troubling that above 64 threads, iSCSI incurred average latencies of over a millisecond. This I/O performance is unacceptable for flash drives in today’s massively multi-core hyperscale cloud services. These findings illustrate the inherent performance problem in iSCSI. The protocol’s single-queue, single-threaded model limits scalability.
The NVMe/TCP test result showed the benefits of the natively multithreaded architecture. I/Os for a single volume (NVMe namespace) easily rose to 725K IOPS in this application scenario. This is an order of magnitude higher IOPS for the application. Moreover, at 256 threads, the initiator did not saturate. Importantly, NVMe/TCP I/O latencies were much lower than iSCSI latencies at a similar thread count and much lower IOPS. Even at 256 threads and 725K IOPS, NVMe/TCP latency was still under 500μs.
Why NVMe/TCP is the Right Choice for Cloud Application Acceleration
SSD growth is creating a performance conflict. The iSCSI single-threaded model is not optimal for connecting compute servers with network-attached SSDs. Instead, NVMe delivers far superior performance characteristics. That said, some NVMe-oF solutions have distinct limitations because they require specialized components and so forth. NVMe/TCP offers a solution. It enables NVMe over the standard, common TCP/IP network transport protocol. In side-by-side tests, NVMe/TCP vs. iSCSI NVMe is clearly resolved. NVMe/TCP enables IOP performance an order of magnitude higher than what is possible with iSCSI.
Additional Resources
Direct Attached Storage (DAS) Disadvantages & Alternatives
Cloud-Native Storage for Kubernetes
Disaggregated Storage
Ceph Storage
Persistent Storage
Kubernetes Storage
Edge Cloud Storage
NVMe® over TCP
