The ascent of Scale-Out Flash Storage (SOFS) has fundamentally transformed traditional storage deployments. New storage solutions, particularly those leveraging NVMe flash storage, empower data center teams to achieve optimal capacity, performance, and data service availability for their diverse applications.
What is SOFS and NVMe Flash Storage?
SOFS solutions integrate software packages with servers typically equipped with NVMe® storage devices. Evolving from Software-Defined Storage (SDS), SOFS offers unparalleled scalability, capable of expanding to hundreds or even thousands of nodes. This architecture draws heavily from the Direct-Attached Storage (DAS)-centric models commonly employed by hyperscalers.At its core, SOFS leverages NVMe over Fabrics (NVMe-oF™) as a backbone. This allows SOFS solutions to provide remote file system access across a vast number of servers and storage appliances utilizing Solid State Drives (SSDs), all while maintaining local flash performance and latency. Essentially, SOFS takes NVMe flash storage within software packages and servers, then molds it into a comprehensive network storage solution. By utilizing a network fabric to connect storage and compute nodes, SOFS enables a flexible topology that delivers remote storage performance comparable to that of local storage, with reliability and low latency.
What Are the Leading Use Cases for NVMe Flash Storage Solutions?
The applications for Scale-Out Flash Storage, particularly those that rely on high-performance NVMe flash storage, are diverse and growing. There are many use cases, three that require performance at scale are: AI/Machine Learning, life sciences and genomics, and fraud analysis. Both traditional and scale-out databases, such as distributed open-source options like Cassandra, MongoDB, and CockroachDB, also commonly use this architecture. Furthermore, smaller public cloud providers with specialized services, like High-Performance Computing (HPC), can utilize SOFS to offer customers Amazon EBS (Elastic Block Storage) functionality out of the box. Another significant use case is private cloud for companies seeking a cloud operations model, public cloud capabilities, but with the cost benefits of private cloud, as well as enterprises migrating to Kubernetes or OpenStack. Other less prominent use cases include post-production, analytics for the finance and medical sectors, government applications, and data warehouses.
Advantages Over Array-Based Storage Solutions
As hybrid cloud and private cloud architectures become standard in enterprises, SOFS solutions offer distinct advantages over traditional array-based storage, especially when it comes to leveraging NVMe flash storage. Products that are designed with cloud environments in mind, featuring a share-nothing architecture that enables organizations to scale as needed, offer significant advantages. The separation of data paths and a software-only approach allows for flexible storage application design, offering choices in NVMe drives and networking.Core values include the ability to leverage either private or public cloud platforms, as well as integrations with object storage and orchestration ecosystems. These capabilities amortize the cost of flash, delivering hybrid cloud benefits, and enable snapshots of metadata and data, providing a comprehensive view of the storage system that can be migrated to a public cloud-based storage platform for a pure SaaS offering.The cloud is an undeniable reality for most organizations, and the ability to run virtual machines (VMs) or containers is paramount. Companies are actively seeking storage solutions that standardize block storage, particularly with CSI in Kubernetes, to ensure workload portability across public or private clouds. Lightbits’ software-defined block storage can be run in the cloud with NVMe over TCP. It provides thin provisioning and compression, maximizing the value of provisioned cloud storage, and offers an even better experience on-premises. As companies transition to containers and Kubernetes, a simple plug-in solution and standardized environments will be transformative and more cost-effective.
Best Practices for Migrating to NVMe Storage Solutions
For organizations currently using SAN or NAS storage and considering a migration to an SOFS solution powered by NVMe flash storage, several best practices are crucial.
Identify highly transactional applications that demand low bandwidth, as SOFS may not be the optimal solution for every workload. Migrating to SOFS will necessitate changes to networking processes, particularly with RDMA, making it a more labor-intensive effort that requires teamwork. Organizations should explore different solutions, comparing ease of use and performance, and seek out-of-the-box solutions that utilize standard networking equipment and protocols.
And while cost is essential, it should not be the sole driver of the modernization initiative. Organizations should consider the return on investment (ROI) and the broader impact of the new storage solution on the organization. It’s essential to assess whether new solutions will disrupt existing workflows and to ensure the solution provider offers ample flexibility. As data storage capacity expands, forward-thinking about granular scalability and avoiding vendor lock-in is critical.
I recommend starting with clear business goals and the ultimate objectives you want to achieve. Beginning small and gaining familiarity with the chosen solution, along with determining how to operationalize it, is advisable. When considering the data pipeline, breaking down silos to understand user needs and collaboration with the IT department is vital. It’s important to select a solution that can cater to the entire pipeline and to consider hybrid solutions whenever possible.
The Future is NVMe Flash Storage
Given the numerous benefits offered by SOFS solutions, particularly their integration with NVMe flash storage, they are poised to surpass traditional storage solutions over time. The inherent human desire for ease of use, combined with faster return on investment and cloud operations models for efficiency, suggests that NVMe/TCP is the most potent NVMe-oF technology. It delivers high performance while reducing deployment costs and design complexity, extending NVMe across the entire data center using a simple and efficient TCP/IP fabric.
