Innovation often begins with a proprietary non-standard solution that locks in users to a specific vendor, and eventually is replaced by standardized options that are as good as or better. This cycle is true in everything from name-brand goods versus store brands to computer technology. In the technology field, pioneering proprietary solutions typically address an unmet need in the market until something standard comes along. Sometimes even then they are still seen as better, but in reality users abhor lock in when there is an open, standard choice. They want to be able to move between multiple suppliers and preserve their investments.
Stifled Innovation and Other Risks
Some would argue that the original idea owner pays a price for being first, and that’s probably true. But the reason the market moves away from proprietary solutions is because over time they generally become more difficult and costly for the users to support. Non-standard solutions may require specialized hardware and custom software, and deploying and maintaining them generally involves special expertise. And, as first-to-market, there is no established ecosystem offering interoperability with other vendors – essentially offering users a “take it or leave it” type proposition. There is no alternative, no way to experiment and let innovation and evolution take their natural courses.
In the case of complex technologies, such as those involving both networking and storage, it can be quite difficult to find and retain staff who understand both well enough to keep a non-standard solution in good order. In some cases, the tech expert must be certified and trained by the vendor, which further narrows the talent pool. Not only does that make finding replacement workers difficult for the enterprise, it also limits the options for the workers themselves.
Perhaps even more frightening is that in cases where there is just one vendor holding the patents, the vendor’s demise could mean there is essentially no way to replicate the solution. And guess who’s left holding the bag?
Market Pressures Give Rise to Options
Each of these concerns has consequences that matter immensely to those footing the bill, so it’s not surprising that in time market pressures give rise to new options built around commodity, open standards, and open source technologies. That, in turn, births a growing ecosystem that allows for continuous innovation: better features, better performance, better capabilities, and increasing product maturity. As an example, today, a full 90% of the public cloud workload runs on the Linux open source operating system, according to the Linux Foundation. Moreover, Linux is in more than 60% of the embedded systems world wide, and Linux-based Android overtook Windows as the Internet’s most used operating system in early 2017.
Though the products and how they address the market are likely different, the need filled by more open replacements is generally the same, and often include a slew of other benefits. For enterprises that want to lead in their fields, proprietary solutions may give them a leg up over their competitors initially, but it’s almost inevitable they will be replacing them with more open solutions at some point in the future.
An Example in Storage
At Lightbits, for example, we are strong believers in delighting users through standard solutions. As we saw the rising demand for flash storage, the on-going reduction in prices for flash media, and the growth of cloud-native applications, we realized that existing solutions for disaggregated storage using NVMe over Fabrics were falling far short of the mark. Initially they were non-standard; when they followed the standards existing at the time, they did not work well; and when they did work, they were custom, non-standard, proprietary, and locked in their users tighter than a constricting boa.
Amazon, Facebook and the RoTW
Unless you are working at one of the very largest of companies with the most massive storage needs in the world, such as Amazon and Facebook, non-standard storage solutions simply are not cost effective. What that means is nearly any company with significant storage requirements should be building its infrastructure to scale, all while looking at solutions that are not going to lock them in to one vendor. It’s not fun getting constricted.
Storage disaggregation with NVMe/TCP
To build storage infrastructure at scale, you need the right standard, and NVMe/TCP is the gold standard. Lightbits team members having been among the creators of the earlier NVMe and NVMe-oF standards, we knew what was missing to truly enable storage disaggregation at scale and without lock in. NVMe/TCP will soon be included in every major operating system and hypervisor and is already a standard part of the Linux kernel.
Using the Lightbits software-defined disaggregated storage solutions, which are based on NVMe/TCP and speak NVMe/TCP and only NVMe/TCP between servers in the data center, any company can build its own large scale storage infrastructure, without getting locked in to non-standard solutions. Among the benefits of Lightbits’ NVMe/TCP solutions is that for companies that cannot move their storage to the cloud for reasons such as industry regulations, latency-sensitive applications, and more, NVMe/TCP and standard solid-state drives (SSDs) allow them to remain competitive without moving to the cloud (at a cost of around 3 cents per GB of flash storage).
While we weren’t the first to try and use NVMe-oF for storage disaggregation, we were the first to make it work in a standard way over TCP/IP at any data center scale and without vendor lock in. Some vendors tried to do it with iSCSI, which was designed when the average CPU core count was one, leading to low performance and high latencies. Other vendors tried to use RDMA technology, which simply doesn’t scale. Furthermore, they required proprietary NICs with custom firmware that locked in the user to a specific NIC vendor. Yet other vendors required users to install fat client-side drivers on every server in the data center, creating operational complexity rather than sparking joy. As Marie Kondo might say, better get rid of that clutter. Overall these were all reasonable approaches to try and failure is an inherent part of the innovation cycle. But with NVMe/TCP, the right protocol is finally here, and with Lightbits LightOS, it’s ready for production.
Avoid a Path to Extinction
Our experience tells us that customers don’t want to be stuck with technological dinosaurs. Solutions that only work at small scale, with lots of hand holding, require non-standard software or hardware, and in general do not spark joy, are well on their way to extinction. And with the growing need for hyperscale storage, yesterday’s approaches definitely needed a rethink. The future for data storage at scale is open.
