In collaboration with Intel, we recently commissioned the Evaluator Group, a technology analyst firm, to run performance benchmarks comparing Ceph Storage with Lightbits. Both are storage systems suited for cloud-native application environments. The requirement was to run the benchmarks on identical hardware and storage media, with workloads running as containers in an OpenShift Kubernetes environment.
Software-Defined Storage
Ceph and Lightbits are both software-defined storage (SDS) solutions, which are storage software that handles policy-based provisioning and storage management independent of underlying hardware. SDS solutions usually include some sort of storage virtualization, which separates the storage hardware from the software that manages it. An SDS solution also typically provides functions like compression, replication, thin provisioning, snapshots, and backup.
Why This Comparison Matters
Comparing Ceph and Lightbits performance matters because two trends are driving significant change in the IT landscape. These are the shift toward cloud computing and the rise of container-based applications. Both technologies let you run and operate applications with greater flexibility and, in some cases, lower costs than was traditionally possible.
However, realizing the benefits of the cloud and containers involves challenges. For example, storage for container-based applications in the cloud needs to be highly scalable. It must provide performance without extensive administration. In Kubernetes, applications can transparently use multiple compute and storage resources. This ability to retain access to persistent data, however, means the underlying storage infrastructure must be resilient while also meeting the capacity and performance requirements of potentially thousands of applications and microservices.
About Ceph Storage
Ceph Storage is an open-source SDS that is often used in cloud-native environments. It was originally designed 15 years ago in the era of spinning hard disk drives (HDDs). At the time, NAND and solid-state drives (SSDs) were in their infancy for mainstream storage systems. Since then, Ceph’s design has adapted to solid-state flash.
Ceph comes with interfaces that enable block, filesystem, and object storage access. However, Ceph has not been optimized for modern Persistent Memory and does not yet support NVMe over fabrics, including NVMe over TCP. Ceph also doesn’t contain optimizations for new solid-state storage, such as Quad-Level Cell (QLC) and Triple-Level Cell (TLC) storage.
About Lightbits
The Lightbits Cloud Data Platform is SDS designed for private, public, and edge clouds. It was built to utilize recent solid-state storage technologies such as NAND flash and Intel Optane persistent memory (PMem). Lightbits also supports NVMe® over TCP (NVMe/TCP) and features a disaggregated architecture. This lets you scale CPU, memory, PMem, or NVMe devices independently as needed.
Lightbits Intelligent Flash Management™ (IFM) is a set of features that maximize performance and extend SSD endurance—delivering 20X more endurance for the storage media. Key IFM features include write striping, IOP-less metadata access, Smart Garbage Collection, Append Write Strategy, and Parallel Read/Write Pipelines.
For this comparison benchmark, Lightbits was configured to work with Intel technologies: components included Intel® Xeon® processors, Intel network adapters and Intel Optane™ technology delivered both as NVMe solid-state drives and as memory form-factor Optane Persistent Memory (Pmem) devices. Lightbits further added Intel Ethernet 800 Series with Application Device Queues (ADQ) that provide high-performance, low-latency NVMe/TCP network interface cards.
The Comparison
Evaluator Group compared Ceph with Lightbits, with each running as containers in a Red Hat® OpenShift® Kubernetes application environment. The test environment comprised 12 Kubernetes nodes running container workloads against either a dedicated three-node Lightbits cluster or a three-node Ceph cluster, running on equivalent hardware. The comparison measured storage performance using the well-known ‘vdbench’ tool to create workloads. The test used eight container instances running vdbench per node, for a total of 96 instances of vdbench.
Ceph was configured to create three data copies for data protection. The Ceph instance consisted of Ceph version 16.2.6, aka “Pacific.” The three Ceph nodes ran on CentOS version 8.4. Each NVMe SSD was configured as six logical object storage devices (OSDs), with six OSDs per device.
The testing covered five access patterns and block sizes. The idea was to mimic the common workloads in performance-sensitive applications:
- 4KB, 100% read with 100% random access
- 4KB, 100% write with 100% random access
- 8KB, 80% read/20% write with 80% random access
- 16KB, 70% read/30% write with 80% random access
- 32KB, 50% read/50% write with 80% random access
These workloads were used to compare the performance of Lightbits and Ceph using Quad-level cell (QLC) solid-state media as the primary storage media. Where appropriate, Evaluator Group deployed higher-speed persistent media, such as Intel Optane™, for each storage system. In all cases, Evaluator Group ran each of the five workloads multiple times to provide an average result for comparison.
The Results
Lightbits outperformed Ceph with each workload. In terms of inputs/outputs per second (IOPs), Lightbits had 3.94X the performance in the 4k-100% read workload, 16.78X in the 4k-100% write, 12.5X in 8k-80% read, 10.41X in 16k-70% read, and 5.65X in 32k-50% read. The chart below summarizes the results. The report contains more detailed findings.
| Ceph IOPs | Lightbits IOPs | Lightbits Advantage | |
|---|---|---|---|
| 4K-100% Read | 1,032,428 | 4,068,462 | 3.94 X |
| 4K-100% Write | 30,728 | 515,697 | 16.78 X |
| 8K-80% Read | 90,363 | 1,129,335 | 12.50 X |
| 16K-70% Read | 35,740 | 372,009 | 10.41 X |
| 32K-50% Read | 19,797 | 111,852 | 5.65 X |
Conclusion
Using the same hardware configuration, Lightbits outperformed Ceph by 4X to 16X. The results demonstrate that leveraging new technologies like Persistent Memory and end-to-end NVMe enabled Lightbits to provide significantly better performance with lower latency than Ceph. The comparison between Ceph and Lightbits matters because storage performance is always an essential consideration for IT managers, especially when running cloud-native applications. Not every application or microservice requires ultra-high I/O, but when you need storage that can support thousands of microservices, it is a good practice to go with the higher-performing storage option when it is available.
Additional Resources:
