Ceph Storage Overview
In today’s competitive business environment, cloud hosting for enterprise-level deployments requires a highly scalable storage solution to streamline as well as manage important business data. With technology and best practices fast moving towards cloud-based services to keep pace with evolving businesses, Ceph Storage (Ceph) emerged as a solution to fulfill the need for a software storage solution that encourages a very sustainable model for growth.
In this blog, we outline various aspects of Ceph and how it fulfills the demanding storage needs of businesses.
What is Ceph Storage?
Understanding the Working Of Ceph Block Storage
Ceph Storage Architecture
NVMe/TCP Support for Ceph
Use Cases for Ceph Block and Ceph Object Storage
Advantages and Disadvantages Of Ceph Storage
Ceph Storage Vs AWS S3- Features and Key Differences
Why is Ceph Storage Not Enough for Modern Workloads?
Ceph Alternatives
Ceph Block Storage FAQs
What is Ceph Storage?
Ceph is an open-source, software-defined platform employing a distributed object storage architecture designed for scalability, flexibility, and fault tolerance on commodity infrastructure.
Originally intended for spinning hard disk drives (HDDs), Ceph has incrementally evolved its architecture over the past decade to take advantage of flash storage advances. Recent versions of the software utilize solid-state drives (SSDs) for metadata operations, improving performance. However, many core design elements optimizing HDD behavior remain unchanged.
Ceph aims to provide unified storage, with object, block, and file interfaces in a single platform. The block storage service (RBD) presents virtual block devices to applications. A key benefit of Ceph’s architecture is its ability to leverage commodity hardware to scale capacity across thousands of nodes. However, scaling capacity can substantially increase demands on the supporting cloud network fabric and has been shown to introduce latency in larger-scale private cloud environments.
As a powerful storage solution, Ceph uses its own Ceph file system (CephFS) and is designed to be self-managed and self-healing. It is equipped to handle outages on its own and continually works to reduce administrative costs.
Another highlight of Ceph is that it is quite fault-tolerant, achieving this by easily replicating data. This means there are no bottlenecks in the process while Ceph is operating.
Among some of the latest features of Ceph are:
– High scalability
– Open-source
– High reliability through distributed data storage
– Robust data security through redundant storage
– Advantage of continuous memory allocation
– Convenient software-based increase in availability via an integrated algorithm for locating data
Understanding the Working Of Ceph Block Storage
Ceph primarily uses a Ceph block device, a virtual disk that can be attached to virtual machines or bare-metal Linux-based servers.
A key component of Ceph is RADOS (Reliable Autonomic Distributed Object Store), which provides powerful block storage capabilities such as replication and snapshots that can integrate with OpenStack Block Storage.
Apart from this, Ceph also uses the POSIX (Portable Operating System Interface), a robust file system, to store data in its storage clusters. The advantage is that the file system uses the same clustered system as Ceph block and object storage, enabling it to store massive amounts of data.
Ceph Storage Architecture
Ceph requires several computers connected to one another in a cluster. Each connected computer in that network is called a node.
Below are some of the tasks that must be distributed among the nodes within the network:
– Monitor nodes (ceph-mon): These cluster monitors are mainly used to monitor the status of individual nodes in the cluster, especially the object storage devices, managers, and metadata servers. To ensure maximum reliability, it is recommended to have at least three monitor nodes.
– Object Storage Devices (ceph-osd): Ceph-OSDs are background applications for actual data management and are responsible for storage, duplication, and data restoration. For a cluster, it is recommended to have at least three ODSs.
– Managers (ceph-mgr): They work in tandem with Ceph monitors to manage the status of system load, storage usage, and the capacity of the nodes.
– Metadata servers (ceph-MDS): They help to store metadata such as file names, storage paths, and timestamps of files stored in the CephFS for several performance reasons.
The core of Ceph data storage is an algorithm called CRUSH (Controlled Replication Under Scalable Hashing) that uses a CRUSH Map, an allocation table, to find an OSD for the requested file. CRUSH selects the ideal storage location based on fixed criteria, then duplicates the files and saves them on physically separate media. The network administrator can set the relevant criteria.
The core of the Ceph data storage architecture is RADOS, a highly reliable, distributed object store composed of self-mapping, intelligent storage nodes.
Among some of the ways to access stored data in Ceph are:
– radosgw: In this gateway, data can either be read or written using the HTTP Internet protocol.
– librados: Native access to stored data is possible using the librados software libraries through APIs in programming and different scripting languages, including Python, Java, C/C++, and PHP.
– RADOS Block Device: Data access here requires integration using a virtual system such as QEMU/ KVM or block storage through a kernel module.
NVMe/TCP Support for Ceph
Ceph recently unveiled a technology preview for NVMe/TCP connectivity. However, this implementation involves integrating Ceph’s NVMe-oF gateway. Figure 2 illustrates how the gateway exports RADOS Block Devices (RBD) to clients over NVMe/TCP. This model adds architectural complexity, creating bottlenecks and increasing storage-network latency due to the extra hop and protocol translation.
While Ceph supports modern high-speed protocols such as NVMe/TCP, the current approach relies on protocol gateways and translation layers on top of the existing Ceph architecture. This model may improve Ceph’s interoperability, but it deviates from the initially intended design of NVMe/TCP fabric architectures. Lightbits implements this design and has engineered it to provide direct, high-performance host connectivity.
Ceph Storage Performance
Ceph offers a range of benefits to OpenStack-based private clouds. To better understand Ceph storage performance, here is a look at a few of its features.
High availability & enhanced performance
The coding erasure feature of Ceph improves data availability manifolds simply by adding resiliency and durability. At times, the writing speeds can be almost twice as high as the previous backend.
Robust security
Active Directory integration, encryption features, LDAP, etc., are some of the highlights of Ceph that can limit unwanted access to the system.
Seamless adoption
Shifting to software-defined storage platforms can sometimes be complicated. Ceph overcomes this issue by allowing block and object storage in the same cluster, eliminating the need to administer separate storage services via other APIs.
Cost-effectiveness: Ceph runs on commodity hardware, making it a cost-effective solution without requiring additional or expensive hardware.
Use Cases of Ceph Block and Ceph Object Storage
Ceph was primarily designed to run smoothly on general-purpose server hardware. It supports elastic provisioning, making it economically feasible to build and maintain petabyte- to exabyte-scale data clusters.
Unlike several other mass storage systems that excel at storage but quickly run out of throughput or IOPS before they run out of capacity, Ceph scales performance and capacity independently, enabling deployments optimized for specific use cases.
Some of the most common use cases for Ceph Block & Object Storage are listed below:
Use cases of Ceph Block
– Deploy elastic block storage with on-premises cloud
– Storage for smooth-running VM disk volumes
– Storage for SharePoint, Skype and other collaboration applications
– Primary storage for MYSQL and other similar SQL database apps
– Dev/Test Systems
– Storage for IT management apps
Use cases of Ceph Object Storage
– VM disk volume snapshots
– Video/audio/image repository services
– ISO image store and repository service
– Backup and archive
– Deploy services similar to Dropbox within the enterprise
– Deploy Amazon S3 object store-like services with on-premises cloud
Advantages and Disadvantages Of Ceph
While Ceph storage is a good choice in many situations, it also has disadvantages. Let’s discuss both in this section.
Advantages
– Despite its limited development history, Ceph is free and is an established storage method.
– The application has been extensively and well-documented by the manufacturer.
– A great deal of helpful information is available online for Ceph regarding its setup and maintenance.
– The scalability and integrated redundancy of Ceph storage ensure data security and flexibility within the network.
CRUSH algorithm of Ceph ensures high availability.
Disadvantages
– To be able to fully use all of Ceph’s functionalities, a comprehensive network is required due to the variety of components being provided.
– The setup of Ceph storage is relatively time-consuming, and sometimes the user cannot be entirely sure where the data is physically being stored.
– It requires additional engineering oversight to implement and manage
Ceph vs AWS S3- Features and Key Differences
In this section, we will briefly compare two popular object stores, AWS S3 and Ceph Object Gateway (RadosGW). We will focus on standard features and key differences.
While Amazon S3 (released in 2006) is primarily a public object store provided by AWS and guarantees robust 99.9% availability of objects, Ceph storage is an open-source software that offers distributed object, block and file storage.
Operated under the LGPL 2.1 license, the Ceph Object Gateway daemon (released in 2006) offers two different sets of APIs-
1. One that is compatible with a subset of the Amazon S3 RESTful APIs
2. Second, which is compatible with a subset of the OpenStack Swift API.
One of the key differences between S3 and Ceph Storage is that while Amazon S3 is a proprietary solution available only on Amazon’s commercial public cloud (AWS), Ceph is an open-source product that can be easily installed on-premises as part of a private cloud.
Another difference is that Ceph provides strong consistency, meaning new objects and changes to objects are guaranteed to be visible to all clients. In contrast, Amazon S3 offers read-after-write consistency for creating new objects and eventual consistency for updating and deleting objects.
Why is Ceph Not Enough for Modern Workloads?
While Ceph is highly scalable and a one-size-fits-all solution, it has inherent architectural limitations. Ceph storage is not suitable for modern workloads for the following reasons:
1. Enterprises either working with the public cloud, using their own private cloud, or moving to modern applications require low latency and consistent response times. While BlueStore (a back-end object store for Ceph OSDs) helps to improve average and tail latency to an extent, it cannot necessarily take advantage of the benefits of NVMe® flash.
2. Modern workloads typically deploy local flash (local NVMe flash) on bare metal to get the best possible performance, and Ceph is not equipped to realize the optimized performance of this new media. In fact, Ceph in a Kubernetes environment where local flash is recommended can be an order of magnitude slower than local flash.
3. Ceph has a comparatively poor flash utilization (15-25%). If Ceph or the host fails, rebuild times for shared storage can be very slow because massive traffic must flow over the network for a long period.
To learn more about Ceph performance in a Kubernetes environment, read the “Kubernetes Assimilation: Persistence is Futile” blog.
Ceph Alternatives
While Ceph is a versatile, unified storage system, its architecture, originally designed for HDDs, can be a bottleneck for I/O-intensive tasks, even with all-flash configurations. Ceph’s block storage component, RBD, is scalable; however, it has inherent architectural characteristics that can limit performance for extremely demanding, low-latency block workloads. When a specific workload demands the absolute highest performance, minimal latency, and optimal resource utilization, the specialized alternatives listed below, often built on modern protocols like NVMe/TCP, can provide a significant advantage.
Ceph Alternatives
| Alternative | Key Strengths | Use Case |
|---|---|---|
| GlusterFS | Traditionally file-centric, it’s a distributed storage system that can also support block-based workloads. | Environments that need a simple, effective, open-source solution, especially for file-based workloads with some block requirements. |
| Lightbits | Disaggregated and SDS architecture, high IOPS density, low latency, and efficient resource use. Natively designed with NVMe/TCP | Cloud-native environments with seamless Kubernetes, OpenShift, and OpenStack integrations. Demanding real-time analytics, transactional, AI workloads where flexible, on-demand scaling of compute and storage, cost-efficiency, ultra fast and low latency are essential. |
| QuantaStor | A unified SDS platform supporting block, file, and object storage with a focus on simplifying management. | Organizations needing a flexible and scalable solution for a variety of storage workloads, with a built-in management system. |
| OpenEBS | A container-native storage solution for Kubernetes offering various storage engines, | Kubernetes deployments where persistent storage needs to be tightly integrated with the container orchestration platform. |
Ceph Block Storage FAQs
- What are the key differences between replication and erasure coding for Ceph block storage? Replication creates multiple complete copies of your data. For example, with a 3x replication factor, Ceph stores three identical copies of every piece of data. Erasure coding is more like RAID. It breaks data into “chunks” and creates additional “parity” chunks, distributing them across the cluster.
- How do I manage and create Ceph block devices? You typically manage it via the Ceph dashboard or the CLI.
- What is Ceph block storage and how does it work? This is already covered in this blog. To reiterate, Ceph’s block storage, also known as RBD, provides a virtual block device you can use with VMs, containers, and other applications.
Conclusion
With data at the center of almost every business, choosing the right storage platform is becoming increasingly important. Ceph storage is designed to make your data more accessible to you and your business applications.
However, while Ceph is a good choice for applications that can tolerate spinning-disk performance, it has architectural shortcomings that make it unsuitable for high-performance, scale-out databases and similar web-scale software infrastructure.
Lightbits, a software-defined storage solution for NVMe over TCP, is an excellent alternative to Ceph storage, especially when simplicity, cost optimization, low latency, and consistent performance are priorities. Beyond its resilience, it offers rich data features not typically associated with NVMe, such as optional compression and thin provisioning.
Lightbits also works well with various cloud-native application environments because of plugins for OpenStack (Cinder) and Kubernetes (CSI), and it can also run on bare metal while offering robust scalability.
