This article is written to help choose the right solution and understand the differences between SDS options such as Gluster, Ceph, and Vstorage (Virtuozzo).
The text contains links to articles that provide more detailed insights into various issues, so the descriptions will be as concise as possible, highlighting key points without unnecessary fluff and introductory information, which you can find online if desired.
In reality, the topics covered do require a certain tone, but in today's world, more and more people dislike lengthy reading))), so you can quickly skim through and make a choice, and if something is unclear, you can follow the links or Google any unfamiliar terms))), and this article serves as a transparent wrapper for these in-depth topics, showcasing the core – the main key points of each solution.
Gluster
Let's start with Gluster, which is actively used by manufacturers of hyper-converged platforms with open-source-based SDS for virtual environments and can be found on the RedHat website in the storage section, where you can choose between two SDS options: Gluster or Ceph.
Gluster consists of a stack of translators – services that handle all tasks related to file distribution, etc. Brick – is the service that manages a single disk, and Volume – a pool that aggregates these bricks. Then there’s the file distribution service through the DHT (distributed hash table) function. We won’t include the Sharding service in the description as the links provided below will address issues related to it.
When writing, a file is entirely placed in a brick, and its copy is written simultaneously to a brick on the second server. After that, the second file will be written to the second group of two bricks (or more) on different servers.
If files are approximately the same size and the volume consists of only one group, everything is fine. However, under different conditions, the following issues may arise from the descriptions:
- space in the groups is utilized unevenly, depending on file sizes. If there isn’t enough space in a group to write a file, you will encounter an error, and the file will not be recorded or redistributed to another group;
- when writing a single file, I/O operations occur only on one group, while the others remain idle;
- it’s not possible to achieve the I/O of the entire volume when writing a single file;
- and the overall concept appears less performant due to the lack of data distribution across blocks, where balancing is easier and the issue of even distribution can be resolved, rather than having the file laid out in a brick in its entirety as it is now.
From the official description it also involuntarily comes to mind that Gluster operates as a file storage layer on top of classical hardware RAID. There have been attempts to shard files into blocks, but all of this is an addition that incurs performance losses to the already existing architectural approach, plus the use of freely distributed components with performance limitations such as Fuse. There are no metadata services, which limits the performance and fault tolerance capabilities of the storage when distributing files into blocks. Better performance indicators can be observed with the "Distributed Replicated" configuration, and the number of nodes should be at least 6 to organize a reliable replica of 3 with optimal load distribution.
These conclusions are also related to descriptions of the utilization experience and when compared to , there is also an account of the experience leading to the understanding of this more performant and reliable configuration
The image shows the load distribution while writing two files, where copies of the first file are spread across the first three servers grouped in volume 0, and three copies of the second file are placed on the second group volume1 made up of three servers. Each server has one disk.
The overall conclusion is that Gluster can be used, but with the understanding that there will be limitations in performance and fault tolerance which create difficulties under certain conditions of hyper-converged solutions, where resources are also needed for the computational loads of virtual environments.
There are also some performance indicators of Gluster that can be achieved under certain conditions, limiting in
Ceph
Now let's consider Ceph from the descriptions of the architecture that I managed to There is also a comparison between , where it can be immediately understood that Ceph should ideally be deployed on separate servers, as its services require all hardware resources under load.
Architecture is more complex than Gluster and includes services such as metadata services, but the entire stack of components is quite intricate and not very flexible for virtualization solutions. Data is stored in blocks, which appears to be more efficient, but within the hierarchy of all services (components), there can be losses and latency under certain loads and emergency conditions, as illustrated in the following example.
The architecture description shows that the heart of the system is CRUSH, which determines where data is placed. Next is PG — this is the most complex abstraction (logical group) to understand. PGs are needed to make CRUSH more efficient. The main purpose of PGs is to group objects to reduce resource consumption, improve performance, and enhance scalability. Addressing objects directly, individually, without aggregating them into PGs would be very costly. OSD is the service for each individual disk.
A cluster can have one or multiple data pools with different purposes and configurations. Pools are divided into placement groups. Objects that clients access are stored in these placement groups. This is where the logical layer ends and the physical layer begins, as each placement group is assigned one primary disk and several replica disks (the exact number depends on the pool's replication factor). In other words, at the logical level, an object is stored in a specific placement group, while at the physical level — on the disks attached to it. These disks may physically reside on different nodes or even in different data centers.
In this scheme, placement groups appear as a necessary level for the overall flexibility of the solution, but at the same time, they also seem like an unnecessary link in this chain, which inevitably raises concerns about performance loss. For instance, when writing data, the system needs to break it down into these groups and then on the physical level distribute it onto the main disk and other disks for replicas. In other words, the hash function operates during the search and insertion of an object, but there is a downside — there are substantial costs and limitations on restructuring the hash (when adding or removing a disk). Another issue with the hash is the fixed location of data, which cannot be changed. Therefore, if a disk experiences increased load, the system cannot write to it (by choosing another disk); the hash function mandates that data be placed according to the rules, regardless of how poorly a disk is performing. Consequently, Ceph consumes a lot of memory when reconstructing PGs during self-healing or expansion of storage. The conclusion is that Ceph works well (albeit slowly) but only in the absence of scaling, emergencies, and upgrades.
There are certainly options to improve performance through caching and cache tiering, but this requires good hardware and will still incur losses. However, overall, Ceph seems more appealing than Gluster for production use. Additionally, when using these products, an important factor to consider is the high level of expertise, experience, and professionalism with a strong emphasis on Linux, as it is crucial to deploy, configure, and maintain everything correctly, which places even greater responsibility and burden on the administrator.
Vstorage
The architecture of , which can be used alongside the hypervisor on the same nodes, on the same , but it is essential to configure everything correctly to achieve good performance. Thus, deploying such a product out of the box on any configuration without regard to architectural guidelines will be very easy, but not efficient.
What can coexist for storage alongside KVM-QEMU hypervisor services, which is just a few services where a compact optimal hierarchy of components is found: a client service mounted via FUSE (modified, not open source), a metadata service (MDS), and a chunk service that operates at the physical level equivalent to a single disk, and that’s it. For speed, it is optimal to use a fault-tolerant scheme with two replicas, but if caching and logging on SSDs are used, then error correction coding (erase coding or RAID 6) can be significantly accelerated on a hybrid scheme or even better on all flash. With EC (erase coding) there is a slight drawback: when changing one data block, parity sums need to be recalculated. To mitigate the loss during this operation, Ceph writes to EC with a delay and performance issues can arise with a certain request when, for example, all blocks need to be read, whereas in Virtuozzo Storage, the recording of changed blocks is done using a “log-structured file system” approach, which minimizes parity computation costs. To estimate the options for accelerating operations with and without EC, there is – the numbers can be approximate depending on the accuracy coefficient of the equipment manufacturer, but the calculation results help plan the configuration well.
A simple storage component scheme does not mean that these components do not consume but if all expenses are calculated in advance, one can expect them to work alongside the hypervisor.
There is a comparison scheme of hardware resource consumption by Ceph and Virtuozzo storage services.
Previously, comparing Gluster and Ceph could be done based on old articles using the most important lines from them, but with Virtuozzo, it's more complex. There aren’t many articles on this product, and information can only be drawn from documentation in or in Russian if considering Vstorage as storage used in some hyper-converged solutions in companies such as and Acronis.
I will try to help with the description of this architecture, so the text will be a bit longer. However, to fully understand the documentation, it requires a lot of time, and the existing documentation can only be used as a reference by reviewing the table of contents or searching by a keyword.
Let's consider the recording process in a hybrid hardware configuration with the components described above: the recording begins on the node that the client (the FUSE mounting point service) initiated. However, the master metadata service (MDS) will direct the client directly to the necessary chunk service (block storage service - CS), meaning the MDS is not involved in the writing process but simply directs to the required chunk service. In general, we can draw an analogy of writing with pouring water into barrels. Each barrel represents a data block of 256MB.
This means one disk is made up of a certain number of those barrels, i.e., the disk capacity divided by 256MB. Each copy is poured into one node, while the second copy is almost simultaneously directed to another node, and so on. If we have three replicas and there are SSDs for caching (for read and write logs), then the write confirmation will occur after writing the log to the SSD, while the parallel flush from the SSD will continue to the HDD, operating in the background. In the case of three replicas, the write commit will happen after confirmation from the SSD of the third node. It may seem that the total writing speed of three SSDs can be divided by three to obtain the writing speed of one replica, but writing the copies occurs in parallel, and the network latency is typically higher than that of SSDs. Essentially, the write performance will depend on the network. Therefore, to see real IOPS, it is necessary to load the entire Vstorage correctly based on , meaning to test the actual load rather than memory and cache, taking into account the correct block size, number of streams, and so on.
The aforementioned log on the SSD works such that as soon as data enters it, the service immediately reads it and writes it to the HDD. There are several metadata services (MDS) per cluster, and their number is determined by the quorum that operates according to the Paxos algorithm. From the client's perspective, the FUSE mount point is a directory of the cluster storage that is simultaneously visible to all nodes of the cluster. Each node has a mounted client based on this principle, making the storage accessible to every node.
For the performance of any of the approaches described above, it is crucial at the planning and deployment stage to correctly configure the network, where balancing occurs through aggregation and appropriate bandwidth of the network link is selected. In aggregation, it is important to choose the correct hashing mode and frame sizes. There is also a significant difference from the SDS described above; it is the fuse with fast path technology in Virtuozzo Storage. It not only features an upgraded fuse, unlike other open-source solutions, but also significantly improves IOPS and allows for scaling beyond horizontal or vertical limits. Overall, compared to the architectures described above, this one appears to be more powerful, but, of course, licensing is required, unlike Ceph and Gluster.
In conclusion, we can highlight a top three: the first place in performance and reliability is taken by Virtuozzo Storage, the second by Ceph, and the third by Gluster.
The criteria for choosing Virtuozzo Storage include an optimal set of architectural components, an upgraded fuse with fast path tailored for this approach, a flexible hardware configuration, lower resource consumption, and the ability to share with compute (calculations/virtualization), making it fully suitable for hyper-converged solutions, of which it is a part. The second place goes to Ceph, as it is a more performant architecture compared to Gluster, due to block manipulation and more flexible scenarios, as well as the ability to operate in larger clusters.
The plan includes a desire to write a comparison between vSAN, Space Direct Storage, Vstorage, and Nutanix Storage, testing Vstorage on HPE and Huawei hardware, as well as integration scenarios for Vstorage with external hardware storage systems. Therefore, if you enjoyed the article, it would be great to receive your feedback, which could enhance motivation for new articles considering your comments and wishes.
Source: habr.com
