Hello, readers of Habr. In this article, we are launching a series that will tell you about our developed hyper-converged system AERODISK vAIR. Initially, we wanted to cover everything in the first article, but the system is quite complex, so we will tackle it piece by piece.
Let's start with the history of the system's creation, delve into the ARDFS file system, which is the foundation of vAIR, and also reflect a bit on the positioning of this solution in the Russian market.
In future articles, we will describe different architectural components (cluster, hypervisor, load balancer, monitoring system, etc.), the setup process, discuss licensing issues, separately showcase crash tests, and of course, we will write about load testing and sizing. We will also dedicate a separate article to the community version of vAIR.
Is AERODISK a story about storage systems? Or why did we even start working on hyper-convergence?
Initially, the idea to create our own hyper-converged solution came to us around 2010. At that time, there were no AERODISK or similar solutions (commercial box hyper-converged systems) on the market. Our task was as follows: from a set of servers with local disks, interconnected via Ethernet protocol, we needed to create a stretched storage and also run virtual machines and a software network there. All of this needed to be implemented without a storage system (as we simply didn’t have the budget for a storage system and hadn’t invented our own yet).
We tried many open-source solutions and eventually solved this task, but the solution was very complex and difficult to replicate. Moreover, this solution was of the sort 'It works? Don't touch it!'. So, having solved that problem, we didn't pursue the idea of turning the result of our work into a full-fledged product.
After that case, we moved away from this idea, but we still had the feeling that this problem was quite solvable, and the benefits of such a solution were more than evident. Subsequently, the HCI products released by foreign companies only confirmed this feeling.
Therefore, in the middle of 2016, we returned to this task as part of creating a full-fledged product. At that time, we didn't have any relationships with investors, so we had to purchase the development stand with our limited funds. After searching for used servers and switches on Avito, we got to work.
The primary initial task was to create our own, albeit simple, file system that could automatically and evenly distribute data in the form of virtual blocks across an n-number of nodes in a cluster, interconnected via Ethernet. The file system should also scale well and easily and be independent of adjacent systems, meaning it could be detached from vAIR as a 'simple storage solution.'
The first concept of vAIR
We deliberately decided against using existing open-source solutions for organizing a distributed storage system (like Ceph, Gluster, Lustre, and similar) in favor of our own development since we already had significant project experience with them. Undeniably, these solutions are excellent on their own, and prior to working on Aerodisk, we executed several integration projects with them. However, it's one thing to address the specific task of a single client, train personnel, and possibly purchase support from a large vendor, and entirely another to create a product that can be easily replicated for various tasks, some of which we, as a vendor, may not even be aware of. For our second goal, existing open-source products were unsuitable, so we decided to develop the distributed file system ourselves.
After two years of efforts by several developers (who combined working on vAIR with classic storage system Engine), we achieved certain results.
By 2018, we had written a basic file system and supplemented it with the necessary bindings. The system unified physical (local) disks from different servers via an internal interconnect into one flat pool and 'sliced' them into virtual blocks. These virtual blocks were then used to create block devices with varying degrees of fault tolerance, on which virtual machines were created and executed using the KVM hypervisor.
We didn't get too caught up in the name of the file system and simply called it ARDFS (guess what it stands for))
This prototype looked good (not visually, of course, as there was no visual design at that time) and demonstrated strong performance and scalability. After the first real results, we put this project into motion by organizing a full development environment and a separate team dedicated solely to vAIR.
By that time, the overall architecture of the solution had matured, and it has not undergone any significant changes since.
Diving into the ARDFS file system
ARDFS is the foundation of vAIR, providing distributed fault-tolerant data storage for the entire cluster. One of (but not the only) the distinguishing features of ARDFS is that it doesn't use any additional dedicated servers for metadata and management. This was initially designed to simplify the configuration of the solution and ensure its reliability.
Storage Structure
Across all cluster nodes, ARDFS organizes a logical pool of all available disk space. It is important to understand that a pool is not yet data or formatted space, but simply a layout; that is, any nodes with vAIR installed automatically get added to the common ARDFS pool when they join the cluster, and the disk resources become shared across the cluster (and available for future data storage). This approach allows for adding and removing nodes on the fly without any significant impact on the already functioning system. In other words, the system can be easily scaled in 'bricks,' adding or removing nodes in the cluster as needed.
On top of the ARDFS pool, virtual disks (storage objects for virtual machines) are added, constructed from virtual blocks sized at 4 megabytes. The data is stored directly on these virtual disks. A fault-tolerance scheme is also defined at the level of virtual disks.
As you might have guessed, for the redundancy of the disk subsystem, we do not use the concept of RAID (Redundant Array of Independent Disks) but rather RAIN (Redundant Array of Independent Nodes). That is, redundancy is measured, automated, and managed based on nodes, not disks. Disks, of course, are also storage objects; they, like everything else, are monitored, and all standard operations can be performed on them, including building a local hardware RAID, but the cluster operates specifically on nodes.
In situations where RAID is highly desired (for example, scenarios that support multiple failures in small clusters), nothing prevents the use of local RAID controllers while creating a stretched storage and RAIN architecture on top. Such scenarios are quite viable and supported by us, which is why we will discuss it in the article about typical vAIR use cases.
Storage Redundancy Schemes
There can be two types of redundancy schemes for virtual disks in vAIR:
1) Replication factor or simply replication – this method of redundancy is as simple as 'a stick and a rope'. It involves synchronous replication between nodes with a factor of 2 (2 copies in the cluster) or 3 (3 copies, respectively). An RF-2 allows a virtual disk to withstand the failure of one node in the cluster but 'consumes' half of the usable capacity, while RF-3 can withstand the failure of two nodes in the cluster but reserves 2/3 of the usable capacity for its needs. This schema very much resembles RAID-1, meaning that a virtual disk configured in RF-2 is resilient to the failure of any single node in the cluster. In this case, the data will be fine, and even input/output will not stop. When the failed node comes back online, automatic recovery/synchronization of data will begin.
Below are examples of RF-2 and RF-3 data distribution in normal operation and during failure situations.
We have a virtual machine with 8MB of unique (useful) data, operating on 4 vAIR nodes. It is understood that in reality, such a small volume is unlikely, but for a diagram reflecting the logic of ARDFS, this example is the clearest. AB represents virtual blocks of 4MB containing the unique data of the virtual machine. In RF-2, two copies of these blocks A1+A2 and B1+B2 are created, respectively. These blocks are 'spread out' across nodes, avoiding the overlap of the same data on one node, meaning copy A1 will not be on the same node as copy A2. The same goes for B1 and B2.
In case one of the nodes fails (for example, node #3, which contains copy B1), this copy will be automatically activated on the node where its copy (i.e., copy B2) does not exist.
Thus, the virtual disk (and the VM, respectively) will easily survive the failure of one node in the RF-2 scheme.
The replication scheme, despite its simplicity and reliability, suffers from the same issue as RAID1 – limited usable space.
2) Erasure coding, or EC, also known as 'redundant coding' or 'erasure coding', exists to solve the above problem. EC is a redundancy scheme that ensures high data availability with lower overhead costs in disk space compared to replication. The principle of this mechanism is similar to RAID 5, 6, 6P.
During encoding, the EC process divides a virtual block (by default 4MB) into several smaller 'data chunks' depending on the EC scheme (for example, the 2+1 scheme divides each 4MB block into 2 parts of 2MB each). Then, this process generates 'parity chunks' with a size of no more than one of the previously divided parts for the 'data chunks'. When decoding, EC generates the missing chunks by reading the 'surviving' data throughout the cluster.
For example, a virtual disk with an EC scheme of 2 + 1, implemented on 4 nodes of a cluster, will easily withstand the failure of one node in the cluster just like RF-2. The overhead costs will be lower, with the useful capacity ratio in RF-2 being 2, while in the EC 2+1 it will be 1.5.
To describe it more simply, the essence is that the virtual block is divided into 2-8 (the reason for 2 to 8 is explained below) 'chunks', and for these chunks, 'parity' chunks of the same volume are calculated.
As a result, data and parity are evenly distributed across all nodes in the cluster. Similarly to replication, ARDFS automatically distributes data across nodes in such a way that it prevents identical data (data copies and their parity) from being stored on the same node, thus eliminating the risk of data loss due to both the data and its parity residing on a single storage node that may fail.
Below is an example with the same virtual machine of 8 MB and 4 nodes, but using the EC 2+1 scheme.
Blocks A and B are divided into two pieces of 2 MB each (two because of 2+1), resulting in A1+A2 and B1+B2. Unlike a replica, A1 is not a copy of A2; it is a virtual block A split into two parts, as is block B. Thus, we obtain two sets of 4 MB, each containing two two-megabyte pieces. Next, for each of these sets, parity is calculated with a volume of no more than one piece (i.e., 2 MB), resulting in an additional + 2 pieces of parity (A-P and B-P). In total, we have 4×2 data + 2×2 parity.
Next, the pieces are distributed across nodes in such a way that the data does not overlap with their parity. That is, A1 and A2 will not reside on the same node with A-P.
In the event of a failure of one node (let's say, the third one), the fallen block B1 will be automatically restored from the parity B-P, which is stored on node number 2, and will be activated on a node where there is no B parity, i.e., the piece B-P. In this example, that is node number 1.
I’m sure the reader is wondering:
"Everything you described has long been implemented by competitors and in open-source solutions; what distinguishes your implementation of EC in ARDFS?"
And then we will go into the interesting features of how ARDFS works.
Erasure coding with a focus on flexibility
Initially, we designed a fairly flexible EC X+Y scheme, where X ranges from 2 to 8, and Y ranges from 1 to 8, but always less than or equal to X. This scheme is intended for flexibility. Increasing the number of data pieces (X) into which a virtual block is divided helps reduce overhead, thereby increasing usable space.
On the other hand, increasing the number of parity pieces (Y) enhances the reliability of the virtual disk. The higher the value of Y, the more nodes in the cluster can fail. Naturally, increasing the volume of parity reduces the effective capacity, but this is the price for reliability.
The performance dependency on EC schemes is almost direct: the more "pieces" there are, the lower the performance. Here, a balanced perspective is clearly needed.
This approach allows administrators to configure stretched storage as flexibly as possible. Within the ARDFS pool, any fault tolerance schemes and their combinations can be used, which we also consider very useful.
Below is a comparison table of several (not all possible) RF and EC schemes.
The table shows that even the most "extreme" combination of EC 8+7, allowing for the loss of up to 7 nodes in the cluster simultaneously, "consumes" less usable space (1.875 versus 2) than standard replication, while providing 7 times better protection. This makes the protection mechanism more complex, but significantly more attractive in situations where maximum reliability is needed in conditions of limited disk space. At the same time, it should be understood that each "plus" to X or Y will be an additional overhead on performance, so in the triangle between reliability, economy, and performance, one must choose very carefully. For this reason, we will dedicate a separate article to the sizing of erasure coding.
Reliability and autonomy of the file system
ARDFS runs locally on all nodes of the cluster and synchronizes them using dedicated Ethernet interfaces. An important point is that ARDFS synchronizes not only data but also metadata related to storage. During the development of ARDFS, we simultaneously studied a number of existing solutions and found that many perform file system metadata synchronization using an external distributed DBMS, which we also use for synchronization, but only for configurations, not for file system metadata (about this and other related subsystems in the next article).
Synchronizing FS metadata using an external DBMS is a workable solution, but it would mean that the consistency of the data stored on ARDFS would depend on the external DBMS and its behavior (which is, frankly, quite capricious). In our opinion, that’s a flaw. Why? If the FS metadata gets corrupted, the data itself could also say "goodbye." Therefore, we decided to take a more complex but reliable path.
We developed the metadata synchronization subsystem for ARDFS independently, and it operates completely autonomously from adjacent subsystems. That means no other subsystem can corrupt ARDFS data. In our view, this is the most reliable and correct approach, but whether that’s truly the case will be revealed over time. Additionally, this approach offers another advantage: ARDFS can be used independently of vAIR, simply as an extended storage, which we will definitely utilize in future products.
As a result, by developing ARDFS, we received a flexible and reliable file system that allows choosing where to save on capacity or dedicate everything to performance, or to make the storage extremely reliable at a moderate cost while reducing performance requirements.
Along with a straightforward licensing policy and a flexible supply model (to give a preview, vAIR is licensed by nodes and supplied either as software or as a package), this allows us to finely tailor the solution to a variety of customer requirements and easily maintain this balance going forward.
Who needs this wonder?
On one hand, one could argue that there are already players in the market with serious hyper-converged solutions, which makes one wonder why we are entering this space. This statement seems correct, BUT…
On the other hand, going "to the fields" and communicating with customers, we and our partners see that this is not the case at all. There are many tasks for hyper-convergence, and sometimes people simply didn’t know that such solutions exist, other times it seemed too expensive, or there were unsuccessful tests of alternative solutions, and sometimes purchasing is outright prohibited due to sanctions. In short, the field turned out to be untapped, so we set out to cultivate it))).
When is a SAN better than a hyper-converged solution?
During our work with the market, we are often asked when it is better to use the classic architecture with storage systems and when to use hyper-converged solutions. Many companies that produce hyper-converged systems (especially those without storage systems in their portfolio) claim: 'Storage systems are becoming outdated, hyper-converged only!'. This is a bold statement, but it doesn't fully reflect reality.
To be honest, the storage system market is indeed shifting towards hyper-convergence and similar solutions, but there is always a caveat.
Firstly, data centers and IT infrastructures built with traditional storage systems cannot be easily restructured, so the modernization and expansion of such infrastructures will take another 5-7 years.
Secondly, most infrastructures currently being built (specifically in Russia) are constructed using traditional storage systems, not because people are unaware of hyper-convergence, but because the hyper-converged market is new, solutions and standards are not yet established, IT professionals are still untrained, there is little experience, and data centers need to be built here and now. This trend will continue for 3-5 years (and then there will be legacy issues, see point 1).
Thirdly, there is a purely technical limitation with small additional delays of 2 milliseconds for writes (not counting local cache, of course), which is the price for distributed storage.
And let's not forget the use of large physical servers that favor vertical scaling of the disk subsystem.
There are many necessary and popular tasks where storage systems perform better than hyper-converged systems. Naturally, those manufacturers without storage systems in their product portfolio will disagree, but we are ready to argue our case. Of course, as developers of both products, we will conduct a comparison of storage systems and hyper-converged systems in one of our future publications, where we will clearly demonstrate under which conditions each is superior.
And where will hyper-converged solutions work better than storage systems?
Based on the points above, three obvious conclusions can be drawn:
- Where the additional 2 milliseconds of write delay, which consistently occur in any productive environment (we are not talking about synthetic benchmarks here; synthetic metrics can show even nanoseconds), are not critical, hyper-convergence is applicable.
- Where the load from large physical servers can be transformed into many small virtual ones and distributed across nodes, hyper-convergence will fit in well.
- Where horizontal scaling is more prioritized than vertical scaling, hyperconverged solutions will do wonderfully.
What are these solutions?
- All standard infrastructure services (directory service, email, electronic document management systems, file servers, small or medium ERP and BI systems, etc.). We call this 'common computing.'
- The infrastructure of cloud providers, where it's necessary to rapidly and consistently scale horizontally and easily 'slice' a large number of virtual machines for clients.
- Infrastructure virtual desktops (VDI), where many small user virtual machines are launched and smoothly 'float' within a uniform cluster.
- Branch networks, where each branch needs to achieve a standard, fault-tolerant, yet affordable infrastructure made up of 15-20 virtual machines.
- Any distributed computing (big data services, for example). Where the load goes not 'deep,' but 'wide.'
- Testing environments, where small delays are acceptable, but there are budget limits because this is testing.
At this moment, we have developed AERODISK vAIR specifically for these tasks, and we are focusing on them (so far successfully). This may change soon, as the world is constantly evolving.
So...
This concludes the first part of a large cycle of articles; in the next article, we will discuss the architecture of the solution and the components used.
We welcome questions, suggestions, and constructive discussions.
Source: habr.com
