Today, in IT infrastructure, with the widespread use of virtualization, storage systems are core components housing all virtual machines. A failure of this node can completely halt the operation of a data center. Although a significant portion of server equipment offers fault tolerance in one form or another by default, due to the special role of storage systems within a data center, higher requirements are placed on their ‘survivability’.
The most effective method of ensuring fault tolerance in IT is the use of multiple instances of hardware and software (in the simplest case – duplication). Of course, a storage system can be fully duplicated. And for disaster recovery, this is indeed the approach used. However, not all companies can afford such a solution. This is not only about the doubled cost of equipment, but also about other expenses related to organizing such a solution and its subsequent maintenance.
However, the possibility of duplicating hardware does not negate the need to ensure fault tolerance at the component level. In particular, storage systems utilize redundancy for power supplies, cooling modules, storage drives, and, of course, controllers. This has long become commonplace. It is difficult to find a storage system that does not use such a design. This is no exception. But in this article, we want to discuss what doesn't immediately stand out, and is primarily aimed at increasing the overall fault tolerance of the system.
Cooling Modules
Very often in 2U-3U chassis, combined modules are used, bringing together power supplies and fans. On one hand, this is convenient, as only one unit needs to be serviced. On the other hand, if the cooling system fails, the power supply may be forcibly shut down to avoid overheating. While this may not create a critical situation, clearly adding vulnerabilities to the storage system is not advisable.
Cooling in the Qsan storage system is organized in the form of separate modules with 'hot' swapping, independent of the power supplies. Each power supply unit has its own fans intended for cooling itself. The cooling module contains two independent fans that back each other up. There are two such modules in the storage system, one on the right and one on the left, for effective cooling of all components. If one of the fans fails, all other fans automatically increase their speed to compensate for the resulting airflow deficiency. This is why a fan failure does not pose a risk of overheating the entire device.
Topology of expansion shelf connections
Classic connection scheme to the storage system implies a topology called cascading. In this case, the corresponding shelf and storage system controllers are connected by a single SAS cable. Thus, there are 2 cables for a dual-controller system. If a second one needs to be connected, it is linked in the same manner to the first shelf, and so forth. The advantage of this topology is its simplicity in terms of equipment implementation. However, the downside is a certain vulnerability to sudden SAS chain breaks due to the failure of unconnected controllers of the storage system and shelf, or due to power loss at one of the expansion shelves midway through the chain. The result will be loss of access to some drives and possible RAID group collapse if it is 'spread' across several enclosures.
To protect against cross-controller failures, Qsan has a safeguard in the form of an internal logical link between controllers via the storage system backplane. This means that the storage system controller can see not only the JBOD controller directly connected to it but also the 'neighboring' controller through a special link in the backplane. As a result, if such a situation occurs and no one physically unplugs the SAS cables between the storage system and the shelf, access to all drives will be maintained.
To protect against SAS chain breaks, for example, due to power loss at an expansion shelf, a different connection topology is usually applied – reverse cascading. In this case, the storage system is connected to both the first and last shelves in the chain, gaining access to the drives from two sides.
If you're looking for stronger protection, you can build larger configurations using, for instance, a tree topology. Alternatively, you can complicate it further by combining the mentioned topologies. This is possible due to the large number of SAS connectors on devices (2 for each storage controller and 5 for each JBOD controller) with automatic input/output mode detection. The main thing is that the administrator doesn't get confused. The storage system will be able to configure correctly.
Fast rebuild
Having backup disks in the system for 'hot' replacement (hot spare) significantly increases the reliability of information storage. However, just the fact of allocating such disks does not mean absolute protection. The recovery (rebuild) process is quite labor-intensive and often time-consuming. The labor intensity arises from the continuous access to the main data. That is, the system, while performing its current tasks, must also copy data to the new disk. The duration of the rebuild directly depends on the capacity of the storage device and its speed characteristics. Since the system does not know the actual occupied space on the disks, it simply copies everything block by block during the rebuild.
As a result, recovering a modern large-capacity disk of 10+ TB under significant load on the storage system can easily take a week or more. One should also keep in mind that during the rebuild, the likelihood of other drives failing significantly increases due to the added load on them. This can pose a serious risk, especially when using RAID5, for example.
To address this issue, many storage system developers have sought to accelerate the recovery process. Various approaches can be used, but the essence remains the same – copy only the actually occupied blocks during the rebuild. Qsan has not remained indifferent to this problem. In the storage systems of this vendor, when the option is activated, the system tracks the blocks used for writing, thereby being able, in the event of a disk failure, to copy only those blocks to the new storage device.
The Fast Rebuild option is not enabled by default when creating new volumes because its use affects performance, especially during random write operations, because:
- Tracking of entries in blocks must be maintained;
- When rebuilding, checksums for unallocated space are not recalculated. Therefore, when writing to this area, it is necessary to first 'initialize' it.
Thus, it is not recommended to use Fast Rebuild for volumes, for example, with high-load databases or in video surveillance systems, where the volume will eventually be filled to 100%. However, this option can be very useful for file or mail servers.
In conclusion
Every storage system manufacturer assumes that their devices are reliable. If there are no fatal design flaws in the development of devices and an incredible eagerness to save in the production and testing process, then one can generally agree with the vendor. However, it is important to understand:
- Basic fault tolerance in storage systems primarily means the ability to continue accessing data in the event of a failure of one or more components;
- Additional fault tolerance options (like those described above) are exceptions to certain types of failures, increasing your chances of accessing data;
- 100% reliability, alas, does not exist. However, to get as close to it as possible, most reasonable storage vendors (including themselves) make maximum efforts to continuously improve their products in both hardware and software.
It should also be remembered that no absolute reliability of storage systems negates the need for backups, clear and rehearsed recovery plans in case of a disaster, and prompt technical support from the vendor.
Source: habr.com
