{"id":95870,"date":"2020-10-04T13:41:59","date_gmt":"2020-10-04T11:41:59","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pochemu-vazhno-proverit-po-na-vashej-shd-vysokoj-dostupnosti-999999"},"modified":"2020-10-04T13:41:59","modified_gmt":"2020-10-04T11:41:59","slug":"pochemu-vazhno-proverit-po-na-vashej-shd-vysokoj-dostupnosti-999999","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/pochemu-vazhno-proverit-po-na-vashej-shd-vysokoj-dostupnosti-999999","title":{"rendered":"Why it is important to check the software on your high availability storage system (99.9999%)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Why<\/p>\n

Which firmware version is the most \"correct\" and \"functional\"? If the storage system guarantees 99.9999% fault tolerance, does that mean it will operate reliably even without software updates? Or conversely, to achieve maximum fault tolerance, should one always install the latest firmware? Let's try to answer these questions based on our experience.<\/p>\n

A small introduction<\/h3>\n

We all understand that every software version, whether it's an operating system or a driver for some device, often contains flaws\/bugs and other \"features\" that may either not \"manifest\" until the equipment\u2019s end of life or only become apparent under certain conditions. The number and significance of such nuances depend on the complexity (functionality) of the software and the quality of testing during its development. <\/p>\n

Often, users stick with the \"factory firmware\" (the famous \u2014 \"if it works, don't touch it\") or always install the latest version (in their understanding, the latest is the most functional). We use a different approach \u2014 we check the release notes for all the software we use. and carefully select the appropriate firmware for each unit of equipment.<\/a><\/noindex> We reached this conclusion through experience. Using our own operational instance, we will explain why the promised 99.9999% reliability of the storage system means nothing if you don't timely monitor updates and descriptions of the software. Our case study is suitable for users of storage systems from any vendor, as such situations can occur with hardware from any manufacturer. <\/p>\n

Choosing a new data storage system<\/p>\n

At the end of last year, an interesting data storage system was added to our infrastructure: the junior model from the IBM FlashSystem 5000 line, which at the time of acquisition was called Storwize V5010e. It is now sold under the name FlashSystem 5010, but in fact, it is the same hardware base with the same Spectrum Virtualize inside.<\/h3>\n

At the end of last year, our infrastructure added an interesting data storage system: the lower model from the IBM FlashSystem 5000 series, which at the time of purchase was known as the Storwize V5010e. It is currently sold under the name FlashSystem 5010, but it is essentially the same hardware base with the same Spectrum Virtualize inside. <\/p>\n

The presence of a unified management system \u2014 is, by the way, the main distinction of the IBM FlashSystem. For the entry-level models, it is practically indistinguishable from the higher-performing models. Choosing a specific model only provides the corresponding hardware base, the characteristics of which allow for certain functionality or provide a higher level of scalability. The software identifies the hardware part and provides the necessary and sufficient functions for this platform. <\/p>\n

\"WhyIBM FlashSystem 5010<\/p>\n

A brief overview of our model 5010. This is a dual-controller block storage system for entry-level use. It can accommodate NLSAS, SAS, and SSD disks. NVMe installation is not available, as this model is intended for tasks that do not require the performance of NVMe disks.<\/p>\n

The storage system was purchased for archiving information or data that is not accessed frequently. Therefore, we were content with the standard feature set: tiering (Easy Tier) and Thin Provisioning. Performance on NLSAS disks at 1000-2000 IOPS was also satisfactory. <\/p>\n

Our experience \u2014 how we didn't update the firmware in time<\/h3>\n

Now, regarding the software update itself. At the time of purchase, the system had a slightly outdated version of Spectrum Virtualize software, specifically, 8.2.1.3.<\/strong><\/p>\n

We have studied the firmware descriptions and have planned an update to 8.2.1.9<\/strong>. If we had been a little more prompt, this article wouldn't exist \u2014 the bug would not have occurred with the newer firmware. However, for certain reasons, the update of this system was postponed.<\/p>\n

As a result, a small delay in the update led to a very unpleasant situation, as described in the link: https:\/\/www.ibm.com\/support\/pages\/node\/6172341<\/u><\/a><\/noindex>. <\/p>\n

Yes, the firmware of that version contained the relevant so-called APAR (Authorized Program Analysis Report) HU02104. It manifests as follows: under load, under certain circumstances, the cache begins to overflow, and then the system enters a protective mode, which disables input-output for the pool. In our case, this looked like disabling three disks for the RAID group in RAID 6 mode. The disconnection lasts for 6 minutes. After that, access to the volumes in the pool is restored.<\/p>\n

If anyone is unfamiliar with the structure and naming of logical entities in the context of IBM Spectrum Virtualize, I'll briefly explain now.<\/p>\n

\"WhyStructure of Logical Elements of Storage Systems<\/p>\n

Disks are grouped together, referred to as MDisk (Managed Disk). An MDisk can represent a traditional RAID (0,1,10,5,6) or a virtualized one \u2013 DRAID (Distributed RAID). Using DRAID enhances the performance of the array, as all disks in the group are utilized, and it decreases rebuild time since only specific blocks need to be restored instead of all data from a failed disk.<\/p>\n

\"WhyDistribution of data blocks across disks when using Distributed RAID (DRAID) in RAID-5 mode.<\/p>\n

This diagram illustrates the logic of DRAID rebuild in the event of a single disk failure:<\/p>\n

\"WhyLogic of the DRAID rebuild process when a single disk fails<\/p>\n

Next, one or several MDisk will form what is called a Pool. Within a single pool, it is not recommended to use MDisk with different levels of RAID\/DRAID on disks of the same type. We won\u2019t delve too deeply into this, as we plan to cover it in one of the upcoming articles. Moreover, the Pool is divided into Volumes (Tomas), which are presented via a specific block access protocol to the hosts.<\/p>\n

So, we encountered a situation described in APAR HU02104<\/em>, due to a logical failure of three disks, resulting in the MDisk becoming non-operational, which, in turn, led to the failure of the Pool and corresponding Volumes.<\/p>\n

Since these systems are quite \"smart,\" they can be connected to IBM Storage Insights cloud monitoring system, which automatically sends a service request to IBM support when a fault occurs. A ticket is created, and IBM specialists perform diagnostics remotely and contact the system user. <\/p>\n

Thanks to this, the issue was resolved promptly, and we received a quick recommendation from support to update our system to the previously selected firmware version 8.2.1.9, in which this issue had already been fixed at that time. This is confirmed by the relevant Release Note<\/u><\/a><\/noindex>.<\/u><\/p>\n

Conclusions and our recommendations<\/h3>\n

As the saying goes: \"all's well that ends well.\" The firmware bug did not lead to serious issues \u2014 server operations were restored quickly and without data loss. Some clients had to restart their virtual machines, but overall, we were prepared for worse outcomes, as we make daily backups of all infrastructure components and client machines. <\/p>\n

We have confirmed that even reliable systems with 99.9999% promised availability require attention and timely maintenance. Based on the situation, we have drawn several conclusions and are sharing our recommendations:<\/p>\n