{"id":34264,"date":"2019-10-31T21:57:17","date_gmt":"2019-10-31T18:57:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/glavnaya-prichina-avarij-v-data-tsentrah-prokladka-mezhdu-kompyuterom-i-kreslom\/"},"modified":"2019-10-31T21:57:17","modified_gmt":"2019-10-31T18:57:17","slug":"glavnaya-prichina-avarij-v-data-tsentrah-prokladka-mezhdu-kompyuterom-i-kreslom","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/glavnaya-prichina-avarij-v-data-tsentrah-prokladka-mezhdu-kompyuterom-i-kreslom","title":{"rendered":"The main cause of outages in data centers is the distance between the computer and the chair","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

The topic of major outages in modern data centers raises questions that were not addressed in the first article \u2014 we decided to expand on it.<\/em><\/p>\n

\n\"The<\/a><\/noindex><\/p>\n

According to statistics from the Uptime Institute, the majority of incidents in data centers are related to power supply failures \u2014 they account for 39% of occurrences. Following that is the human factor \u2014 contributing another 24% of outages. The third most significant cause (15%) was failures in the cooling system, while natural disasters took fourth place (12%). The total share of other issues is only 10%. While not questioning the data of a reputable organization, let's highlight something common in different outages and try to understand whether they could have been avoided. Spoiler: in most cases, they could.<\/p>\n

<\/p>\n

The Science of Contacts<\/h2>\n

<\/p>\n

In simple terms, there are only two problems with power supply: either there's no contact where it should be, or there is contact where it shouldn't be. One can argue for a long time about the reliability of modern uninterruptible power supply systems, but they don\u2019t always save the day. Take, for instance, the notorious case involving British Airways' data center, owned by the parent company, International Airlines Group. There are two such facilities located near Heathrow Airport \u2014 Boadicea House and Comet House. In the former, an accidental power shutdown occurred on May 27, 2017, which led to an overload and failure of the UPS system. As a result, some IT equipment was physically damaged, and it took three days to rectify the last outage. <\/p>\n

<\/p>\n

The airline had to cancel or postpone over a thousand flights, with around 75,000 passengers unable to depart on time \u2014 compensation payments amounted to $128 million, excluding the costs associated with restoring the data centers' functionality. The reasons behind the blackout remain unclear. If we believe the results of the internal investigation voiced by International Airlines Group's CEO, Willie Walsh, it occurred due to an engineer error. Nevertheless, the uninterruptible power supply system should have withstood such a shutdown \u2014 that\u2019s what it was installed for. The data center was managed by specialists from the outsourcing company CBRE Managed Services, which led British Airways to attempt to claim damages through the London courts. <\/p>\n

\n\"The<\/p>\n

Power outages occur in similar scenarios: first, a shutdown happens due to the energy supplier's fault, sometimes due to bad weather or internal issues (including staff errors), and then the uninterruptible power supply system fails to handle the load or a brief interruption in the sine wave causes failures of many services, which take a lot of time and money to restore. Can such accidents be avoided? Absolutely. If the system is designed correctly, however, even the creators of large data centers are not immune to mistakes. <\/p>\n

<\/p>\n

Human Factor<\/h2>\n

<\/p>\n

When the immediate cause of an incident is the incorrect actions of data center personnel, the problems most often (but not always) affect the software part of the IT infrastructure. Such accidents happen even in large corporations. In February 2017, due to a typo made by a member of the technical operations group at one of the data centers, some servers of Amazon Web Services were shut down. The mistake occurred during the debugging of the billing process for customers of Amazon Simple Storage Service (S3). The employee attempted to delete a certain number of virtual servers used by the billing system, but mistakenly affected a larger cluster.<\/p>\n

\n\"The<\/p>\n

As a result of the engineer's error, servers running important software modules of Amazon's cloud storage were deleted. Primarily, the indexing subsystem, which contains information about the metadata and location of all S3 objects in the US region US-EAST-1, was affected. The subsystem used for data storage and managing the available storage space was also impacted. After the virtual machines were deleted, these two subsystems required a full restart, and Amazon engineers faced a surprise \u2014 for an extended period, the public cloud storage could not handle customer requests. <\/p>\n

<\/p>\n

The impact was significant, as many major resources utilize Amazon S3. The outages affected Trello, Coursera, IFTTT, and, most unfortunately, services of Amazon's major partners from the S&P 500 list. It's hard to quantify the damage in such cases, but it was estimated to be in the hundreds of millions of dollars. As you can see, a single incorrect command is enough to disrupt the service of the largest cloud platform. This is not an isolated incident; on May 16, 2019, during maintenance, the Yandex.Cloud service deleted<\/a><\/noindex> virtual machines of users in the ru-central1-c zone that had ever been in SUSPENDED status. Client data has already suffered here, some of which was irretrievably lost. Of course, people are imperfect, but modern information security systems have long been able to monitor the actions of privileged users before executing the commands they input. If such solutions were implemented in Yandex or Amazon, similar incidents could be avoided.<\/p>\n

\n\"The<\/p>\n

Frozen cooling<\/h2>\n

<\/p>\n

In January 2017, a major incident occurred at the Dmitrov data center of the company Megafon. At that time, the temperature in the Moscow region dropped to -35 \u00b0C, which led to the cooling system of the facility failing. The operator's press service did not elaborate on the causes of the incident \u2014 Russian companies are extremely reluctant to discuss accidents at their facilities; in terms of publicity, we lag significantly behind the West. There was a rumor circulating on social media about the coolant freezing in the pipes laid along the street and an ethylene glycol leak. If this is to be believed, the maintenance service could not promptly obtain 30 tons of refrigerant due to long holidays and had to resort to makeshift solutions, organizing improvised free cooling while violating operational rules. The severe cold exacerbated the problem \u2014 in January, Russia unexpectedly experienced winter, although no one was expecting it. As a result, staff had to power down part of the server racks, which caused some of the operator's services to be unavailable for two days. <\/p>\n

\n\"The<\/p>\n

One could say this is related to a weather anomaly, but such severe frosts are not unusual for the capital region. Winter temperatures in the Moscow region can drop to much lower levels, which is why data centers are built to operate reliably at \u221242\u00b0C. Most often, cooling systems fail in the cold due to insufficient glycol concentration and excess water in the coolant solution. There can also be problems with pipe installation or design calculations and testing related primarily to the desire to cut costs. As a result, a serious accident occurs that could have been easily avoided.<\/p>\n

<\/p>\n

Natural Disasters <\/h2>\n

<\/p>\n

Most often, storms and\/or hurricanes disrupt the engineering infrastructure of the data center, leading to service outages and\/or physical equipment damage. Weather-related incidents happen quite frequently. In 2012, Hurricane Sandy swept along the U.S. West Coast with heavy rainfall. The Peer 1 data center located in a skyscraper in Lower Manhattan lost external power<\/a><\/noindex>, after saltwater flooded the basements. The facility's emergency generators were placed on the 18th floor, and their fuel supply was limited \u2014 regulations introduced in New York after the 9\/11 attacks prohibit storing large amounts of fuel on upper floors. <\/p>\n

\n