{"id":75726,"date":"2020-03-28T07:42:08","date_gmt":"2020-03-28T05:42:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/klaster-iz-dvuh-uzlov-dyavol-v-detalyah"},"modified":"2020-03-28T07:42:08","modified_gmt":"2020-03-28T05:42:08","slug":"klaster-iz-dvuh-uzlov-dyavol-v-detalyah","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/klaster-iz-dvuh-uzlov-dyavol-v-detalyah","title":{"rendered":"A two-node cluster - the devil is in the details","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Hello, Habr! I present to your attention the translation of the article. \u00abTwo Nodes \u2014 The Devil is in the Details\u00bb<\/a><\/noindex> by Andrew Beekhof.<\/p>\n

Many people prefer clusters consisting of two nodes because they seem conceptually simpler and are also 33% cheaper than their three-node counterparts. While it is entirely possible to build a good cluster with two nodes, in most cases, due to unconsidered scenarios, such a configuration will create many non-obvious problems.
\n<\/a><\/noindex>
\nThe first step in creating any high-availability system is to identify and attempt to eliminate single points of failure, often abbreviated as SPoF<\/i> (single point of failure).<\/p>\n

It is worth noting that it is impossible to eliminate all potential downtime risks in any system. This stems, at least in part, from the fact that typical risk mitigation involves introducing some redundancy, which increases system complexity and creates new points of failure. Thus, we initially make compromises and focus on events related to individual points of failure rather than on chains of related and, consequently, increasingly improbable events.<\/p>\n

Given the compromises, we not only look for SPoF but also balance risks and consequences, resulting in the conclusion that what is critical may differ for each deployment.<\/p>\n

Not everyone needs alternate power suppliers with independent power lines. Although paranoia paid off at least for one client when their monitoring identified a faulty transformer. The client called to warn the utility company until the faulty transformer exploded.<\/p><\/blockquote>\n

\nA natural starting point is to have more than one node in the system. However, before the system can move services to the surviving node after a failure, generally, it must be ensured that the services being migrated are not active elsewhere.<\/p>\n

A two-node cluster has no shortcomings if, in the event of a failure, both nodes serve the same static website. However, everything changes if both sides independently manage a shared job queue or provide uncoordinated write access to a replicated database or shared file system.<\/p>\n

Therefore, to prevent data corruption due to a node failure, we rely on what is called fencing<\/i> (fencing).<\/p>\n

The principle of fencing<\/h2>\n

\nThe basis of the fencing principle lies in the question: can a competing node cause data corruption? If data corruption is a likely scenario, a good solution would be to isolate the node from both incoming requests and persistent storage. The most common approach to fencing is to disable the faulty nodes.<\/p>\n

There are two categories of fencing methods that I will refer to as direct<\/i> and indirect<\/i>, but they can equally be termed active<\/i> and passive<\/i>. Direct methods involve actions from the surviving peer nodes, such as interacting with the IPMI (Intelligent Platform Management Interface) device or iLO (server management mechanism in the absence of physical access), while indirect methods rely on the failed node to somehow recognize that it is in an unhealthy state (or at least prevents other members from recovering) and signal hardware watchdog<\/a><\/noindex> the necessity to shut down the failed node.<\/p>\n

Quorum helps in using both direct and indirect methods.<\/p>\n

Direct fencing<\/h3>\n

\nIn the case of direct fencing, we can use a quorum to prevent fencing races in the event of a network failure.<\/p>\n

With the quorum concept, the system has enough information (even without being connected to its partners) for nodes to automatically know whether they should initiate fencing and\/or recovery.<\/p>\n

Without a quorum, both sides of the network split reasonably assume that the other side is dead and will strive to fence the other. In the worst case, both sides manage to disconnect the entire cluster. An alternative scenario is a deathmatch, an endless cycle of nodes appearing, not seeing their peers, rebooting them, and initiating recovery only to reboot when their peer follows the same logic.<\/p>\n

The problem with quorum is that the most commonly used devices become unavailable due to the very failure events we want to leverage for recovery. Most IPMI and iLO cards are installed on the hosts they control and, by default, use the same network, which leads the target nodes to believe that the other nodes are offline.<\/p>\n

Unfortunately, the operational peculiarities of IPMI and iLO devices are rarely considered at the time of purchasing equipment.<\/p>\n

Indirect quorum<\/h3>\n

\nQuorum is also important for managing indirect quorums; if done correctly, quorum can allow the surviving nodes to assume that the lost nodes will enter a safe state after a certain period.<\/p>\n

In this setup, the hardware watchdog timer is reset every N seconds if a quorum is not lost. If the timer (usually several multiples of N) expires, the device performs an ungraceful power off (not a shutdown).<\/p>\n

This approach is very effective, but without a quorum for its management, there is insufficient information within the cluster. It's not easy to distinguish between a network outage and a partner node failure. The reason this matters is that without the ability to distinguish between the two cases, you are forced to choose the same mode of behavior in both scenarios.<\/p>\n

The problem with choosing a single mode is that there is no set of actions that maximally increases availability and prevents data loss.<\/p>\n