Cloister → simple OTP cluster management

Almost every successful business application eventually enters a phase where horizontal scaling is required. In many cases, you can simply start a new instance and reduce the load average. But there are also less trivial cases where we need to ensure that the different nodes know about each other and distribute the workload neatly.

It turned out so well that erlang:, which we chose for its nice syntax and the hype around it, has a first-class support for distributed systems. In theory, this sounds generally trivial:

Message passing between processes on different nodes, and between links and monitors is transparent […]

In practice, things are a little more complicated. Distributed erlang: was developed when "container" meant such a large iron box to transport, and "docker" was just a synonym for a longshoreman. IN IP4 there were a lot of unused addresses, network breaks were usually caused by rats that chewed through the cable, and the average uptime of the production system was measured in decades.

Now we are all unimaginably self-sufficient, packaged, and running a distributed erlang: in an environment where dynamic IP addresses are given out by chance, and nodes can appear and disappear at the will of the scheduler's left heel. To avoid a bunch of boilerplate code in every project that runs a distributed erlang:help is needed to fight the hostile environment.

Note: I know what is libcluster. He's really cool, he has over a thousand stars, the author is known in the community, and all that. If the methods of creating and maintaining a cluster offered by this package are enough for you, I am happy for you. Unfortunately I need a lot more. I want to control the tuning in detail and not be a bystander in the theater of cluster reshaping.

Requirements

What I personally needed was a library that would take over the management of the cluster and would have the following properties:

• transparent work with both a hard-coded list of nodes and dynamic discovery through services erlang:;
• full-featured callback on every topology change (node ​​in, node out, network instability, splits);
• transparent interface for launching a cluster with long and short names, as with :nonode@nohost;
• docker support out of the box, without the need to write infrastructure code.

The latter means that after I tested the application locally in :nonode@nohost, or in an artificially distributed environment using test_cluster_taskI just want to run docker-compose up --scale my_app=3 and see how it runs three instances in docker without any code changes. I also want dependent applications like mnesia - when the topology changes, the cluster was rebuilt behind the scenes for profit without any additional kick from the application.

cloister was not conceived as a library capable of everything from cluster support to making coffee. It is not a silver bullet that aims to cover all possible cases, or be academically complete in the sense that theorists from CS invest in this term. This library is meant to serve a very clear purpose, but to do its not-so-great scope of work perfectly. This goal will be to provide complete transparency between a local development environment and a distributed elastic environment full of hostile containers.

Chosen approach

cloister it is supposed to be run as an application, although advanced users can work with cluster assembly and maintainance manually by directly running Cloister.Manager in the target application's supervisor tree.

When run as an application, the library relies on config, from which it subtracts the following main values:

config :cloister,
  otp_app: :my_app,
  sentry: :"cloister.local", # or ~w|n1@foo n2@bar|a
  consensus: 3,              # number of nodes to consider
                             #    the cluster is up
  listener: MyApp.Listener   # listener to be called when
                             #    the ring has changed

The parameters above mean verbatim the following: cloister used for OTP application :my_app, uses erlang service discovery to connect nodes, three at least, and MyApp.Listener module (implementing @behaviour Cloister.Listener) is configured to receive notifications of topology changes. A detailed description of the complete configuration can be found in documentation.

With this configuration, the application cloister will be launched in stages, delaying the process of starting the main application until a consensus is reached (the three nodes are up and connected, as in the example above.) This leaves the main application to assume that when it starts up, the cluster is already available. Each time the topology changes (there will be a lot of them, because the nodes do not start completely synchronously), the handler will be called MyApp.Listener.on_state_change/2. In most cases, we perform an action when we receive a status message. %Cloister.Monitor{status: :up}, which means: "hello, the cluster is assembled."

In most cases, the installation consensus: 3 is optimal because even if we expect more nodes to connect, the callback will go through status: :rehashing → status: :up on any newly added or removed node.

When running in development mode, simply set consensus: 1 и cloister will joyfully skip waiting for the cluster assembly, seeing :nonode@nohost, or :node@host, or :[email protected] - depending on how the node was configured (:none | :shortnames | :longnames).

Distributed Application Management

Distributed applications not in a vacuum usually involve distributed dependencies, such as mnesia. It's easy for us to handle their reconfiguration from the same callback on_state_change/2. For example, here is a detailed description of how to reconfigure mnesia on the fly in documentation cloister.

The main advantage of using cloister is that it performs all the necessary operations to rebuild the cluster after changing the topology under the hood. The application simply runs in an already prepared distributed environment, with all nodes connected, regardless of whether we know the IP addresses and therefore the names of the nodes in advance, or they have been dynamically assigned/changed. It requires absolutely no special docker configuration settings and from the application developer's point of view, there is no difference between running in a distributed environment or in a local :nonode@nohost. You can read more about this in documentation.

Although sophisticated handling of topology changes is possible through native implementation MyApp.Listener, there can always be edge cases where these library limitations and biased configuration approach prove to be a cornerstone in the implementation path. It's ok, just take the above libcluster, which is more versatile, or even handle the low-level cluster yourself. The goal of this code library is not to cover every possible scenario, but to use the most common scenario without unnecessary pain and cumbersome copy-pastes.

Note: in this place in the original there was the phrase “Happy clustering!”, And Yandex, which I translate (not to climb the dictionaries myself), offered me the option “Happy clustering!”. A better translation, perhaps, especially in the light of the current geopolitical situation, is impossible to imagine.

Source: habr.com