Cloister → simple management of the OTP cluster

Almost every successful business application eventually reaches a stage where horizontal scaling is required. In many cases, you can simply launch a new instance to reduce the average load. However, there are less trivial scenarios where we must ensure that different nodes are aware of each other and carefully distribute the workload.

It so happens that erlang, which we chose for its pleasant syntax and the hype surrounding it, has excellent support for distributed systems. In theory, this sounds entirely trivial:

Message passing between processes on different nodes, as well as between links and monitors, is transparent […]

In practice, it's a bit more complex. The distributed erlang was developed when "container" meant a large metal box for transport and "docker" was simply a synonym for dockworker. In IP4 there were many unused addresses, network outages were generally caused by rats gnawing on cables, and the average uptime of a production system was measured in decades.

Now we’re all unimaginably self-sufficient, packaged, and running distributed erlang in an environment where dynamic IP addresses are allocated at the whim of great chance, and nodes can appear and disappear at the scheduler's whim. To avoid a heap of boilerplate code in every project running distributed erlang, in order to cope with a hostile environment, some assistance is required.

Note: I know there’s libcluster. It's really cool, has over a thousand stars, the author is well-known in the community, and all that. If you are satisfied with the ways this package offers to create and maintain a cluster — I'm happy for you. Unfortunately, I need much more. I want to control the setup in detail and not be a passive observer in the theater of cluster reformation.

Requirements

What I personally needed was a library that would take care of cluster management and have the following features:

• Transparent operation with both a hard-coded list of nodes and dynamic discovery via services erlang;
• Full-featured callback on every topology change (node here, node there, network instability, splits);
• A transparent interface for launching a cluster with both long and short names, just like with :nonode@nohost;
• out-of-the-box Docker support, without the need to write infrastructure code.

This means that after I have tested the application locally in :nonode@nohost, or in an artificially distributed environment using test_cluster_task, I just want to run docker-compose up --scale my_app=3 and see it running three instances in Docker without any code changes. I also want dependent applications, such as mnesia — when the topology changes, to rebuild the cluster live in the background without any extra nudge from the application.

Cloister is not designed as a library that can do everything from cluster support to making coffee. It is not a silver bullet aiming to cover all possible cases or to be an academically complete solution in the sense that theorists from CS attribute to this term. This library is intended to serve a very specific purpose, but to execute its rather limited tasks perfectly. That purpose will be to ensure full transparency between the local development environment and the distributed elastic environment, full of hostile containers.

The chosen approach

Cloister is to run as an application, although experienced users may work with assembly and maintenance of the cluster manually by launching Cloister.Manager in the supervisor tree of the target application.

When running as an application, the library relies on config, from which it reads the following essential values:

config :cloister,
  otp_app: :my_app,
  sentry: :"cloister.local", # or ~w|n1@foo n2@bar|a
  consensus: 3,              # number of nodes to consider
                             #    the cluster is up
  listener: MyApp.Listener   # listener to be called when
                             #    the ring has changed

The above parameters mean literally the following: Cloister used for the OTP application :my_app, uses Erlang service discovery to connect nodes, at least three, and MyApp.Listener module (implementing @behaviour Cloister.Listener) is set up to receive notifications of topology changes. A detailed description of the full configuration can be found in the documentation.

With this configuration, the application Cloister it will be will start gradually., delaying the main application startup process until consensus is reached (three nodes are connected and linked, as shown in the example above.) This allows the main application to assume that when it starts, the cluster is already available. With every change in topology (which will happen frequently since nodes do not start fully synchronously), a handler will be triggered MyApp.Listener.on_state_change/2. In most cases, we take action when we receive a message with the status %Cloister.Monitor{status: :up}, which means: 'Hello, the cluster is assembled.'

In most cases, setting consensus: 3 is optimal, because even if we expect more nodes to connect, the callback will go through status: :rehashing → status: :up on any newly added or removed node.

When running in development mode, it is sufficient to simply set consensus: 1 and Cloister which will happily skip the cluster assembly wait, seeing :nonode@nohost, or :node@host, or :node@host.domain — depending on how the node was configured (:none | :shortnames | :longnames).

Managing Distributed Applications

Distributed applications do not operate in a vacuum; they usually include distributed dependencies, such as mnesia. We can easily handle their reconfiguration from the same callback on_state_change/2. Here, for example, is a detailed explanation of how to reconfigure mnesia on the fly in the documentation Cloister.

The main advantage of using Cloister is that it performs all necessary cluster rebuilding operations after a topology change under the hood. The application simply starts in an already prepared distributed environment, with all connected nodes, regardless of whether we already know their IP addresses and, therefore, node names in advance or they have been dynamically assigned/changed. This requires exactly no special Docker configuration settings, and from the developer's perspective, there is no difference between running in a distributed environment or locally on :nonode@nohost. More details on this can be found in the documentation.

Although complex topology change handling is possible through a custom implementation MyApp.Listener, there will always be corner cases where these library limitations and a biased approach to configuration will become a stumbling block to implementation. That's okay, just consider the above-mentioned libclusterwhich is more versatile, or even manage a low-level cluster on your own. The goal of this code library is not to cover all possible scenarios, but to optimize the most common scenario without unnecessary pain and cumbersome copy-pasting.

Note: the original phrase here was 'Happy clustering!', and Yandex, which I am translating (it’s not like I can just look it up in dictionaries), suggested the option 'Happy clustering!'. A better translation, especially in light of the current geopolitical situation, is simply unimaginable.

Source: habr.com