🥇 Compilable Configuration of a Distributed System

In this post, we’d like to share an interesting way of managing the configuration of a distributed system.
The configuration is directly represented in the Scala language in a type-safe manner. An example implementation is described in detail. Various aspects of the proposal are discussed, including its influence on the overall development process.

(in Russian)

Introduction

Building robust distributed systems requires the correct and coherent configuration across all nodes. A typical solution is to use a textual deployment description (such as Terraform, Ansible, or similar) along with automatically generated configuration files (often dedicated for each node/role). We also want to ensure that the same protocols and versions are used on each communicating node to avoid compatibility issues. In the JVM world, this means that at least the messaging library should be of the same version on all communicating nodes.

What about testing the system? Of course, we should have unit tests for all components before moving on to integration tests. To reliably extrapolate test results to the runtime environment, we must ensure that the versions of all libraries remain identical in both the runtime and testing environments.

When running integration tests, it’s often much easier to have the same classpath on all nodes. We just need to ensure that the same classpath is used during deployment. (Different classpaths can be used on different nodes, but it is more challenging to represent and deploy this configuration correctly.) To keep things simple, we will only consider identical classpaths on all nodes.

Configuration tends to evolve alongside the software. We typically use versions to identify various
stages of software evolution. It seems reasonable to manage configuration through version control while labeling different configurations. If there is only one configuration in production, we can use a single version as an identifier. Sometimes we may have multiple production environments, each requiring a separate configuration branch. Configurations can thus be labeled with both branch and version to uniquely identify each. Each branch label and version corresponds to a specific combination of distributed nodes, ports, external resources, and classpath library versions on each node. Here, we will only consider a single branch, identifying configurations by a three-component decimal version (1.2.3), similar to other artifacts.

In modern environments, configuration files are no longer modified manually. Typically, we generate
configuration files at deployment time and never touch them afterwards. One might wonder why we still use text format for configuration files. A viable option is to embed the configuration within a compilation unit and benefit from compile-time configuration validation.

In this post, we will explore the idea of maintaining the configuration within the compiled artifact.

Compilable configuration

In this section, we will discuss an example of static configuration. Two simple services — an echo service and the client of the echo service — are configured and implemented. Then, two distinct distributed systems featuring both services are instantiated: one for a single-node configuration and another for a two-node configuration.

A typical distributed system consists of multiple nodes, which can be identified using a type:

sealed trait NodeId
case object Backend extends NodeId
case object Frontend extends NodeId

or simply

case class NodeId(hostName: String)

or even

object Singleton
type NodeId = Singleton.type

These nodes perform various roles, run services, and must communicate with other nodes via TCP/HTTP connections.

For a TCP connection, at least a port number is required. We also need to ensure that the client and server use the same protocol. To model a connection between nodes, let’s declare the following class:

case class TcpEndPoint[Protocol](node: NodeId, port: Port[Protocol])

where Port is just an Int within the allowed range:

type PortNumber = Refined[Int, Closed[_0, W.`65535`.T]]

Refined types

See refined library. In short, it allows adding compile time constraints to other types. In this case Int is only allowed to have 16-bit values that can represent port numbers. There is no requirement to use this library for this configuration approach. It simply appears to fit very well.

For HTTP (REST) we might also need a path for the service:

type UrlPathPrefix = Refined[String, MatchesRegex[W.`"[a-zA-Z_0-9\/]*"`.T]]
case class PortWithPrefix[Protocol](portNumber: PortNumber, pathPrefix: UrlPathPrefix)

Phantom type

To identify the protocol during compilation, we are using the Scala feature of declaring a type argument Protocol that is not utilized in the class. It’s a so-called phantom type. At runtime, we rarely need an instance of the protocol identifier, hence we don't store it. During compilation, this phantom type provides additional type safety. We cannot pass a port with an incorrect protocol.

One of the most commonly used protocols is the REST API with JSON serialization:

sealed trait JsonHttpRestProtocol[RequestMessage, ResponseMessage]

where RequestMessage is the base type of messages that the client can send to the server and ResponseMessage is the response message from the server. Of course, we can create other protocol descriptions that specify the communication protocol with the desired precision.

For the purposes of this post, we will use a simpler version of the protocol:

sealed trait SimpleHttpGetRest[RequestMessage, ResponseMessage]

In this protocol, the request message is appended to the URL and the response message is returned as a plain string.

A service configuration could be described by the service name, a collection of ports, and some dependencies. There are a few possible ways to represent all these elements in Scala (for example, HList, algebraic data types). For the purposes of this post, we’ll use the Cake Pattern and represent combinable pieces (modules) as traits. (The Cake Pattern is not a requirement for this compilable configuration approach. It's just one possible implementation of the idea.)

Dependencies could be represented using the Cake Pattern as endpoints of other nodes:

  type EchoProtocol[A] = SimpleHttpGetRest[A, A]

  trait EchoConfig[A] extends ServiceConfig {
    def portNumber: PortNumber = 8081
    def echoPort: PortWithPrefix[EchoProtocol[A]] = PortWithPrefix[EchoProtocol[A]](portNumber, "echo")
    def echoService: HttpSimpleGetEndPoint[NodeId, EchoProtocol[A]] = providedSimpleService(echoPort)
  }

The Echo service only requires a configured port. We declare that this port supports the echo protocol. Note that we do not need to specify a particular port at this moment because traits allow abstract method declarations. If we use abstract methods, the compiler will require an implementation in a configuration instance. Here, we have provided the implementation (8081) and it will be used as the default value if we skip it in a concrete configuration.

We can declare a dependency in the configuration of the Echo service client:

  trait EchoClientConfig[A] {
    def testMessage: String = "test"
    def pollInterval: FiniteDuration
    def echoServiceDependency: HttpSimpleGetEndPoint[_, EchoProtocol[A]]
  }

The dependency has the same type as the echoService. In particular, it demands the same protocol. Therefore, we can be sure that if we connect these two dependencies, they will work correctly.

Service implementation

A service needs a function to start and gracefully shut down. (The ability to shut down a service is critical for testing.) Again, there are a few options for specifying such a function for a given configuration (for instance, we could use type classes). For this post, we will again use the Cake Pattern. We can represent a service using cats.Resource which already provides bracketing and resource release. In order to acquire a resource we should provide a configuration and some runtime context. So the service starting function might look like:

  type ResourceReader[F[_], Config, A] = Reader[Config, Resource[F, A]]

  trait ServiceImpl[F[_]] {
    type Config
    def resource(
      implicit
      resolver: AddressResolver[F],
      timer: Timer[F],
      contextShift: ContextShift[F],
      ec: ExecutionContext,
      applicative: Applicative[F]
    ): ResourceReader[F, Config, Unit]
  }

where

Config — type of configuration that is required by this service starter
AddressResolver — a runtime object that has the ability to obtain real addresses of other nodes (keep reading for details).

the other types comes from cats:

F[_] — effect type (In the simplest case F[A] could be just () => A. In this post we’ll use cats.IO.)
Reader[A,B] — is more or less a synonym for a function A => B
cats.Resource — has ways to acquire and release
Timer — allows to sleep/measure time
ContextShift — analog of ExecutionContext
Applicative — wrapper of functions in effect (almost a monad) (we might eventually replace it with something else)

Using this interface we can implement a few services. For instance, a service that does nothing:

  trait ZeroServiceImpl[F[_]] extends ServiceImpl[F] {
    type Config  Resource.pure[F, Unit](()))
  }

(See Source code for other services implementations — echo service,
echo client and lifetime controllers.)

A node is a single object that runs a few services (starting a chain of resources is enabled by Cake Pattern):

object SingleNodeImpl extends ZeroServiceImpl[IO]
  with EchoServiceService
  with EchoClientService
  with FiniteDurationLifecycleServiceImpl
{
  type Config = EchoConfig[String] with EchoClientConfig[String] with FiniteDurationLifecycleConfig
}

Note that in the node we specify the exact type of configuration that is needed by this node. Compiler won’t let us build the object (Cake) with insufficient type, because each service trait declares a constraint on the Config type. Also we won’t be able to start the node without providing complete configuration.

Node address resolution

In order to establish a connection we need a real host address for each node. It might be known later than other parts of the configuration. Hence, we need a way to supply a mapping between node id and its actual address. This mapping is a function:

case class NodeAddress[NodeId](host: Uri.Host)
trait AddressResolver[F[_]] {
  def resolve[NodeId](nodeId: NodeId): F[NodeAddress[NodeId]]
}

There are a few possible ways to implement such a function.

If we know actual addresses before deployment, during node hosts instantiation, then we can generate Scala code with the actual addresses and run the build afterwards (which performs compile time checks and then runs integration test suite). In this case our mapping function is known statically and can be simplified to something like a Map[NodeId, NodeAddress].
Sometimes we obtain actual addresses only at a later point when the node is actually started, or we don’t have addresses of nodes that haven’t been started yet. In this case we might have a discovery service that is started before all other nodes and each node might advertise its address in that service and subscribe to dependencies.
If we can modify /etc/hosts, we can use predefined host names (like my-project-main-node and echo-backend) and just associate this name with ip address at deployment time.

In this post we don’t cover these cases in more details. In fact, in our toy example all nodes will have the same IP address — 127.0.0.1.

In this post we’ll consider two distributed system layouts:

Single node layout, where all services are placed on a single node.
Two node layout, where service and client are on different nodes.

The configuration for a single node layout is as follows:

Single node configuration

object SingleNodeConfig extends EchoConfig[String] 
  with EchoClientConfig[String] with FiniteDurationLifecycleConfig
{
  case object Singleton \/\/ identifier of the single node 
  \/\/ configuration of server
  type NodeId = Singleton.type
  def nodeId = Singleton

  \/** Type safe service port specification. *\/ 
  override def portNumber: PortNumber = 8088

  \/\/ configuration of client

  \/** We'll use the service provided by the same host. *\/ 
  def echoServiceDependency = echoService

  override def testMessage: UrlPathElement = "hello"

  def pollInterval: FiniteDuration = 1.second

  \/\/ lifecycle controller configuration
  def lifetime: FiniteDuration = 10500.milliseconds \/\/ additional 0.5 seconds so that there are 10 requests, not 9.
}

Here we create a single configuration that extends both server and client configuration. We also configure a lifecycle controller that will normally terminate the client and server after lifetime the interval passes.

The same set of service implementations and configurations can be used to create a system layout with two separate nodes. We just need to create two separate node configs with the appropriate services:

Two nodes configuration

  object NodeServerConfig extends EchoConfig[String] with SigTermLifecycleConfig
  {
    type NodeId = NodeIdImpl

    def nodeId = NodeServer

    override def portNumber: PortNumber = 8080
  }

  object NodeClientConfig extends EchoClientConfig[String] with FiniteDurationLifecycleConfig
  {
    \/\/ NB! dependency specification
    def echoServiceDependency = NodeServerConfig.echoService

    def pollInterval: FiniteDuration = 1.second

    def lifetime: FiniteDuration = 10500.milliseconds \/\/ additional 0.5 seconds so that there are 10 requests, not 9.

    def testMessage: String = "dolly"
  }

See how we specify the dependency. We mention the service provided by the other node as a dependency of the current node. The type of dependency is checked because it involves a phantom type that describes the protocol. At runtime, we’ll have the correct node ID. This is one of the important aspects of the proposed configuration approach. It allows us to set the port only once and ensure that we are referencing the correct port.

Two nodes implementation

For this configuration, we use exactly the same service implementations. No changes at all. However, we create two different node implementations that contain different sets of services:

  object TwoJvmNodeServerImpl extends ZeroServiceImpl[IO] with EchoServiceService with SigIntLifecycleServiceImpl {
    type Config = EchoConfig[String] with SigTermLifecycleConfig
  }

  object TwoJvmNodeClientImpl extends ZeroServiceImpl[IO] with EchoClientService with FiniteDurationLifecycleServiceImpl {
    type Config = EchoClientConfig[String] with FiniteDurationLifecycleConfig
  }

The first node implements the server and only needs the server-side config. The second node implements the client and needs a different part of the config. Both nodes require some lifetime specification. For the purposes of this post, the service node will have an infinite lifetime that could be terminated using SIGTERM, while the echo client will terminate after the configured finite duration. See the starter application for details.

Overall development process

Let’s see how this approach changes the way we work with configuration.

The configuration as code will be compiled and produce an artifact. It seems reasonable to separate the configuration artifact from other code artifacts. Often we can have a multitude of configurations on the same codebase. And of course, we can have multiple versions of various configuration branches. In a configuration, we can select particular versions of libraries, which will remain constant whenever we deploy this configuration.

A configuration change becomes a code change. So it should be covered by the same quality assurance process:

Ticket -> PR -> review -> merge -> continuous integration -> continuous deployment

The following are the consequences of the approach:

The configuration is consistent for a specific instance of the system. It appears that there is no way to create an incorrect connection between nodes.
Changing the configuration in just one node is not straightforward. Logging in to modify text files seems impractical. Thus, configuration drift becomes less likely.
Making small configuration changes is not easy.
Most configuration changes will adhere to the same development process and will undergo some review.

Is a separate repository needed for production configuration? The production configuration may contain sensitive information that we wish to keep away from many people. Therefore, it might be worthwhile to maintain a separate repository with restricted access containing the production configuration. We could divide the configuration into two parts—one containing most public parameters of production and the other containing the confidential parts. This would allow most developers access to the majority of parameters while limiting access to truly sensitive information. This can be achieved easily using intermediate traits with default parameter values.

Variations

Let’s examine the pros and cons of the suggested approach against other configuration management techniques.

First, we’ll outline a few alternatives regarding different aspects of the proposed way to handle configuration:

Text file on the target machine.
Centralized key-value storage (like etcd/zookeeper).
Subprocess components that could be reconfigured/restarted without restarting the entire process.
Configuration external to artifacts and version control.

A text file offers some flexibility for ad-hoc fixes. An administrator can log in to the target node, make a change, and simply restart the service. This might not be ideal for larger systems. No trace of the change remains. The change isn’t reviewed by another set of eyes. It can be challenging to determine what caused the change. It remains untested. From the distributed system's perspective, an administrator might simply forget to update the configuration in one of the other nodes.

(By the way, if there eventually arises a need to start using text config files, we’ll only need to add a parser + validator to produce the same Config type, and that would suffice to begin using text configs. This also indicates that the complexity of compile-time configuration is slightly lower than that of text-based configs, as we need some additional code for the text version.)

Centralized key-value storage is an effective mechanism for distributing application meta parameters. Here, we need to consider what we regard as configuration values and what is merely data. Given a function C => A => B we typically refer to values that change infrequently as "configuration," while data that changes frequently is just input data. Configuration should be supplied to the function before the data. C . Based on this idea, we can state that the expected frequency of changes can be employed to differentiate configuration data from regular data. Additionally, data typically originates from one source (the user), whereas configuration comes from another (the administrator). Managing parameters that can change after process initialization increases application complexity. For such parameters, we need to manage their delivery mechanism, parsing and validation, and handle incorrect values. Therefore, to reduce program complexity, it's advisable to limit the number of parameters that can change at runtime (or even eliminate them entirely). A — just input data. A. Given this idea we can say that it’s expected frequency of changes what could be used to distinguish configuration data from just data.

From the perspective of this post, we should distinguish between static and dynamic parameters. If the service logic requires infrequent changes to certain parameters at runtime, then we can call them dynamic parameters. Otherwise, they are static and can be configured using the proposed approach. For dynamic reconfiguration, other methods may be necessary. For example, parts of the system might need to be restarted with new configuration parameters, similar to restarting individual processes in a distributed system.
(In my humble opinion, it is best to avoid runtime reconfiguration, as it increases system complexity.
It may be more straightforward to rely on OS support for restarting processes. However, this might not always be feasible.)

An important consideration when using static configuration that sometimes leads people to contemplate dynamic configuration (without other reasons) is service downtime during configuration updates. Indeed, if we need to modify the static configuration, the system must be restarted for the new values to take effect. The downtime requirements vary across different systems, so it might not always be critical. If it is critical, we need to plan for any system restarts in advance. For instance, we could implement AWS ELB connection draining. In this scenario, whenever we need to restart the system, we start a new instance of the system in parallel, then switch the ELB to it, allowing the old system to finish servicing existing connections.

What about keeping configuration within a versioned artifact versus separately? Keeping configuration within an artifact usually means that it has undergone the same quality assurance process as other artifacts. Therefore, one can be confident that the configuration is of high quality and reliable. On the other hand, configuration in a separate file means that there are no records of who made changes and for what reasons. Is this important? We believe that for most production systems, having a stable and high-quality configuration is preferable.

The version of the artifact allows us to determine when it was created, what values it contains, what features are enabled or disabled, and who was responsible for each change in the configuration. It may require some effort to keep configuration within an artifact, and it's a design choice to consider.

Pros & cons

Here, we want to highlight some advantages while also discussing some disadvantages of the proposed approach.

Advantages

Features of the compilable configuration of a complete distributed system:

Static configuration checks. This provides a high level of confidence that the configuration adheres to type constraints.
Rich configuration language. Typically, other configuration approaches are limited to variable substitution at most.
Using Scala allows one to leverage a wide array of language features to enhance configuration. For instance, we can use traits to provide default values, objects to define different scopes, and we can refer to vals defined only once in the outer scope (DRY). It is possible to use literal sequences or instances of certain classes (Seq, Map, etc.).
DSL. Scala has strong support for DSL writers. These features can be utilized to establish a configuration language that is more user-friendly and convenient, ensuring that the final configuration is at least readable by domain users.
Integrity and coherence across nodes. One of the advantages of having configuration for the entire distributed system in one place is that all values are defined strictly once and then reused wherever needed. Also, type-safe port declarations ensure that in all valid configurations, the system’s nodes will communicate uniformly. There are explicit dependencies between nodes, which reduces the likelihood of forgetting to provide certain services.
High quality of changes. The overall practice of passing configuration changes through the standard PR process enforces high-quality standards in configuration as well.
Simultaneous configuration changes. Whenever we make any changes in the configuration, automatic deployment ensures that all nodes are updated.
Application simplification. The application doesn’t need to parse and validate configuration or handle incorrect configuration values. This simplifies the overall application. (Some complexity may increase in the configuration itself, but this is a deliberate trade-off for safety.) It’s straightforward to revert to ordinary configuration — just add the missing elements. Getting started with compiled configuration is easier, allowing the implementation of additional components to be postponed to a later time.
Versioned configuration. Since configuration changes follow the same development process, we obtain an artifact with a unique version. This allows us to revert the configuration if needed. We can even deploy a configuration that was used a year ago, and it will function exactly the same way. Stable configuration enhances the predictability and reliability of the distributed system. The configuration is fixed at compile time and cannot be easily altered in a production environment.
Modularity. The proposed framework is modular, and modules can be combined in various ways to
support different configurations (setups/layouts). In particular, it's possible to have a small scale single-node layout and a large-scale multi-node setting. It's reasonable to have multiple production layouts.
Testing. For testing purposes, one might implement a mock service and use it as a dependency in a type-safe manner. Several different testing layouts with various components replaced by mocks could be maintained concurrently.
Integration testing. In distributed systems, it can sometimes be challenging to run integration tests. By using the described approach for type-safe configuration of the entire distributed system, we can run all distributed components on a single server in a controllable manner. Emulating the situation
when one of the services becomes unavailable is straightforward.

Disadvantages

The compiled configuration approach differs from standard configuration and might not meet all needs. Here are some disadvantages of the compiled configuration:

Static configuration. It might not be suitable for all applications. In some instances, there's a need to quickly fix the configuration in production without adhering to safety precautions. This approach complicates matters. After any configuration change, compilation and redeployment are required. This serves as both a feature and a burden.
Configuration generation. When configurations are generated by some automation tool, this approach necessitates subsequent compilation (which may fail). It may take extra effort to integrate this step into the build system.
Instruments. Many tools currently in use rely on text-based configs. Some of them
won’t function when the configuration is compiled.
A shift in mindset is necessary. Developers and DevOps professionals are accustomed to text configuration files. The concept of compiling configuration might seem unusual to them.
Before introducing compilable configuration, a high-quality software development process is essential.

There are some limitations of the implemented example:

If we provide extra configuration that is not required by the node implementation, the compiler won’t assist us in detecting the missing implementation. This issue could be addressed by using HList or ADTs (case classes) for node configuration instead of traits and the Cake Pattern.
We have to provide some boilerplate in the config file: (package, import, object declarations;
override def‘s for parameters that have default values). This could be partially resolved using a DSL.
This post does not cover the dynamic reconfiguration of clusters of similar nodes.

Conclusion

In this post, we discuss the concept of directly representing configuration within the source code in a type-safe manner. This approach can be utilized across various applications as a replacement for XML and other text-based configurations. Although our example is implemented in Scala, it can also be adapted for other compiled languages such as Kotlin, C#, Swift, and more. You can experiment with this method in a new project and, if it doesn't fit well, revert to traditional configurations.

Naturally, compiled configuration demands a high-quality development process. In return, it promises to deliver equally robust and reliable configurations.

This approach can be expanded in various ways:

Macros can be employed to validate configurations and trigger failures at compile time if any business logic constraints are violated.
A DSL (Domain-Specific Language) could be created to represent configuration in a manner that is user-friendly for domain experts.
Dynamic resource management with automatic configuration adjustments. For example, when adjusting the number of cluster nodes, we may want (1) the nodes to receive slightly modified configurations; (2) the cluster manager to be updated with new node information.

Thank you

I would like to express my gratitude to Andrey Saksonov, Pavel Popov, and Anton Nehaev for their insightful feedback on the draft of this post, which helped clarify my ideas.

Source: habr.com

Compilable configuration of a distributed system