{"id":31124,"date":"2019-10-31T21:39:36","date_gmt":"2019-10-31T18:39:36","guid":{"rendered":"https:\/\/prohoster.info\/blog\/stroitelnye-bloki-raspredelennyh-prilozhenij-vtoroe-priblizhenie\/"},"modified":"2019-10-31T21:39:36","modified_gmt":"2019-10-31T18:39:36","slug":"stroitelnye-bloki-raspredelennyh-prilozhenij-vtoroe-priblizhenie","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/stroitelnye-bloki-raspredelennyh-prilozhenij-vtoroe-priblizhenie","title":{"rendered":"Building Blocks of Distributed Applications. A Second Approach","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Announcement<\/strong><\/p>\n

<\/p>\n

Colleagues, I plan to release another series of articles on the design of mass service systems in mid-summer: \"Experiment VTrade\" \u2014 an attempt to write a framework for trading systems. The series will cover the theory and practice of building an exchange, auction, and store. At the end of the article, I invite you to vote for the topics that interest you the most.
\n<\/em><\/p>\n

<\/p>\n

\"Building<\/p>\n

<\/p>\n

This is the concluding article in the series on distributed reactive applications in Erlang\/Elixir. In first article)<\/a><\/noindex> you can find the theoretical foundations of reactive architecture. The second article<\/a><\/noindex> illustrates the main patterns and mechanisms for building such systems.<\/p>\n

<\/p>\n

Today, we will raise questions about the development of the codebase and the projects as a whole. <\/p>\n

<\/a><\/noindex><\/p>\n

Service Organization<\/h2>\n

<\/p>\n

In real life, when developing a service, it is often necessary to combine several interaction patterns in one controller. For example, the users service, which handles user profile management tasks for the project, should respond to req-resp requests and notify about profile updates via pub-sub. This case is quite simple: there is one controller behind messaging, implementing the service logic and publishing updates.<\/p>\n

<\/p>\n

The situation becomes more complicated when we need to implement a fault-tolerant distributed service. Let\u2019s imagine that the requirements for users have changed: <\/p>\n

<\/p>\n

    \n
  1. now the service must handle requests on 5 nodes in the cluster, <\/li>\n
  2. be capable of performing background processing tasks, <\/li>\n
  3. and also be able to dynamically manage subscription lists for profile updates.<\/li>\n<\/ol>\n

    <\/p>\n

    Note:<\/em> We are not addressing the issue of consistent data storage and replication. Let\u2019s assume that these issues were resolved earlier, and there already exists a reliable and scalable storage layer in the system, and handlers have mechanisms for interacting with it.<\/p>\n

    <\/p>\n

    The formal description of the users service has become more complicated. From a programmer\u2019s perspective, due to the use of messaging, changes are minimal. To meet the first requirement, we need to set up load balancing at the req-resp exchange point. <\/p>\n

    <\/p>\n

    The need for processing background tasks often arises. In users, this may include verifying user documents, processing uploaded multimedia, or syncing data with social networks. These tasks need to be distributed within the cluster and monitored for progress. Therefore, we have two options: either use the task distribution template from the previous article, or, if it doesn't fit, write a custom task scheduler that will manage the pool of handlers in the necessary way. <\/p>\n

    <\/p>\n

    Point 3 requires extending the pub-sub template. To implement this, after creating the pub-sub exchange point, we need to additionally launch the controller of this point within our service. Thus, we essentially extract the logic of subscription and unsubscription handling from the messaging layer to the implementation in users.<\/p>\n

    <\/p>\n

    As a result, the decomposition of the task showed that in order to meet the requirements, we need to start 5 instances of the service on different nodes and create an additional entity \u2013 a pub-sub controller responsible for subscriptions.
    \nLaunching 5 handlers does not require changing the service code. The only additional action is configuring the load balancing rules at the exchange point, which we will discuss shortly.
    \nAdditionally, there is a further complication: the pub-sub controller and the custom task scheduler must operate as a single instance. Again, the messaging service, being fundamental, must provide a leader selection mechanism.<\/p>\n

    <\/p>\n

    Leader Selection<\/h3>\n

    <\/p>\n

    In distributed systems, leader selection is the procedure for appointing a single process responsible for scheduling distributed processing of some load. <\/p>\n

    <\/p>\n

    In systems resistant to centralization, universal algorithms and consensus-based algorithms, such as Paxos or Raft, are applied.
    \nSince messaging acts as a broker and central element, it is aware of all the service controllers \u2013 candidates for leadership. Messaging can appoint a leader without conducting a vote.<\/p>\n

    <\/p>\n

    All services receive a system message after starting up and connecting to the exchange point #'$leader'{exchange = ?EXCHANGE, pid = LeaderPid, servers = Servers}<\/code>. In the event that LeaderPid<\/code> matches the pid<\/code> current process, it is appointed as the leader, and the list Servers<\/code> includes all nodes and their parameters.
    \nWhen a new node is added and an active node is disabled in the cluster, all service controllers receive #'$slave_up'{exchange = ?EXCHANGE, pid = SlavePid, options = SlaveOpts}<\/code> and #'$slave_down'{exchange = ?EXCHANGE, pid = SlavePid, options = SlaveOpts}<\/code> respectively.<\/p>\n

    <\/p>\n

    As a result, all components are aware of all changes, and there is guaranteed to be one leader in the cluster at all times.<\/p>\n

    <\/p>\n

    Mediators<\/h2>\n

    <\/p>\n

    To implement complex distributed processing tasks and optimize existing architecture, it is convenient to use mediators.
    \nTo avoid changing the service code and to handle additional processing, routing, or logging of messages, a proxy handler can be introduced in front of the service to manage all additional tasks.<\/p>\n

    <\/p>\n

    A classic example of pub-sub optimization is a distributed application with a business core generating update events, such as a market price change, and an access layer \u2014 N servers providing a websocket API for web clients.
    \nIf handled directly, client servicing would look like this:<\/p>\n

    <\/p>\n