On February 26, we held an Apache Ignite GreenSource meetup, featuring contributors to the open source project. . A significant event in the life of this community was the restructuring of the , which allows for the deployment of user microservices directly within the Ignite cluster. This complex process was discussed at the meetup by , a software engineer and contributor to Apache Ignite for over two years.
Let's start with what Apache Ignite actually is. It's a database that serves as a distributed Key/Value store with support for SQL, transactions, and caching. Additionally, Ignite allows for the deployment of user services directly within the Ignite cluster. Developers gain access to all tools provided by Ignite — distributed data structures, Messaging, Streaming, Compute, and Data Grid. For instance, by using Data Grid, the issues related to managing a separate infrastructure for data storage are eliminated, along with the associated overhead.
By using the Service Grid API, a service can be deployed simply by specifying the deployment scheme and, accordingly, the service itself in the configuration.
Typically, the deployment scheme specifies the number of instances that should be deployed on the cluster nodes. There are two common deployment schemes. The first is Cluster Singleton: at any given time, exactly one instance of the user service will be available in the cluster. The second is Node Singleton: one instance of the service is deployed on each node of the cluster.
Users can also specify the number of service instances across the entire cluster and define a predicate for filtering suitable nodes. In such a scenario, the Service Grid will autonomously calculate the optimal distribution for deploying services.
Additionally, there is a feature called Affinity Service. Affinity is a function that determines the connection of keys to partitions and associates partitions with nodes in the topology. Based on the key, one can identify the primary node where the data is stored. This way, you can associate your own service with the key and the cache of the affinity function. In case the affinity function changes, an automatic redeployment will occur. Thus, the service will always be located near the data it needs to manipulate, thereby reducing the overhead of accessing information. This scheme can be called a kind of collocated computing.
Now that we have discussed the advantages of the Service Grid, let's delve into its development history.
What came before
The previous implementation of the Service Grid was based on a transactional replicated system cache called Ignite. By "cache" in Ignite, we mean a storage. So, it is not something temporary, as one might think. Although the cache is replicated and each node contains the entire dataset, internally the cache has a partitioned representation. This is related to the optimization of storage.
What happened when a user wanted to deploy a service?
- All nodes in the cluster subscribed to data updates in the storage using an embedded mechanism called Continuous Query.
- The initiating node, under a read-committed transaction, made an entry in the database containing the service configuration, including the serialized instance.
- Upon receiving a notification of a new entry, the coordinator calculated the distribution based on the configuration. The obtained object was written back to the database.
- If a node was included in the distribution, the coordinator had to deploy it.
What we were dissatisfied with
At some point, we concluded that this was not an acceptable way to work with services. There were several reasons.
If an error occurred during deployment, it could only be identified from the logs of the node where it happened. There was only asynchronous deployment, so after returning control to the user from the deployment method, there was a delay before the service started—during this time, the user could not manage anything. To further develop the Service Grid, implement new features, attract new users, and make life easier for everyone, something needed to change.
When designing the new Service Grid, we primarily aimed to ensure synchronous deployment: as soon as control is returned to the user from the API, they can immediately use the services. We also wanted to give the initiator the ability to handle deployment errors.
In addition, we aimed to simplify implementation by moving away from transactions and rebalancing. Despite the fact that cache is replicable and there is no balancing, issues arose during large deployments with numerous nodes. When topology changes, nodes need to exchange information, and during a large deployment, this data can be quite substantial.
When the topology was unstable, the coordinator needed to recalculate service distribution. Moreover, working with transactions in an unstable topology can lead to hard-to-predict errors.
Issues
What are global changes without accompanying problems? The first problem was the change in topology. It is essential to understand that at any moment, even during the deployment of a service, a node can join or leave the cluster. Furthermore, if a node joins the cluster during deployment, all information about services needs to be consistently transmitted to the new node. This includes not only what has already been deployed but also current and future deployments.
This is just one of the problems that can be compiled into a separate list:
- How to deploy statically configured services when starting a node?
- What to do if a node exits the cluster while hosting services?
- What to do if the coordinator changes?
- What to do if a client reconnects to the cluster?
- Should activation/deactivation requests be processed and how?
- What if the cache destruction was called, and we have affinity services bound to it?
And this is far from all.
Solution
As a target, we chose the Event Driven approach by implementing process communication via messages. Ignite already has two components that allow nodes to send messages to each other — communication-spi and discovery-spi.
Communication-spi allows nodes to communicate directly and send messages. It is well-suited for transmitting large volumes of data. Discovery-spi enables messages to be sent to all nodes in the cluster. In the standard implementation, this is done using a 'ring' topology. There is also integration with Zookeeper, in which case a 'star' topology is used. It's important to note that discovery-spi provides guarantees that messages will be delivered in the correct order to all nodes.
Let's consider the deployment protocol. All user requests for deployment and undeployment are sent via discovery-spi. This provides the following guarantees:
- The request will be received by all nodes in the cluster. This allows processing to continue during coordinator changes. It also means that for a single message, each node will have all the necessary metadata, such as service configuration and its serialized instance.
- Strict message delivery order allows resolving configuration conflicts and competing requests.
- Since a node's entry into the topology is also handled via discovery-spi, all necessary data for working with services will reach the new node.
Upon receiving the request, nodes in the cluster validate it and create tasks for processing. These tasks are queued and then processed in another thread by a separate worker. This is implemented this way because deployment can take a significant amount of time, and blocking the expensive discovery stream is unacceptable.
All requests in the queue are processed by the deployment manager. It has a special worker that pulls a task from the queue and initializes it to start the deployment. After that, the following actions occur:
- Each node independently calculates the distribution using a new deterministic assignment function.
- Nodes create a message containing deployment results and send it to the coordinator.
- The coordinator aggregates all messages and forms the result of the entire deployment process, which is sent via discovery-spi to all nodes in the cluster.
- Upon receiving the result, the deployment process completes, after which the task is removed from the queue.
New event-driven design: org.apache.ignite.internal.processors.service.IgniteServiceProcessor.java
If an error occurs during deployment, the node immediately includes this error in the message sent to the coordinator. After aggregating the messages, the coordinator will have information about all errors during the deployment and will send this message via discovery-spi. Error information will be available on any node in the cluster.
This algorithm processes all important events in the Service Grid. For example, a change in topology is also a message via discovery-spi. Overall, when compared to what was before, the protocol has proven to be quite lightweight and reliable. Reliable enough to handle any situation during deployment.
What happens next
Now about the plans. Any significant enhancement in the Ignite project is carried out as an initiative for improving Ignite, known as an IEP. The redesign of the Service Grid also has an IEP — with the amusing name 'Oil Change in the Service Grid.' But in reality, we didn't change the oil in the engine; we changed the engine entirely.
We've divided the tasks in the IEP into 2 phases. The first phase is a significant one, which involves reworking the deployment protocol. It has already been merged into the master, so you can try the new Service Grid that will appear in version 2.8. The second phase includes many other tasks:
- Hot redeploy
- Versioning of services
- Improving fault tolerance
- Thin client
- Monitoring tools and metrics calculation
Finally, we can recommend the Service Grid for building fault-tolerant high-availability systems. We also invite you to join us in the and to share your experience. Your experience is truly valuable to the community; it will help understand where to move forward and how to develop the component in the future.
Source: habr.com
