Fault tolerance and high availability are vast topics, so let's dedicate separate articles to RabbitMQ and Kafka. This article focuses on RabbitMQ, while the next one will cover Kafka in comparison to RabbitMQ. It's a lengthy read, so get comfortable.
We will explore fault tolerance strategies, consistency, and high availability (HA), as well as the trade-offs that come with each strategy. RabbitMQ can operate on a cluster of nodes, classifying it as a distributed system. When discussing distributed systems, we often talk about consistency and availability.
These concepts describe how a system behaves during failures. Network connection failures, server crashes, hard drive failures, temporary server unavailability due to garbage collection, packet loss, or network slowdowns can all lead to data loss or conflicts. It turns out that it is virtually impossible to create a system that is both completely consistent (without data loss, without data discrepancies) and available (acceptable for read and write operations) for all types of failures.
We will see that consistency and availability exist at opposite ends of the spectrum, and you need to choose which direction to optimize for. The good news is that with RabbitMQ, such choices are possible. You have those 'nerdy' levers to shift the balance towards greater consistency or greater availability.
Particular attention will be given to configurations that lead to data loss due to acknowledged messages. There is a chain of responsibility between publishers, brokers, and consumers. Once a message is handed over to the broker, it is their job not to lose it. When the broker acknowledges the receipt of a message to the publisher, we do not expect that it will be lost. However, we will see that this can indeed happen depending on your broker and publisher's configuration.
Single-node resilience primitives
Resilient queues/routing
In RabbitMQ, there are two types of queues: durable and non-durable. All queues are stored in the Mnesia database. Durable queues are redeclared when the node starts and thus survive restarts, system crashes, or server failures (as long as the data is preserved). This means that as long as you declare the routing (exchange) and queue as durable, the queue/routing infrastructure will return to operational mode.
Non-durable queues and routing are deleted upon restarting the node.
Durable messages
Just because a queue is durable does not mean that all its messages will survive the node's restart. Only messages marked by the publisher as durable (persistent) will be restored. Durable messages do create additional load on the broker, but if message loss is unacceptable, there is no other option.
Fig. 1. Durability Matrix
Clustering with Queue Mirroring
To survive a broker loss, we need redundancy. We can combine multiple RabbitMQ nodes into a cluster and then add additional redundancy by replicating queues across several nodes. Thus, if one node fails, we do not lose data and remain accessible.
Queue mirroring:
- one master queue that receives all write and read commands
- one or more mirrors that receive all messages and metadata from the master queue. These mirrors exist not for scaling, but solely for redundancy.
Fig. 2. Queue Mirroring
Mirroring is set up with the appropriate policy. In it, you can choose the replication factor and even the nodes on which the queue should be located. Examples:
ha-mode: allha-mode: exactly, ha-params: 2(one master and one mirror)ha-mode: nodes, ha-params: rabbit@node1, rabbit@node2
Acknowledgment to the Publisher
To achieve consistent message recording, publisher confirmations are necessary. Without them, there is a risk of message loss. Confirmation is sent to the publisher after the message is written to disk. RabbitMQ writes messages to disk not upon receipt, but on a periodic basis, typically within a few hundred milliseconds. When a queue is mirrored, confirmation is sent only after all mirrors have also written their copy of the message to disk. This means that using confirmations adds latency, but if data security is important, they are essential.
Fault-tolerant queue
When the broker shuts down or crashes, all leading queues (masters) on that node fail along with it. The cluster then selects the oldest mirror of each master and promotes it to a new master.
Fig. 3. Multiple mirrored queues and their policies
Broker 3 crashes. Note that the mirror of Queue C on Broker 2 is promoted to master. Also, note that a new mirror of Queue C is created on Broker 1. RabbitMQ always tries to maintain the replication factor specified in your policies.
Fig. 4. Broker 3 fails, causing Queue C to become unavailable
Next, Broker 1 crashes! We are left with only one broker. The mirror of Queue B is promoted to master.
Fig. 5
We have brought Broker 1 back online. Regardless of how successfully the data survived the loss and recovery of the broker, all mirrored messages in the queue are discarded upon restart. This is important to note, as there will be consequences. We will soon discuss these consequences. Thus, Broker 1 is now a member of the cluster again, and the cluster attempts to comply with policies and therefore creates mirrors on Broker 1.
In this case, the loss of Broker 1 was complete, along with the data, so the unmirrored Queue B is entirely lost.
Fig. 6. Broker 1 comes back online
Broker 3 is back online, so queues A and B are receiving their mirrors again to satisfy their HA policies. However, now all primary queues are on one node! This isn't ideal; a more balanced distribution across nodes would be better. Unfortunately, there aren't many options for rebalancing the masters here. We'll come back to this issue later, as we first need to look at queue synchronization.
Fig. 7. Broker 3 is back online. All primary queues are on one node!
Now you should have an understanding of how mirrors provide redundancy and fault tolerance. This guarantees availability in case one node fails and protects against data loss. But we’re not done yet, because it’s actually much more complicated.
Synchronization
When a new mirror is created, all new messages will always be replicated to this mirror and any others. As for the existing data in the primary queue, we can replicate it to the new mirror, which becomes a complete copy of the master. We can also choose not to replicate existing messages, allowing the primary queue and the new mirror to converge over time as new messages come in at the tail, while existing messages leave the head of the primary queue.
Such synchronization is performed automatically or manually and is managed through queue policies. Let’s consider an example.
We have two mirrored queues. Queue A is synchronized automatically, while Queue B is synchronized manually. Both queues contain ten messages.
Fig. 8. Two queues with different synchronization modes
Now we are losing Broker 3.
Fig. 9. Broker 3 has failed
Broker 3 comes back online. The cluster creates a mirror for each queue on the new node and automatically synchronizes the new Queue A with the master. However, the mirror for the new Queue B remains empty. Thus, we have full redundancy for Queue A and only one mirror for the existing messages in Queue B.
Fig. 10. The new mirror for Queue A receives all existing messages, while the new mirror for Queue B does not
Both queues receive another ten messages. Then Broker 2 crashes, and Queue A rolls back to the oldest mirror located on Broker 1. No data loss occurs during the failure. Queue B has twenty messages in the master and only ten in the mirror, as this queue has never replicated the original ten messages.
Fig. 11. Queue A rolls back to Broker 1 without message loss.
Both queues receive another ten messages. Now Broker 1 crashes. Queue A switches to the mirror without any message loss. However, Queue B encounters issues. At this stage, we can optimize for either availability or consistency.
If we want to optimize for availability, we need to set the policy ha-promote-on-failure to always. This is the default value, so we can simply not specify the policy at all. In this case, we essentially allow failures in unsynchronized mirrors. This will lead to message loss, but the queue remains available for reading and writing.
Fig. 12. Queue A rolls back to Broker 3 without message loss. Queue B rolls back to Broker 3 with a loss of ten messages.
We can also set ha-promote-on-failure to when-synced. In this case, instead of rolling back to the mirror, the queue will wait until Broker 1 returns to operational status with its data. After its return, the main queue is again on Broker 1 without data loss. Availability comes at the expense of data safety. However, this is a risky mode that can even lead to total data loss, which we will examine shortly.
Fig. 13. Queue B remains unavailable after losing Broker 1.
You might ask, 'Maybe it’s better not to use automatic synchronization at all?'. The answer is that synchronization is a blocking operation. During synchronization, the main queue cannot perform any read or write operations!
Let's consider an example. Right now, we have very large queues. How can they grow to such a size? For several reasons:
- Queues are not actively used.
- These are high-speed queues, and right now consumers are working slowly.
- These are high-speed queues, a failure has occurred, and consumers are catching up.
Fig. 14. Two large queues with different synchronization modes
Now Broker 3 is down.
Fig. 15. Broker 3 down, leaving one master and one mirror in each queue
Broker 3 comes back online, and new mirrors are created. Main Queue A starts replicating existing messages to the new mirror, and during this time the Queue is unavailable. Replicating data takes two hours, resulting in two hours of downtime for this Queue!
However, Queue B remains available throughout the period. It sacrificed some redundancy for availability.
Fig. 16. Queue remains unavailable during synchronization
After two hours, Queue A also becomes available and can again begin accepting read and write operations.
Updates
This blocking behavior during synchronization complicates updates for clusters with very large queues. At some point, the node with the master needs to be restarted, which means either switching to a mirror or disabling the queue during the server upgrade. If we choose to switch, we will lose messages if the mirrors are not in sync. By default, during broker shutdown, switching to an unsynchronized mirror does not occur. This means that once the broker returns, we do not lose any messages; the only loss incurred is just the downtime of the queue. The behavior rules for broker shutdown are defined by policy. ha-promote-on-shutdown. One of two values can be set:
always= enable switching to unsynchronized mirrorswhen-synced= switch only to a synchronized mirror, otherwise the queue becomes unavailable for reading and writing. The queue comes back online as soon as the broker returns.
Either way, with large queues, one has to choose between data loss and unavailability.
When availability enhances data security
Before making a decision, another complication needs to be considered. While automatic synchronization is better for redundancy, how does it affect data security? Certainly, with better redundancy, RabbitMQ is less likely to lose existing messages, but what about new messages from publishers?
Here is what needs to be considered:
- Can the publisher simply return an error, and the higher-level service or user will try again later?
- Can the publisher save the message locally or in a database to retry later?
If the publisher can only discard the message, then, in fact, improving availability also enhances data security.
Thus, a balance must be sought, and the solution depends on the specific situation.
Issues with ha-promote-on-failure=when-synced
Idea ha-promote-on-failure= when-synced is that we prevent switching to an unsynchronized mirror, thereby avoiding data loss. The queue remains unavailable for reading or writing. Instead, we try to recover the failed broker with intact data so it can resume operations as the master without data loss.
But (and this is a big but) if the broker has lost its data, then we have a big problem: the queue is gone! All data is lost! Even if you have mirrors that are mostly catching up to the main queue, those mirrors are discarded as well.
To re-add a node with the same name, we tell the cluster to forget the lost node (with the command rabbitmqctl forget_cluster_node) and start a new broker with the same hostname. As long as the cluster remembers the lost node, it remembers the old queue and the unsynchronized mirrors. When the cluster is told to forget the lost node, that queue is also forgotten. Now it needs to be declared again. We lost all the data, although we had mirrors with a partial dataset. It would have been better to switch to an unsynchronized mirror!
Therefore, manual synchronization (and not performing synchronization) combined with ha-promote-on-failure=when-synced, in my opinion, is quite risky. Documentation states that this option exists for data security, but it is a double-edged sword.
Rebalancing masters
As promised, we return to the issue of all masters clustering on one or several nodes. This can happen even as a result of a rolling update of the cluster. In a three-node cluster, all the main queues may cluster on one or two nodes.
Rebalancing masters can be problematic for two reasons:
- There are no good tools for performing rebalancing
- Queue synchronization
There is a third-party tool for rebalancing , which is not officially supported. Regarding third-party plugins, the RabbitMQ documentation mentions : "The plugin provides some additional configuration and reporting tools, but is not supported or verified by the RabbitMQ team. Use at your own risk."
There is another trick to move the main queue using HA policies. The documentation mentions for this. It works as follows:
- Removes all mirrors using a temporary policy with a higher priority than the existing HA policy.
- Changes the temporary HA policy to use the 'nodes' mode specifying the node to which the main queue needs to be moved.
- Synchronizes the queue for forced migration.
- After the migration is complete, it removes the temporary policy. The original HA policy takes effect and the required number of mirrors is created.
The downside is that this approach may not work if you have large queues or strict redundancy requirements.
Now let's see how RabbitMQ clusters deal with network partitions.
Network Partition
Nodes in a distributed system are connected by network links, and network links can and will go down. The frequency of outages depends on the local infrastructure or the reliability of the chosen cloud. In any case, distributed systems must be able to cope with them. Again, we have a choice between availability and consistency, and again the good news is that RabbitMQ provides both options (just not at the same time).
With RabbitMQ, we have two main options:
- Allow split-brain. This ensures availability but may lead to data loss.
- Disallow split-brain. This may lead to short-term loss of availability depending on how clients connect to the cluster. It can also lead to complete inaccessibility in a two-node cluster.
But what is a split-brain? It is when a cluster is split in two due to network link loss. On each side, mirrors are promoted to masters, so each queue ends up having multiple masters.
Fig. 17. The main queue and two mirrors, each on a separate node. Then a network failure occurs, and one mirror detaches. The detached node sees that the two others have gone down and promotes its mirrors to master. Now we have two main queues, both allowing read and write operations.
If publishers send data to both masters, we will end up with two diverging copies of the queue.
Different RabbitMQ modes provide either availability or consistency.
Ignore mode (default)
This mode ensures availability. After a partition loss occurs, a logical split happens. Once connectivity is restored, the administrator must decide which partition to prefer. The losing side will be restarted, and all accumulated data on that side will be lost.
Fig. 18. Three publishers are connected to three brokers. Internally, the cluster routes all requests to the main queue on Broker 2.
Now we lose Broker 3. It sees that the other brokers have gone down and promotes its mirror to master. This causes a logical split.
Fig. 19. Logical partition (split-brain). Writes are sent to two main queues, and the two copies diverge.
Connectivity is restored, but the logical split remains. The administrator must manually select the losing side. In the case below, the administrator restarts Broker 3. All messages that it failed to deliver are lost.
Fig. 20. The administrator disconnects Broker 3.
Fig. 21. The administrator starts Broker 3, and it joins the cluster, losing all messages that remained there.
During the loss of connectivity and after its restoration, the cluster and this queue were available for reading and writing.
Autoheal mode
Functions similarly to Ignore mode, except that the cluster itself automatically chooses the losing side after the partition and recovery of connectivity. The losing side returns to the cluster empty, and the queue loses all messages that were sent only to that side.
Pause Minority mode
If we want to avoid logical partitioning, our only option is to refrain from reading and writing on the smaller side after the cluster split. When the broker detects that it is on the smaller side, it suspends operation, closing all existing connections and rejecting any new ones. It checks for connectivity recovery once per second. Once connectivity is restored, it resumes operation and rejoins the cluster.
Fig. 22. Three publishers are connected to three brokers. Internally, the cluster directs all requests to the main queue on Broker 2.
Then Brokers 1 and 2 separate from Broker 3. Instead of promoting its mirror to master, Broker 3 pauses operation and becomes unavailable.
Fig. 23. Broker 3 suspends operation, disconnects all clients, and rejects connection requests.
Once connectivity is restored, it returns to the cluster.
Let’s look at another example where the main queue is on Broker 3.
Fig. 24. The main queue on Broker 3.
Then, the same loss of connectivity occurs. Broker 3 pauses operation as it is on the smaller side. On the other side, the nodes see that Broker 3 has gone down, so the older mirror from Brokers 1 and 2 is promoted to master.
Fig. 25. Transition to Broker 2 when Broker 3 is unavailable.
When connectivity is restored, Broker 3 will join the cluster.
Fig. 26. The cluster has returned to normal operation.
It is important to understand here that we achieve consistency, but we can also gain availability. if we will successfully transfer clients to the larger part of the split. For most scenarios, I personally would choose the Pause Minority mode, but it really depends on the specific case.
To ensure availability, it is crucial to make sure that clients successfully connect to the node. Let’s consider our options.
Ensuring client connectivity
We have several options for directing clients back to the main part of the cluster or to functioning nodes after a loss of connectivity (after a node failure). First, let's remember that a specific queue is hosted on a certain node, but the routing and policies are replicated across all nodes. Clients can connect to any node, and the internal routing will direct them accordingly. However, when a node is suspended, it rejects connections, so clients must connect to another node. If a node has failed, it has very little it can do.
Our options:
- Access to the cluster is provided through a load balancer that simply cycles through the nodes, while clients attempt to reconnect until successful. If a node is not operational or is suspended, attempts to connect to that node will fail, but subsequent attempts will proceed to other servers (in a cyclical manner). This is suitable for brief connectivity losses or a failed server that will be quickly restored.
- Access to the cluster via a load balancer and removal of suspended/failed nodes from the list as soon as they are detected. If this is done quickly, and if clients can attempt to reconnect, then we achieve continuous availability.
- Provide each client with a list of all nodes, and the client randomly selects one from them when connecting. If it encounters an error while trying to connect, it moves to the next node in the list until it successfully connects.
- Remove traffic from the failed/suspended node using DNS. This is done through a short TTL.
Conclusions
RabbitMQ clustering has its advantages and disadvantages. The most serious drawbacks are that:
- when joining the cluster, nodes discard their data;
- blocking synchronization leads to queue unavailability.
All difficult decisions stem from these two architectural characteristics. If RabbitMQ could retain data when reconnecting the cluster, synchronization would occur faster. If it could perform non-blocking synchronization, it would better support large queues. Addressing these two issues would significantly enhance RabbitMQ's performance as a fault-tolerant and highly available messaging technology. I would hesitate to recommend RabbitMQ with clustering in the following situations:
- Unreliable network.
- Unreliable storage.
- Very large queues.
Regarding settings for high availability, consider the following:
ha-promote-on-failure=alwaysha-sync-mode=manualcluster_partition_handling=ignore(orautoheal)- durable messages
- ensure that clients connect to an active node when a node fails
For consistency (data safety), consider the following settings:
- Publisher Confirms and Manual Acknowledgements on the consumer side
ha-promote-on-failure=when-synced, if publishers can retry later and if you have a very reliable storage! Otherwise, set=always.ha-sync-mode=automatic(but for large inactive queues, manual mode may be required; also, consider whether downtime would lead to message loss)- Pause Minority mode
- durable messages
We have not yet covered all aspects of fault tolerance and high availability; for example, how to safely perform administrative procedures (such as rolling updates). We also need to discuss federating and the Shovel plugin.
If I missed anything else, please let me know.
See also my , where I conduct a deep dive into the RabbitMQ cluster using Docker and Blockade to test some message loss scenarios described in this article.
Previous articles in the series:
#1 —
#2 —
#3 —
Source: habr.com
