{"id":52525,"date":"2019-11-10T00:00:00","date_gmt":"2019-11-09T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah"},"modified":"2020-02-18T14:00:16","modified_gmt":"2020-02-18T11:00:16","slug":"rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah","title":{"rendered":"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/d8e584d210a6a016a962ad6f956a2078.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFault tolerance and high availability are vast topics, so let's dedicate separate articles to RabbitMQ and Kafka. This article focuses on RabbitMQ, while the next one will cover Kafka in comparison to RabbitMQ. It's a lengthy read, so get comfortable.<\/p>\n<p>We will explore fault tolerance strategies, consistency, and high availability (HA), as well as the trade-offs that come with each strategy. RabbitMQ can operate on a cluster of nodes, classifying it as a distributed system. When discussing distributed systems, we often talk about consistency and availability. <\/p>\n<p>These concepts describe how a system behaves during failures. Network connection failures, server crashes, hard drive failures, temporary server unavailability due to garbage collection, packet loss, or network slowdowns can all lead to data loss or conflicts. It turns out that it is virtually impossible to create a system that is both completely consistent (without data loss, without data discrepancies) and available (acceptable for read and write operations) for all types of failures.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nWe will see that consistency and availability exist at opposite ends of the spectrum, and you need to choose which direction to optimize for. The good news is that with RabbitMQ, such choices are possible. You have those 'nerdy' levers to shift the balance towards greater consistency or greater availability.<\/p>\n<p>Particular attention will be given to configurations that lead to data loss due to acknowledged messages. There is a chain of responsibility between publishers, brokers, and consumers. Once a message is handed over to the broker, it is their job not to lose it. When the broker acknowledges the receipt of a message to the publisher, we do not expect that it will be lost. However, we will see that this can indeed happen depending on your broker and publisher's configuration.<\/p>\n<h1>Single-node resilience primitives<\/h1>\n<p><\/p>\n<h3>Resilient queues\/routing<\/h3>\n<p>\nIn RabbitMQ, there are two types of queues: durable and non-durable. All queues are stored in the Mnesia database. Durable queues are redeclared when the node starts and thus survive restarts, system crashes, or server failures (as long as the data is preserved). This means that as long as you declare the routing (exchange) and queue as durable, the queue\/routing infrastructure will return to operational mode.<\/p>\n<p>Non-durable queues and routing are deleted upon restarting the node.<\/p>\n<h3>Durable messages<\/h3>\n<p>\nJust because a queue is durable does not mean that all its messages will survive the node's restart. Only messages marked by the publisher as <i>durable<\/i> (persistent) will be restored. Durable messages do create additional load on the broker, but if message loss is unacceptable, there is no other option.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/8b8e2de4ef82bf8497960657aac0ddb0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 1. Durability Matrix<\/i><\/p>\n<h1>Clustering with Queue Mirroring<\/h1>\n<p>\nTo survive a broker loss, we need redundancy. We can combine multiple RabbitMQ nodes into a cluster and then add additional redundancy by replicating queues across several nodes. Thus, if one node fails, we do not lose data and remain accessible. <\/p>\n<p>Queue mirroring:<\/p>\n<ul>\n<li>one master queue that receives all write and read commands\n<\/li>\n<li>one or more mirrors that receive all messages and metadata from the master queue. These mirrors exist not for scaling, but solely for redundancy.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/35f3d4bdc4f5e10da0e40eb91c5d8280.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 2. Queue Mirroring<\/i><\/p>\n<p>Mirroring is set up with the appropriate policy. In it, you can choose the replication factor and even the nodes on which the queue should be located. Examples:<\/p>\n<ul>\n<li><code>ha-mode: all<\/code>\n<\/li>\n<li><code>ha-mode: exactly, ha-params: 2<\/code> (one master and one mirror)\n<\/li>\n<li><code>ha-mode: nodes, ha-params: rabbit@node1, rabbit@node2<\/code><\/li>\n<\/ul>\n<p><\/p>\n<h1>Acknowledgment to the Publisher<\/h1>\n<p>\nTo achieve consistent message recording, publisher confirmations are necessary. Without them, there is a risk of message loss. Confirmation is sent to the publisher after the message is written to disk. RabbitMQ writes messages to disk not upon receipt, but on a periodic basis, typically within a few hundred milliseconds. When a queue is mirrored, confirmation is sent only after all mirrors have also written their copy of the message to disk. This means that using confirmations adds latency, but if data security is important, they are essential.<\/p>\n<h1>Fault-tolerant queue<\/h1>\n<p>\nWhen the broker shuts down or crashes, all leading queues (masters) on that node fail along with it. The cluster then selects the oldest mirror of each master and promotes it to a new master.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/8d8227c00643f35e0dc752b6617e18fb.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 3. Multiple mirrored queues and their policies<\/i><\/p>\n<p>Broker 3 crashes. Note that the mirror of Queue C on Broker 2 is promoted to master. Also, note that a new mirror of Queue C is created on Broker 1. RabbitMQ always tries to maintain the replication factor specified in your policies.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/d08b3ada22e79686d448714f7bab009a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 4. Broker 3 fails, causing Queue C to become unavailable<\/i> <\/p>\n<p>Next, Broker 1 crashes! We are left with only one broker. The mirror of Queue B is promoted to master.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/dcc1d542c40c441194fd07e51046c619.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 5<\/i><\/p>\n<p>We have brought Broker 1 back online. Regardless of how successfully the data survived the loss and recovery of the broker, all mirrored messages in the queue are discarded upon restart. This is important to note, as there will be consequences. We will soon discuss these consequences. Thus, Broker 1 is now a member of the cluster again, and the cluster attempts to comply with policies and therefore creates mirrors on Broker 1.<\/p>\n<p>In this case, the loss of Broker 1 was complete, along with the data, so the unmirrored Queue B is entirely lost.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/7ac7ab65ea6dff9c5a49372c04c83a37.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 6. Broker 1 comes back online<\/i><\/p>\n<p>Broker 3 is back online, so queues A and B are receiving their mirrors again to satisfy their HA policies. However, now all primary queues are on one node! This isn't ideal; a more balanced distribution across nodes would be better. Unfortunately, there aren't many options for rebalancing the masters here. We'll come back to this issue later, as we first need to look at queue synchronization. <\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/a54e53bb01e2f830a54f88acaa668984.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 7. Broker 3 is back online. All primary queues are on one node!<\/i><\/p>\n<p>Now you should have an understanding of how mirrors provide redundancy and fault tolerance. This guarantees availability in case one node fails and protects against data loss. But we\u2019re not done yet, because it\u2019s actually much more complicated.<\/p>\n<h1>Synchronization<\/h1>\n<p>\nWhen a new mirror is created, all new messages will always be replicated to this mirror and any others. As for the existing data in the primary queue, we can replicate it to the new mirror, which becomes a complete copy of the master. We can also choose not to replicate existing messages, allowing the primary queue and the new mirror to converge over time as new messages come in at the tail, while existing messages leave the head of the primary queue.<\/p>\n<p>Such synchronization is performed automatically or manually and is managed through queue policies. Let\u2019s consider an example.<\/p>\n<p>We have two mirrored queues. Queue A is synchronized automatically, while Queue B is synchronized manually. Both queues contain ten messages.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/304f2a475fdd0cb6d3069140c667a0a7.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 8. Two queues with different synchronization modes<\/i><\/p>\n<p>Now we are losing Broker 3.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/49114966f440c407488bb467ff6dbf93.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 9. Broker 3 has failed<\/i><\/p>\n<p>Broker 3 comes back online. The cluster creates a mirror for each queue on the new node and automatically synchronizes the new Queue A with the master. However, the mirror for the new Queue B remains empty. Thus, we have full redundancy for Queue A and only one mirror for the existing messages in Queue B.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/adffc1f5eff8806edbd83772f338d67f.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 10. The new mirror for Queue A receives all existing messages, while the new mirror for Queue B does not<\/i><\/p>\n<p>Both queues receive another ten messages. Then Broker 2 crashes, and Queue A rolls back to the oldest mirror located on Broker 1. No data loss occurs during the failure. Queue B has twenty messages in the master and only ten in the mirror, as this queue has never replicated the original ten messages.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/c6cfef4bced5627bf243c2f020a42646.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 11. Queue A rolls back to Broker 1 without message loss.<\/i><\/p>\n<p>Both queues receive another ten messages. Now Broker 1 crashes. Queue A switches to the mirror without any message loss. However, Queue B encounters issues. At this stage, we can optimize for either availability or consistency. <\/p>\n<p>If we want to optimize for availability, we need to set the policy <b><i>ha-promote-on-failure<\/i><\/b> to <b><i>always<\/i><\/b>. This is the default value, so we can simply not specify the policy at all. In this case, we essentially allow failures in unsynchronized mirrors. This will lead to message loss, but the queue remains available for reading and writing.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/cd704cf4397184c212e1576b986abbaa.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 12. Queue A rolls back to Broker 3 without message loss. Queue B rolls back to Broker 3 with a loss of ten messages.<\/i><\/p>\n<p>We can also set <code>ha-promote-on-failure<\/code> to <code>when-synced<\/code>. In this case, instead of rolling back to the mirror, the queue will wait until Broker 1 returns to operational status with its data. After its return, the main queue is again on Broker 1 without data loss. Availability comes at the expense of data safety. However, this is a risky mode that can even lead to total data loss, which we will examine shortly.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/03fed936fe75220e22f68432ca81b654.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 13. Queue B remains unavailable after losing Broker 1.<\/i><\/p>\n<p>You might ask, 'Maybe it\u2019s better not to use automatic synchronization at all?'. The answer is that synchronization is a blocking operation. During synchronization, the main queue cannot perform any read or write operations!<\/p>\n<p>Let's consider an example. Right now, we have very large queues. How can they grow to such a size? For several reasons:<\/p>\n<ul>\n<li>Queues are not actively used.\n<\/li>\n<li>These are high-speed queues, and right now consumers are working slowly. \n<\/li>\n<li>These are high-speed queues, a failure has occurred, and consumers are catching up.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/dd4b0fd70cbbc0b7defc06536157770d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 14. Two large queues with different synchronization modes<\/i><\/p>\n<p>Now Broker 3 is down.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/8c31cfdf485ab4c9dfe13e86a151df4f.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 15. Broker 3 down, leaving one master and one mirror in each queue<\/i><\/p>\n<p>Broker 3 comes back online, and new mirrors are created. Main Queue A starts replicating existing messages to the new mirror, and during this time the Queue is unavailable. Replicating data takes two hours, resulting in two hours of downtime for this Queue!<\/p>\n<p>However, Queue B remains available throughout the period. It sacrificed some redundancy for availability.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/28e641300a1c3f4d70e29a16db51521e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 16. Queue remains unavailable during synchronization<\/i><\/p>\n<p>After two hours, Queue A also becomes available and can again begin accepting read and write operations.<\/p>\n<h3>Updates<\/h3>\n<p>\nThis blocking behavior during synchronization complicates updates for clusters with very large queues. At some point, the node with the master needs to be restarted, which means either switching to a mirror or disabling the queue during the server upgrade. If we choose to switch, we will lose messages if the mirrors are not in sync. By default, during broker shutdown, switching to an unsynchronized mirror does not occur. This means that once the broker returns, we do not lose any messages; the only loss incurred is just the downtime of the queue. The behavior rules for broker shutdown are defined by policy. <code>ha-promote-on-shutdown<\/code>. One of two values can be set:<\/p>\n<ul>\n<li><code>always<\/code>= enable switching to unsynchronized mirrors\n<\/li>\n<li><code>when-synced<\/code>= switch only to a synchronized mirror, otherwise the queue becomes unavailable for reading and writing. The queue comes back online as soon as the broker returns.<\/li>\n<\/ul>\n<p>\nEither way, with large queues, one has to choose between data loss and unavailability.<\/p>\n<h3>When availability enhances data security<\/h3>\n<p>\nBefore making a decision, another complication needs to be considered. While automatic synchronization is better for redundancy, how does it affect data security? Certainly, with better redundancy, RabbitMQ is less likely to lose existing messages, but what about new messages from publishers?<\/p>\n<p>Here is what needs to be considered:<\/p>\n<ul>\n<li>Can the publisher simply return an error, and the higher-level service or user will try again later?\n<\/li>\n<li>Can the publisher save the message locally or in a database to retry later?<\/li>\n<\/ul>\n<p>\nIf the publisher can only discard the message, then, in fact, improving availability also enhances data security.<\/p>\n<p>Thus, a balance must be sought, and the solution depends on the specific situation.<\/p>\n<h1>Issues with ha-promote-on-failure=when-synced<\/h1>\n<p>\nIdea <i><b>ha-promote-on-failure<\/b><\/i>= <i><b>when-synced<\/b><\/i> is that we prevent switching to an unsynchronized mirror, thereby avoiding data loss. The queue remains unavailable for reading or writing. Instead, we try to recover the failed broker with intact data so it can resume operations as the master without data loss. <\/p>\n<p>But (and this is a big but) if the broker has lost its data, then we have a big problem: the queue is gone! All data is lost! Even if you have mirrors that are mostly catching up to the main queue, those mirrors are discarded as well.<\/p>\n<p>To re-add a node with the same name, we tell the cluster to forget the lost node (with the command <i>rabbitmqctl forget_cluster_node<\/i>) and start a new broker with the same hostname. As long as the cluster remembers the lost node, it remembers the old queue and the unsynchronized mirrors. When the cluster is told to forget the lost node, that queue is also forgotten. Now it needs to be declared again. We lost all the data, although we had mirrors with a partial dataset. It would have been better to switch to an unsynchronized mirror!<\/p>\n<p>Therefore, manual synchronization (and not performing synchronization) combined with <code>ha-promote-on-failure=when-synced<\/code>, in my opinion, is quite risky. Documentation states that this option exists for data security, but it is a double-edged sword.<\/p>\n<h1>Rebalancing masters<\/h1>\n<p>\nAs promised, we return to the issue of all masters clustering on one or several nodes. This can happen even as a result of a rolling update of the cluster. In a three-node cluster, all the main queues may cluster on one or two nodes.<\/p>\n<p>Rebalancing masters can be problematic for two reasons:<\/p>\n<ul>\n<li>There are no good tools for performing rebalancing<\/li>\n<li>Queue synchronization<\/li>\n<\/ul>\n<p>\nThere is a third-party tool for rebalancing <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Ayanda-D\/rabbitmq-queue-master-balancer\">plugin<\/a><\/noindex>, which is not officially supported. Regarding third-party plugins, the RabbitMQ documentation mentions <noindex><a rel=\"nofollow\" href=\"https:\/\/www.rabbitmq.com\/upgrade.html\">that<\/a><\/noindex>: \"The plugin provides some additional configuration and reporting tools, but is not supported or verified by the RabbitMQ team. Use at your own risk.\"<\/p>\n<p>There is another trick to move the main queue using HA policies. The documentation mentions <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/rabbitmq\/support-tools\/blob\/master\/scripts\/rebalance-queue-masters\">a script<\/a><\/noindex> for this. It works as follows:<\/p>\n<ul>\n<li>Removes all mirrors using a temporary policy with a higher priority than the existing HA policy.\n<\/li>\n<li>Changes the temporary HA policy to use the 'nodes' mode specifying the node to which the main queue needs to be moved.\n<\/li>\n<li>Synchronizes the queue for forced migration.\n<\/li>\n<li>After the migration is complete, it removes the temporary policy. The original HA policy takes effect and the required number of mirrors is created.<\/li>\n<\/ul>\n<p>\nThe downside is that this approach may not work if you have large queues or strict redundancy requirements.<\/p>\n<p>Now let's see how RabbitMQ clusters deal with network partitions.<\/p>\n<h1>Network Partition<\/h1>\n<p>\nNodes in a distributed system are connected by network links, and network links can and will go down. The frequency of outages depends on the local infrastructure or the reliability of the chosen cloud. In any case, distributed systems must be able to cope with them. Again, we have a choice between availability and consistency, and again the good news is that RabbitMQ provides both options (just not at the same time).<\/p>\n<p>With RabbitMQ, we have two main options:<\/p>\n<ul>\n<li>Allow split-brain. This ensures availability but may lead to data loss.\n<\/li>\n<li>Disallow split-brain. This may lead to short-term loss of availability depending on how clients connect to the cluster. It can also lead to complete inaccessibility in a two-node cluster.<\/li>\n<\/ul>\n<p>\nBut what is a split-brain? It is when a cluster is split in two due to network link loss. On each side, mirrors are promoted to masters, so each queue ends up having multiple masters.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/d880a935df450fb994edfed0715e026e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 17. The main queue and two mirrors, each on a separate node. Then a network failure occurs, and one mirror detaches. The detached node sees that the two others have gone down and promotes its mirrors to master. Now we have two main queues, both allowing read and write operations.<\/i> <\/p>\n<p>If publishers send data to both masters, we will end up with two diverging copies of the queue.<\/p>\n<p>Different RabbitMQ modes provide either availability or consistency.<\/p>\n<h3>Ignore mode (default)<\/h3>\n<p>\nThis mode ensures availability. After a partition loss occurs, a logical split happens. Once connectivity is restored, the administrator must decide which partition to prefer. The losing side will be restarted, and all accumulated data on that side will be lost.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/4d82c13b5ef21837ba819f5c246a1e41.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 18. Three publishers are connected to three brokers. Internally, the cluster routes all requests to the main queue on Broker 2.<\/i><\/p>\n<p>Now we lose Broker 3. It sees that the other brokers have gone down and promotes its mirror to master. This causes a logical split.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/c7e01d51b24023bacdd93cfad95c9eab.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 19. Logical partition (split-brain). Writes are sent to two main queues, and the two copies diverge.<\/i><\/p>\n<p>Connectivity is restored, but the logical split remains. The administrator must manually select the losing side. In the case below, the administrator restarts Broker 3. All messages that it failed to deliver are lost.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/51c3ee15024d3c83ba625e3ff9cdeb90.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 20. The administrator disconnects Broker 3.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/0bac8e33de25c0c5381d336d26464858.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 21. The administrator starts Broker 3, and it joins the cluster, losing all messages that remained there.<\/i><\/p>\n<p>During the loss of connectivity and after its restoration, the cluster and this queue were available for reading and writing.<\/p>\n<h3>Autoheal mode<\/h3>\n<p>\nFunctions similarly to Ignore mode, except that the cluster itself automatically chooses the losing side after the partition and recovery of connectivity. The losing side returns to the cluster empty, and the queue loses all messages that were sent only to that side.<\/p>\n<h3>Pause Minority mode<\/h3>\n<p>\nIf we want to avoid logical partitioning, our only option is to refrain from reading and writing on the smaller side after the cluster split. When the broker detects that it is on the smaller side, it suspends operation, closing all existing connections and rejecting any new ones. It checks for connectivity recovery once per second. Once connectivity is restored, it resumes operation and rejoins the cluster.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/7a51cf509b2c94e2ae100c7dedd9d21a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 22. Three publishers are connected to three brokers. Internally, the cluster directs all requests to the main queue on Broker 2.<\/i><\/p>\n<p>Then Brokers 1 and 2 separate from Broker 3. Instead of promoting its mirror to master, Broker 3 pauses operation and becomes unavailable.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/c12ec38a47e1f8ccceebfa1854f54f21.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 23. Broker 3 suspends operation, disconnects all clients, and rejects connection requests.<\/i><\/p>\n<p>Once connectivity is restored, it returns to the cluster.<\/p>\n<p>Let\u2019s look at another example where the main queue is on Broker 3.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/4cc6a28270f00a280ecb5af5252f4e07.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 24. The main queue on Broker 3.<\/i><\/p>\n<p>Then, the same loss of connectivity occurs. Broker 3 pauses operation as it is on the smaller side. On the other side, the nodes see that Broker 3 has gone down, so the older mirror from Brokers 1 and 2 is promoted to master.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/d0d032385533bb53c5a95927afcd3119.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 25. Transition to Broker 2 when Broker 3 is unavailable.<\/i><\/p>\n<p>When connectivity is restored, Broker 3 will join the cluster.<\/p>\n<p><img decoding=\"async\" alt=\"RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters\" src=\"\/wp-content\/uploads\/2019\/11\/53f8186a040ae07724ef59cf79b72a0c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Fig. 26. The cluster has returned to normal operation.<\/i><\/p>\n<p>It is important to understand here that we achieve consistency, but we can also gain availability. <i><b>if<\/b><\/i> we will successfully transfer clients to the larger part of the split. For most scenarios, I personally would choose the Pause Minority mode, but it really depends on the specific case.<\/p>\n<p>To ensure availability, it is crucial to make sure that clients successfully connect to the node. Let\u2019s consider our options.<\/p>\n<h1>Ensuring client connectivity<\/h1>\n<p>\nWe have several options for directing clients back to the main part of the cluster or to functioning nodes after a loss of connectivity (after a node failure). First, let's remember that a specific queue is hosted on a certain node, but the routing and policies are replicated across all nodes. Clients can connect to any node, and the internal routing will direct them accordingly. However, when a node is suspended, it rejects connections, so clients must connect to another node. If a node has failed, it has very little it can do.<\/p>\n<p>Our options:<\/p>\n<ul>\n<li>Access to the cluster is provided through a load balancer that simply cycles through the nodes, while clients attempt to reconnect until successful. If a node is not operational or is suspended, attempts to connect to that node will fail, but subsequent attempts will proceed to other servers (in a cyclical manner). This is suitable for brief connectivity losses or a failed server that will be quickly restored.\n<\/li>\n<li>Access to the cluster via a load balancer and removal of suspended\/failed nodes from the list as soon as they are detected. If this is done quickly, and if clients can attempt to reconnect, then we achieve continuous availability.\n<\/li>\n<li>Provide each client with a list of all nodes, and the client randomly selects one from them when connecting. If it encounters an error while trying to connect, it moves to the next node in the list until it successfully connects.\n<\/li>\n<li>Remove traffic from the failed\/suspended node using DNS. This is done through a short TTL.<\/li>\n<\/ul>\n<p><\/p>\n<h1>Conclusions<\/h1>\n<p>\nRabbitMQ clustering has its advantages and disadvantages. The most serious drawbacks are that:<\/p>\n<ul>\n<li>when joining the cluster, nodes discard their data;\n<\/li>\n<li>blocking synchronization leads to queue unavailability.<\/li>\n<\/ul>\n<p>\nAll difficult decisions stem from these two architectural characteristics. If RabbitMQ could retain data when reconnecting the cluster, synchronization would occur faster. If it could perform non-blocking synchronization, it would better support large queues. Addressing these two issues would significantly enhance RabbitMQ's performance as a fault-tolerant and highly available messaging technology. I would hesitate to recommend RabbitMQ with clustering in the following situations:<\/p>\n<ul>\n<li>Unreliable network.\n<\/li>\n<li>Unreliable storage.\n<\/li>\n<li>Very large queues.<\/li>\n<\/ul>\n<p>\nRegarding settings for high availability, consider the following:<\/p>\n<ul>\n<li><code>ha-promote-on-failure=always<\/code>\n<\/li>\n<li><code>ha-sync-mode=manual<\/code>\n<\/li>\n<li><code>cluster_partition_handling=ignore<\/code> (or <code>autoheal<\/code>)\n<\/li>\n<li>durable messages\n<\/li>\n<li>ensure that clients connect to an active node when a node fails<\/li>\n<\/ul>\n<p>\nFor consistency (data safety), consider the following settings:<\/p>\n<ul>\n<li>Publisher Confirms and Manual Acknowledgements on the consumer side\n<\/li>\n<li><code>ha-promote-on-failure=when-synced<\/code>, if publishers can retry later and if you have a very reliable storage! Otherwise, set <code>=always<\/code>.\n<\/li>\n<li><code>ha-sync-mode=automatic<\/code> (but for large inactive queues, manual mode may be required; also, consider whether downtime would lead to message loss)\n<\/li>\n<li>Pause Minority mode\n<\/li>\n<li>durable messages<\/li>\n<\/ul>\n<p>\nWe have not yet covered all aspects of fault tolerance and high availability; for example, how to safely perform administrative procedures (such as rolling updates). We also need to discuss federating and the Shovel plugin.<\/p>\n<p>If I missed anything else, please let me know.<\/p>\n<p>See also my <noindex><a rel=\"nofollow\" href=\"https:\/\/jack-vanlightly.com\/blog\/2018\/9\/10\/how-to-lose-messages-on-a-rabbitmq-cluster\">post<\/a><\/noindex>, where I conduct a deep dive into the RabbitMQ cluster using Docker and Blockade to test some message loss scenarios described in this article.<\/p>\n<p>Previous articles in the series: <br \/>\n#1 \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/itsumma\/blog\/416629\/\">habr.com\/ru\/company\/itsumma\/blog\/416629<\/a><\/noindex> <br \/>\n#2 \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/itsumma\/blog\/418389\/\">habr.com\/ru\/company\/itsumma\/blog\/418389<\/a><\/noindex> <br \/>\n#3 \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/itsumma\/blog\/437446\/\">habr.com\/ru\/company\/itsumma\/blog\/437446<\/a><\/noindex><br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/itsumma\/blog\/471858\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u044c \u0438 \u0432\u044b\u0441\u043e\u043a\u0430\u044f \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u044c \u2014 \u0431\u043e\u043b\u044c\u0448\u0438\u0435 \u0442\u0435\u043c\u044b, \u0442\u0430\u043a \u0447\u0442\u043e \u043f\u043e\u0441\u0432\u044f\u0442\u0438\u043c RabbitMQ \u0438 Kafka \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0435 \u0441\u0442\u0430\u0442\u044c\u0438. \u0414\u0430\u043d\u043d\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u043e RabbitMQ, \u0430 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0430\u044f \u2014 \u043e Kafka, \u0432 \u0441\u0440\u0430\u0432\u043d\u0435\u043d\u0438\u0438 \u0441 RabbitMQ. \u0421\u0442\u0430\u0442\u044c\u044f \u0434\u043b\u0438\u043d\u043d\u0430\u044f, \u0442\u0430\u043a \u0447\u0442\u043e \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0439\u0442\u0435\u0441\u044c \u043f\u043e\u0443\u0434\u043e\u0431\u043d\u0435\u0435. \u0420\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c \u0441\u0442\u0440\u0430\u0442\u0435\u0433\u0438\u0438 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u0438, \u0441\u043e\u0433\u043b\u0430\u0441\u043e\u0432\u0430\u043d\u043d\u043e\u0441\u0442\u0438 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u0439 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u0438 (HA), \u0430 \u0442\u0430\u043a\u0436\u0435 \u043a\u043e\u043c\u043f\u0440\u043e\u043c\u0438\u0441\u0441\u044b, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u0440\u0438\u0445\u043e\u0434\u0438\u0442\u0441\u044f \u0438\u0434\u0442\u0438 \u0432 \u043a\u0430\u0436\u0434\u043e\u0439 \u0441\u0442\u0440\u0430\u0442\u0435\u0433\u0438\u0438. RabbitMQ \u043c\u043e\u0436\u0435\u0442 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-52525","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\".\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47RabbitMQ \u043f\u0440\u043e\u0442\u0438\u0432 Kafka: \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u044c \u0438 \u0432\u044b\u0441\u043e\u043a\u0430\u044f \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u044c \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430\u0445 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\".\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-11-09T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:00:16+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47RabbitMQ vs Kafka: Fault Tolerance and High Availability in Clusters | ProHoster","description":".","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47RabbitMQ \u043f\u0440\u043e\u0442\u0438\u0432 Kafka: \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u044c \u0438 \u0432\u044b\u0441\u043e\u043a\u0430\u044f \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u044c \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430\u0445 | ProHoster","og:description":".","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rabbitmq-protiv-kafka-otkazoustojchivost-i-vysokaya-dostupnost-v-klasterah","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-11-09T21:00:00+00:00","article:modified_time":"2020-02-18T11:00:16+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"52525","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 03:54:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 20:41:25","updated":"2026-01-24 03:54:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/52525","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=52525"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/52525\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=52525"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=52525"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=52525"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}