In the last article We examined RabbitMQ clustering for fault tolerance and high availability. Now let's delve deeper into Apache Kafka.

In this context, the unit of replication is a partition. Each topic has one or more partitions. Each partition has a leader, with or without followers. When creating a topic, the number of partitions and the replication factor are specified. A typical value is 3, meaning three replicas: one leader and two followers.

Fig. 1. Four partitions are distributed across three brokers.

All read and write requests go to the leader. Followers periodically send requests to the leader for the latest messages. Consumers never connect to followers; they exist solely for redundancy and fault tolerance.

Partition Failure

When a broker goes down, often the leaders of several partitions fail as well. In each case, a follower from another node becomes the leader. However, this is not always the case, as the synchronization factor also plays a role: whether there are synchronized followers, and if not, whether a transition to an unsynchronized replica is allowed. But let's not complicate things for now.

Broker 3 goes offline — and a new leader is elected on Broker 2 for Partition 2.

Fig. 2. Broker 3 dies, and its follower on Broker 2 is elected the new leader of Partition 2.

Then Broker 1 goes down, and Partition 1 loses its leader, whose role switches to Broker 2.

Fig. 3. Only one broker remains. All leaders are on the same broker with zero redundancy.

When Broker 1 returns to the network, it adds four followers, providing some redundancy for each partition. However, all leaders still remain on Broker 2.

Fig. 4. Leaders remain on Broker 2.

When Broker 3 comes online, we revert to three replicas per partition. But all leaders still stay on Broker 2.

Fig. 5. Imbalanced placement of leaders after restoring Brokers 1 and 3.

Kafka has a tool for better leader rebalancing than RabbitMQ. There, external plugins or scripts were required to change policies for migrating the main node at the cost of reducing redundancy during migration. Additionally, for large queues, one had to endure downtime during synchronization.

Kafka has the concept of 'preferred replicas' for the leader role. When topic partitions are created, Kafka tries to distribute leaders evenly across nodes and marks these initial leaders as preferred. Over time, due to server reboots, failures, and connectivity issues, leaders may end up on different nodes, as described in the extreme case above.

To address this, Kafka provides two options:

• Option auto.leader.rebalance.enable=true allows the controller node to automatically reassign leaders back to preferred replicas, thus restoring the even distribution.
• An administrator can run the script kafka-preferred-replica-election.sh to manually reassign them.

Figure 6. Replicas after rebalancing

This was a simplified version of a failure, but the reality is more complex, although nothing too complicated is involved. It all comes down to In-Sync Replicas (ISR).

In-Sync Replicas (ISR)

ISR is a set of replicas of a partition that is considered 'in-sync.' There is a leader here, and there may not be any followers. A follower is considered in-sync if it has made exact copies of all messages from the leader before the interval expires replica.lag.time.max.ms.

A follower is removed from the ISR set if it:

• has not made a fetch request within the interval replica.lag.time.max.ms (considered dead)
• has not caught up within the interval replica.lag.time.max.ms (considered slow)

Followers make fetch requests within the interval replica.fetch.wait.max.ms, which by default is 500 ms.

To clearly explain the purpose of ISR, we need to look at acknowledgments from the producer and some failure scenarios. Producers can choose when to send an acknowledgment to the broker:

• acks=0, no acknowledgment is sent
• acks=1, acknowledgment is sent after the leader has written the message to its local log
• acks=all, acknowledgment is sent after all replicas in ISR have written the message to their local logs

In Kafka terminology, if ISR has saved the message, a 'commit' occurs. Acks=all is the safest option, but it also adds additional latency. Let's consider two failure examples and how the different 'acks' options interact with the ISR concept.

Acks=1 and ISR

In this example, we will see that if the leader does not wait for each message to be acknowledged by all followers, data loss may occur during a leader failure. Switching to an unsynchronized follower can be allowed or prohibited by configuration. unclean.leader.election.enable.

In this example, the producer has acks=1 set. The partition is spread across all three brokers. Broker 3 is lagging behind, having synchronized with the leader eight seconds ago and is currently behind by 7456 messages. Broker 1 has only been lagging by one second. Our producer sends a message and quickly receives back an ack, with no overhead for slow or dead followers that the leader is not waiting for.

Fig. 7. ISR with three replicas

Broker 2 fails, and the producer receives a connection error. After leadership shifts to Broker 1, we lose 123 messages. The follower on Broker 1 was in the ISR but was not fully synchronized with the leader when it failed.

Fig. 8. Messages are lost on failure

In the configuration bootstrap.servers the producer lists several brokers and can ask another broker who has become the new leader of the partition. It then connects to Broker 1 and continues sending messages.

Fig. 9. Message sending resumes after a brief pause

Broker 3 is lagging even further. It is making fetch requests but cannot synchronize. This may be due to a slow network connection between brokers, storage issues, etc. It is removed from the ISR. Now the ISR consists of a single replica—the leader! The producer continues to send messages and receive acknowledgments.

Fig. 10. Follower on Broker 3 is removed from ISR

Broker 1 crashes, and the leadership role shifts to Broker 3 with a loss of 15286 messages! The producer receives a connection error message. Switching to a leader outside of ISR was possible only due to the configuration unclean.leader.election.enable=true. If it were set to false, then the switch would not have occurred, and all read and write requests would have been rejected. In this case, we wait for Broker 1 to return with its intact data in the replica, which will take over the leadership again.

Fig. 11. Broker 1 crashes. A large number of messages are lost during the failure.

The producer establishes a connection with the last broker and sees that it is now the leader of the partition. It starts sending messages to broker 3.

Fig. 12. After a brief pause, messages are sent to partition 0 again.

We observed that besides the brief interruptions for establishing new connections and finding a new leader, the producer continuously sent messages. This configuration ensures availability through consistency (data safety). Kafka lost thousands of messages but continued accepting new entries.

Acks=all and ISR

Let's repeat this scenario again, but with acks=all. Broker 3 has an average delay of four seconds. The producer sends a message with acks=all, and now does not receive a quick response. The leader waits for the message to be stored by all replicas in the ISR.

Fig. 13. ISR with three replicas. One is lagging, causing write delays.

After an additional delay of four seconds, broker 2 sends an ack. All replicas are now fully synchronized.

Fig. 14. All replicas store the messages and send an ack.

Broker 3 is now lagging even further and is removed from the ISR. The delay decreases significantly as there are no slow replicas left in the ISR. Broker 2 is now only waiting for broker 1, which has an average lag of 500 ms.

Fig. 15. Replica on broker 3 removed from ISR.

Then broker 2 crashes, and leadership passes to broker 1 without message loss.

Fig. 16. Broker 2 crashes.

The producer finds a new leader and starts sending messages to it. The delay decreases further, as the ISR now consists of a single replica! Therefore, the option acks=all does not add redundancy.

Fig. 17. Replica on broker 1 takes leadership without message loss.

Then broker 1 crashes, and leadership passes to broker 3 with a loss of 14,238 messages!

Fig. 18. Broker 1 dies, and the leadership transition with unclean settings leads to significant data loss.

We could choose not to set the option unclean.leader.election.enable to true. By default, it is set to false. The configuration acks=all with unclean.leader.election.enable=true ensures availability with some additional data safety. However, as you can see, we can still lose messages.

But what if we want to increase data safety? We can set unclean.leader.election.enable = false, but this does not necessarily protect us from data loss. If the leader fails hard and takes the data with it, then the messages are still lost, plus accessibility is lost until the administrator restores the situation.

It is better to ensure redundancy for all messages, otherwise, it is better to forgo recording. Then, from the broker's perspective, data loss is only possible in the event of two or more simultaneous failures.

Acks=all, min.insync.replicas, and ISR

With the topic configuration min.insync.replicas we increase the level of data security. Let's go over the last part of the previous scenario again, but this time with min.insync.replicas=2.

So, broker 2 has a leader replica, and the follower on broker 3 is removed from ISR.

Fig. 19. ISR of two replicas

Broker 2 fails, and leadership switches to broker 1 without message loss. But now the ISR consists of only one replica. This does not meet the minimum number for recording, so the broker responds to the recording attempt with an error. NotEnoughReplicas.

Fig. 20. The number of ISR is one less than specified in min.insync.replicas

This configuration sacrifices availability for consistency. Before acknowledging the message, we ensure it is written to at least two replicas. This gives the producer much greater confidence. Here, message loss is only possible in the event of simultaneous failure of two replicas in a short interval, while the message is not replicated to an additional follower, which is unlikely. But if you are super paranoid, you can set the replication factor to 5, and min.insync.replicas with 3. Here, all three brokers must fail simultaneously to lose the record! Of course, you will pay with additional latency for such reliability.

When availability is necessary for data security

Like in in the case of RabbitMQ, sometimes availability is necessary for data security. You need to think about the following:

• Can the publisher simply return an error, and the higher-level service or user retry later?
• Can the publisher save the message locally or in a database to retry later?

If the answer is no, then optimizing for availability enhances data security. You will lose less data if you choose availability over rejecting the recording. Thus, it all comes down to finding a balance, and the solution depends on the specific situation.

The meaning of ISR

The ISR set allows for an optimal balance between data security and latency. For instance, it ensures availability in the event of a failure of most replicas, minimizing the impact of dead or slow replicas in terms of latency.

We choose the value ourselves replica.lag.time.max.ms based on our needs. Essentially, this parameter indicates how much latency we are willing to accept when acks=all. The default value is ten seconds. If this is too long for you, you can reduce it. This will increase the frequency of changes in the ISR, as followers will be removed and added more frequently.

In RabbitMQ, there is just a set of mirrors that need to be replicated. Slow mirrors introduce additional latency, and responses from dead mirrors can be delayed until the timeout of the packets that check the availability of each node (net tick). The ISR is an interesting way to avoid these latency issues. However, we risk losing redundancy since the ISR can only shrink down to the leader. To mitigate this risk, use the setting min.insync.replicas.

Client connection guarantees

In settings bootstrap.servers for producers and consumers, multiple brokers can be specified for client connections. The idea is that when one node goes down, several backups remain available for the client to open a connection. These do not necessarily have to be the partition leaders, but rather serve as a platform for initial loading. The client can ask them which node the partition leader for read/write is located on.

In RabbitMQ, clients can connect to any node, and internal routing directs requests accordingly. This means you can place a load balancer in front of RabbitMQ. Kafka requires clients to connect to the node where the leader of the relevant partition is located. In such cases, a load balancer cannot be used. The list bootstrap.servers is critical so that clients can access the necessary nodes and find them after a failure.

Kafka consensus architecture

Thus far, we have not discussed how the cluster becomes aware of a broker failure and how a new leader is chosen. To understand how Kafka deals with network partitions, it is essential first to grasp the consensus architecture.

Each Kafka cluster is deployed alongside a Zookeeper cluster — a distributed consensus service that allows the system to reach consensus on a certain given state with a priority on consistency over availability. A majority agreement of Zookeeper nodes is required for the approval of read and write operations.

Zookeeper maintains the state of the cluster:

• The list of topics, partitions, configuration, current leader replicas, preferred replicas.
• Cluster members. Each broker pings the Zookeeper cluster. If it does not receive a ping within a specified timeframe, Zookeeper marks the broker as unavailable.
• Selecting the primary and backup nodes for the controller.

The controller node is one of the Kafka brokers responsible for electing leaders for replicas. Zookeeper sends notifications about cluster membership and topic changes to the controller, which must act upon these changes.

For example, consider a new topic with ten partitions and a replication factor of 3. The controller must elect a leader for each partition, attempting to distribute the leaders optimally among the brokers.

For each partition, the controller:

• updates Zookeeper with information about the ISR and the leader;
• sends a LeaderAndISRCommand to each broker that hosts a replica of that partition, informing the brokers of the ISR and the leader.

When a broker with the leader fails, Zookeeper sends a notification to the controller, which then selects a new leader. Again, the controller first updates Zookeeper and then sends a command to each broker, notifying them of the leadership change.

Each leader is responsible for a set of ISR. The configuration replica.lag.time.max.ms determines who will be included. When the ISR changes, the leader conveys the new information to Zookeeper.

Zookeeper is always informed of any changes so that in the event of a failure, the leadership can smoothly transition to a new leader.

Fig. 21. Kafka Consensus

Replication Protocol

Understanding the details of replication helps better comprehend potential data loss scenarios.

Fetch requests, Log End Offset (LEO), and Highwater Mark (HW)

We reviewed that followers periodically send fetch requests to the leader. The default interval is 500 ms. This differs from RabbitMQ in that replication is initiated by the master rather than the mirror of the queue. The master pushes changes to the mirrors.

The leader and all followers maintain the Log End Offset (LEO) and the Highwater mark (HW). The LEO mark stores the offset of the last message in the local replica, while HW indicates the offset of the last commit. Remember, for a status of 'commit', the message must be stored in all replicas ISR. This means that LEO usually slightly precedes HW.

When the leader receives a message, it stores it locally. The follower sends a fetch request, passing its LEO. The leader then sends a batch of messages starting from this LEO and also provides the current HW. When the leader receives information that all replicas have stored the message at the specified offset, it moves the HW mark. Only the leader can move HW, and thus all followers learn the current value in their response to requests. This means that followers may lag behind the leader in both messages and knowledge of HW. Consumers only receive messages up to the current HW.

Note that 'persisted' means written to memory, not to disk. For performance, Kafka syncs to disk at a specified interval. RabbitMQ also has such an interval but will only acknowledge the publisher once the master and all mirrors have written the message to disk. Kafka developers opted for performance reasons to send an ack as soon as the message is written to memory. Kafka bets that redundancy will offset the risk of temporarily storing confirmed messages only in memory.

Leader Failure

When the leader fails, Zookeeper notifies the controller, which then selects a new leader replica. The new leader establishes a new HW mark according to its LEO. The followers then receive information about the new leader. Depending on the Kafka version, the follower will choose one of two scenarios:

1. It will truncate the local log to the known HW and send a request to the new leader for messages after this mark.
2. Sends a request to the leader to determine the HW at the time of their selection as leader, and then truncates the log to that offset. It will then start making periodic fetch requests, beginning from that offset.

A follower may need to truncate the log for the following reasons:

• When a leader failure occurs, the first follower in the ISR set registered in Zookeeper wins the election and becomes the leader. All followers in the ISR, although considered 'synchronized', may not have received all messages from the former leader. It is possible that the elected follower does not have the most up-to-date copy. Kafka guarantees that there is no divergence between replicas. Therefore, to avoid divergence, each follower must truncate its log to the HW value of the new leader at the time of their election. This is another reason why configuration acks=all is important for consistency.
• Messages are periodically written to disk. If all cluster nodes fail simultaneously, replicas with different offsets will be saved on the disks. It's possible that when the brokers return to the network, the new leader elected may be behind its followers because it was saved to disk earlier than the others.

Reconnecting to the cluster

When reconnecting to the cluster, replicas behave similarly to during a leader failure: they check the leader's replica and truncate their log to its HW (at the time of election). In contrast, RabbitMQ treats reconnected nodes as completely new. In both cases, the broker discards any existing state. If automatic synchronization is used, the master must replicate absolutely all current content to the new mirror in a 'let the whole world wait' manner. During this operation, the master does not accept any read or write operations. This approach creates issues in large queues.

Kafka is a distributed log, and overall, it retains more messages than RabbitMQ queues, where data is removed after being read. Active queues need to remain relatively small. However, Kafka is a log with its own retention policy, which can set a duration in days or weeks. The approach of locking the queue and full synchronization is absolutely unacceptable for a distributed log. Instead, Kafka followers simply truncate their log to the HW leader (at the time of its election) if their copy is ahead of the leader. In the more likely case where the follower is behind, it simply starts making fetch requests from its current LEO.

New or rejoined followers start outside the ISR and do not participate in commits. They simply work alongside the group, receiving messages as quickly as they can until they catch up to the leader and enter the ISR. There is no locking here, and all their data does not need to be discarded.

Network Partition

Kafka has more components than RabbitMQ, so there is a more complex set of behaviors when connectivity is disrupted in the cluster. However, Kafka was originally designed for clusters, so the solutions are well thought out.

Below are several scenarios of connectivity disruption:

• Scenario 1. The follower does not see the leader but still sees Zookeeper.
• Scenario 2. The leader does not see any followers but still sees Zookeeper.
• Scenario 3. The follower sees the leader but does not see Zookeeper.
• Scenario 4. The leader sees followers but does not see Zookeeper.
• Scenario 5. The follower is completely isolated from both other Kafka nodes and Zookeeper.
• Scenario 6. The leader is completely isolated from both other Kafka nodes and Zookeeper.
• Scenario 7. The Kafka controller node does not see another Kafka node.
• Scenario 8. The Kafka controller does not see Zookeeper.

Each scenario has its own behavior.

Scenario 1. The follower does not see the leader but still sees Zookeeper.

Figure 22. Scenario 1. ISR of three replicas.

Connectivity disruption separates broker 3 from brokers 1 and 2 but not from Zookeeper. Broker 3 can no longer make fetch requests. After a timeout. replica.lag.time.max.ms It is removed from the ISR and does not participate in message commits. Once connectivity is restored, it will resume fetching requests and join the ISR as it catches up with the leader. Zookeeper will continue to receive pings and regard the broker as alive and well.

Fig. 23. Scenario 1. The broker is removed from the ISR if it does not receive a fetch request within the replica.lag.time.max.ms interval.

There is no logical split-brain or node suspension like in RabbitMQ. Instead, redundancy decreases.

Scenario 2. The leader sees no followers but still sees Zookeeper.

Fig. 24. Scenario 2. The leader and two followers.

Network connectivity issues separate the leader from the followers, but the broker still sees Zookeeper. As in the first scenario, the ISR shrinks, but this time only to the leader, as all followers stop sending fetch requests. Again, there is no logical separation. Instead, there is a loss of redundancy for new messages until connectivity is restored. Zookeeper continues to receive pings and considers the broker alive and well.

Fig. 25. Scenario 2. The ISR has shrunk down to just the leader.

Scenario 3. A follower sees the leader but does not see Zookeeper.

The follower is separated from Zookeeper but not from the broker with the leader. As a result, the follower continues to make fetch requests and remains a member of the ISR. Zookeeper no longer receives pings and registers the broker's failure, but since this is only a follower, there are no consequences upon recovery.

Fig. 26. Scenario 3. The follower continues to send fetch requests to the leader.

Scenario 4. The leader sees followers but does not see Zookeeper.

Fig. 27. Scenario 4. The leader and two followers.

The leader is isolated from Zookeeper but not from the brokers with the followers.

Fig. 28. Scenario 4. The leader is isolated from Zookeeper.

After a while, Zookeeper will register the broker's failure and notify the controller. It will select a new leader from the followers. However, the original leader will continue to believe it is the leader and will keep accepting records with acks=1. Followers no longer send it fetch requests, so it will consider them dead and attempt to shrink the ISR to just itself. But since it has no connection to Zookeeper, it will not be able to do this and at that point will refuse to accept further records.

Messages acks=all will not receive acknowledgment because the ISR initially includes all replicas, and messages do not reach them. When the original leader tries to remove them from the ISR, they will be unable to do so and will stop receiving any messages at all.

Clients quickly notice the change in the leader and begin sending records to the new server. Once the network is restored, the original leader sees that they are no longer the leader and trims their log to the HW value that the new leader had at the time of the failure to avoid log divergence. They will then begin sending fetch requests to the new leader. All records from the original leader that were not replicated to the new leader will be lost. That is, messages not acknowledged by the original leader during those few seconds when there were two leaders operating will be lost.

Fig. 29. Scenario 4. The leader on broker 1 becomes a follower after network recovery

Scenario 5. The follower is fully isolated from other Kafka nodes and Zookeeper

The follower is completely isolated from other Kafka nodes and Zookeeper. It is simply removed from the ISR until the network is restored, after which it catches up with the others.

Fig. 30. Scenario 5. The isolated follower is removed from the ISR

Scenario 6. The leader is fully isolated from other Kafka nodes and Zookeeper

Fig. 31. Scenario 6. Leader and two followers

The leader is completely isolated from its followers, the controller, and Zookeeper. For a brief period, it will continue to accept records from acks=1.

Fig. 32. Scenario 6. Leader isolation from other Kafka nodes and Zookeeper

Having not received requests after replica.lag.time.max.ms, it will attempt to shrink the ISR to itself but will be unable to do so due to a lack of connection with Zookeeper; it will then stop accepting records.

Meanwhile, Zookeeper will mark the isolated broker as dead, and the controller will select a new leader.

Fig. 33. Scenario 6. Two leaders

The original leader can accept records for a few seconds but then stops accepting any messages. Clients refresh every 60 seconds with the latest metadata. They will be informed of the change in leader and will start sending records to the new leader.

Fig. 34. Scenario 6. Producers switch to the new leader

All confirmed records made by the original leader since the loss of connectivity will be lost. Once the network is restored, the original leader will detect through Zookeeper that it is no longer the leader. It will then truncate its log to the HW of the new leader at the time of the election and start sending requests as a follower.

Fig. 35. Scenario 6. The original leader becomes a follower after the network connectivity is restored.

In this situation, a logical split may occur for a short period, but only if acks=1 and min.insync.replicas 1. The logical split automatically resolves either after the network is restored when the original leader realizes it is no longer the leader, or when all clients understand that the leader has changed and begin writing to the new leader — depending on which happens first. In any case, some messages will be lost, but only with acks=1.

There is another variant of this scenario where right before the network partition the followers fell behind, and the leader compressed the ISR to just itself. It then becomes isolated due to the loss of connectivity. A new leader is elected, but the original leader continues to accept records, even acks=all, because there is no one else in the ISR besides itself. These records will be lost after the network is restored. The only way to avoid such a variant is min.insync.replicas = 2.

Scenario 7. The Kafka controller does not see another Kafka node.

Generally, after losing connection to the Kafka node, the controller will not be able to send any leader change information to it. In the worst case, this will lead to a short-term logical split, as in scenario 6. Most often, the broker simply will not become a candidate for leadership in case of the latter's failure.

Scenario 8. The Kafka controller does not see Zookeeper.

The disconnected controller will not receive a ping from Zookeeper and will elect a new Kafka node as the controller. The original controller may continue to present itself as such, but it does not receive notifications from Zookeeper, so it will have no tasks to perform. Once the network is restored, it will realize it is no longer the controller but has become a regular Kafka node.

Conclusions from the scenarios.

We observe that the loss of connectivity among followers does not lead to message loss, but merely temporarily reduces redundancy until the network is restored. This, of course, can lead to data loss if one or more nodes are lost.

If a leader becomes isolated from Zookeeper due to loss of connectivity, this can lead to message loss with acks=1. Lack of connection to Zookeeper causes a brief logical split with two leaders. This issue is addressed by the parameter acks=all.

Parameter min.insync.replicas to two or more replicas provides additional guarantees that such short-term scenarios will not result in message loss, as in scenario 6.

Summary of message loss

Let's enumerate all the ways data can be lost in Kafka:

• Any leader failure if messages were acknowledged using acks=1
• Any unclean leadership transition, that is, to a follower outside of the ISR, even with acks=all
• Isolation of the leader from Zookeeper, if messages were acknowledged using acks=1
• Complete isolation of the leader, which has already compressed the ISR group to itself. All messages will be lost, even acks=all. This is true only if min.insync.replicas=1.
• Simultaneous failures of all nodes in the partition. Since messages are acknowledged from memory, some may not yet have been written to disk. After the servers reboot, some messages may be missing.

Unclean leadership transitions can be avoided either by prohibiting them or by ensuring redundancy of at least two. The most robust configuration is a combination of acks=all and min.insync.replicas more than 1.

Direct comparison of RabbitMQ and Kafka reliability

To ensure reliability and high availability, both platforms implement a primary and secondary replication system. However, RabbitMQ has a vulnerability. Upon reconnection after a failure, nodes discard their data, and synchronization is blocked. This double blow questions the longevity of large queues in RabbitMQ. You will have to either accept reduced redundancy or long blockages. Reducing redundancy increases the risk of massive data loss. However, if the queues are small, then for the sake of ensuring redundancy with short downtime periods (a few seconds), reconnection attempts can be managed.

In Kafka, there is no such problem. It only drops data at the point of divergence between the leader and the follower. All common data is retained. Furthermore, replication does not block the system. The leader continues to accept entries while the new follower catches up, making it a trivial task for DevOps to join or rejoin the cluster. Of course, issues still remain, such as network bandwidth during replication. If multiple followers are added simultaneously, you may encounter bandwidth limits.

RabbitMQ outperforms Kafka in reliability when multiple servers in a cluster fail at the same time. As mentioned, RabbitMQ sends confirmations to the publisher only after the message is written to the disk by the master and all mirrors. But this adds additional latency for two reasons:

• fsync every few hundred milliseconds
• Mirror failures can only be detected after the timeout of packets that check the availability of each node (net tick). If a mirror lags or goes down, this adds latency.

Kafka bets on the premise that if a message is stored on multiple nodes, messages can be acknowledged as soon as they are in memory. This creates a risk of losing messages of any type (even acks=all, min.insync.replicas=2) in the event of simultaneous failures.

Overall, Kafka demonstrates higher performance and is originally designed for clusters. The number of followers can be increased to 11 if necessary for reliability. A replication factor of 5 and a minimum number of replicas in a synchronized state min.insync.replicas=3 will make message loss a very rare event. If your infrastructure can support such a replication factor and level of redundancy, you can choose this option.

RabbitMQ clustering is good for small queues. However, even small queues can quickly grow under high traffic. Once queues become large, a tough choice must be made between availability and reliability. RabbitMQ clustering is best suited for atypical scenarios where the advantages of RabbitMQ's flexibility outweigh any shortcomings of its clustering.

One remedy for the RabbitMQ vulnerability regarding large queues is to break them down into many smaller ones. If strict ordering of the entire queue is not required, but only of the relevant messages (e.g., messages from a specific client), or if no ordering is needed at all, then this option is acceptable: check out my project. Rebalancer for breaking up the queue (the project is still in its early stages).

Finally, don't forget about several bugs in the clustering and replication mechanisms of both RabbitMQ and Kafka. Over time, the systems have become more mature and stable, but no message is ever 100% protected against loss! Additionally, large-scale failures can occur in data centers!

If I've missed something, made a mistake, or if you disagree with any of the points, feel free to leave a comment or contact me.

I am often asked, "Which should I choose, Kafka or RabbitMQ? Which platform is better?" The truth is, it really depends on your situation, current experience, etc. I hesitate to express my opinion, as it would be too much of a simplification to recommend one single platform for all use cases and potential limitations. I wrote this series of articles to help you form your own opinion.

I want to say that both systems are leaders in this field. I might be a bit biased because my experience with projects has led me to value aspects such as guaranteed message ordering and reliability more.

I see other technologies lacking this reliability and guaranteed ordering, then I look at RabbitMQ and Kafka — and I understand the incredible value of both of these systems.

Source: habr.com