pause_minority Cluster Partition Handling does not work as expected #8111
Replies: 9 comments 14 replies
-
Please provide the exact commands you are using to do this. Right now you're asking us to guess how to reproduce the issue the same way you have. In addition, please attach your complete configuration file. |
Beta Was this translation helpful? Give feedback.
-
I can't unfortunately, this was caused by a connection drop in the company network, don't know a way to reproduce this reliably locally. Would providing any additional information help in this case? |
Beta Was this translation helpful? Give feedback.
-
|
The cluster is not expected to pause on a single partition since two nodes will be in a majority. Nodes do not necessarily detect such conditions instantly, the default inactivity timeout is 60 seconds IIRC, and can be configured but not to a value that's too low (say, not 5s). |
Beta Was this translation helpful? Give feedback.
-
|
In addition, it's important to distinguish scenarios where, say, node A loses connection to B and C. Then it will pause itself as it will be in the minority. A much trickier situation is when A disconnects from B but not from C. In this case some nodes will In RabbitMQ 4.0, these partition handling strategies will go away. The recovery strategy with Khepri will be that of Raft: the majority of nodes keeps going, and a majority of nodes must always be online, or your cluster will lose availability. So, only the connectivity of each replica to the currently elected leader controls whether it is available or needs to recover, if we oversimplify. |
Beta Was this translation helpful? Give feedback.
-
|
I still don't get it, node 3 was disconnected from 1 and 2, so not connected to any node at all. Why didn't it pause then? (been in this state for 3 days) |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
Checked the history, I tested 25672 so was just a typo. After a service restart ( |
Beta Was this translation helpful? Give feedback.
-
|
That's what I was going to suggest, restarting the disconnected node. This is also what is in the docs. As to why the node wasn't paused, I'm not sure. I'll try to reproduce when I have time. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for looking into this 👍 . Yes, the main issue is that the node isn't paused and therefore still can receive and send messages until manual intervention. |
Beta Was this translation helpful? Give feedback.
-
|
I also found an old bug report describing a similar issue (but in an OpenShift cluster):
|
Beta Was this translation helpful? Give feedback.
-
|
This is the complete config: rabbitmq.confadvanced.config enabled_plugins |
Beta Was this translation helpful? Give feedback.
-
|
@justsomescripts the This doesn't appear to be a 3-node cluster...? |
Beta Was this translation helpful? Give feedback.
-
Yes, three node cluster. Happened during cleanup. Here's the correct output: Node 1Node 3 |
Beta Was this translation helpful? Give feedback.
-
|
Please note that I have used the |
Beta Was this translation helpful? Give feedback.
-
|
@justsomescripts I couldn't reproduce what you report using this project - https://github.com/lukebakken/docker-rabbitmq-cluster When I disconnect a container from the network, RabbitMQ on that node pauses as expected. When I re-connect the container, I can see RabbitMQ try to start. What I did find, however, is this serious bug - #8114 |
Beta Was this translation helpful? Give feedback.
-
|
Interesting 🤔 . Thanks for looking into this, I'll try if I can somehow replicate this in a Docker / local Dev environment when I have time. |
Beta Was this translation helpful? Give feedback.
-
|
I encountered a similar problem. The Reproduction steps: Pause any of the virtual hosts in VMware vSphere, wait for 30 seconds to 1 minute until the Kubernetes cluster reports that the node is unavailable, and then resume the host from the pause. As a result, the RabbitMQ cluster will split, with the cluster divided into two parts – 2 nodes and 1 node will operate in parallel. The nature of the partition is as follows:
At the same time, none of the cluster state check commands will indicate that the 2-node group is in the minority. Examples: I would expect that in such a scenario, the cluster node that is in the minority would at least shut down its port so that applications can't connect to it, but that's not the case. Example of checking port availability from inside the container: Example of checking port availability from another Kubernetes namespace: Could you suggest an alternative solution, other than manually restarting the node as mentioned in the documentation, or waiting for version 4.0? |
Beta Was this translation helpful? Give feedback.
-
|
Hijacking an existing discussion is not particularly nice. It's hard to suggest much without logs from all nodes, generally nodes log at the very least
I'm also not sure why you verify connectivity on port 5672, that's not the port used for inter-node communication but rather for client connections. Pausing a VM has more side effects than interrupting its network connections, as extensively described in #10701. This is not a commonly reported issue and You can try Finally, 4.0 is expected to ship in Sep-Oct but you will have to run Khepri, still a relatively young "feature" FWIW. |
Beta Was this translation helpful? Give feedback.
-
|
@CrazyMushu can you please move it into a separate discussion and share logs from all nodes there? |
Beta Was this translation helpful? Give feedback.
-
|
https://youtu.be/y2HAJBiXsw0?feature=shared&t=1232 is one example of |
Beta Was this translation helpful? Give feedback.
-
|
FTR, https://youtu.be/y2HAJBiXsw0?feature=shared&t=1232 is one example of |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
When the connection of multiple nodes in a RabbitMQ cluster is interrupted at the same time, the partition that has the minority of nodes is still running and accepting messages.
RabbitMQ version: 3.11.13
Erlang version: 25.3
Host: Ubuntu 22.04.2 LTS
Reproduction steps
cluster_partition_handlingtopause_minorityAll nodes still accept messages despite
cluster_partition_handlingset topause_minorityand being partitioned.After a restart of the single node, the cluster works again as expected.
Expected behavior
RabbitMQ is paused on the single node partition.
Additional context
Output of the first nodes (partition of two)
Output of the 3rd node (partition of one)
Relevant log
Configuration
Beta Was this translation helpful? Give feedback.
All reactions