Slow operations in degraded cluster

[koeberlue]

We have a rabbitmq cluster of three nodes. One of our nodes crashed unexpectedly.

We discovered, that operations like declaring queues would now take significant longer than before, say five to seven seconds per operation.

In another discussion (#9194 (comment)) it is stated that in this scenario some operations will take longer because all nodes need to be contacted and we have to wait for timeouts, which is totally understandable.

But two things are not quite clear to me:

1. Doss this affect operations like declaring queues as well?
2. Is the cluster expected to perform these operations slowly until the offline node comes back online again? Or do the online nodes after some time "understand" that the third node is offline and skip it when performing cluster wide operations?

I checked the documentation, issues and discussions but could not find a clear answer. But maybe I just worded my searches poorly.

[michaelklishin]

@koeberlue it may. For example, if the node leader locator used on the node has to contact all nodes, then yes. The node local one would not contact any peers but others likely will.

[koeberlue]

@michaelklishin Thank you for the fast reply. This makes sense to me.

But, what I forgot to mention, we mainly use non-replicated, transient, auto-generated queues. Is it possible that when creating those, the offline node has to be checked as well?

[michaelklishin]

Queue leader locators do not take the type of properties of the queue into account. They use a list of nodes and arbitrary logic to pick the optimal node, e.g. requesting the number of replicas hosted on each node from that node itself.

[koeberlue]

Ah, okay, got it! Thank you very much for taking the time to respond.

[michaelklishin]

No worries. If you do some testing and discover where the degree of degradation ends up unreasonably high (e.g. it should not take 20 seconds to declare a queue, even 5s would be very problematic in some systems), please put together a spreadsheet with your findings and file an issue.

I am certain that the delay can be reduced somewhat by adjusting the timeouts various queue leader locator implementations use. They very likely roll with all defaults, which in Erlang often means a 5s timeout.

Note that inter-node communication timeouts/tick time are also relevant. Just don't set those too low.