3.11: two Raft replicas are in the timeout state, one is a candidate

[gsingla294]

Describe the bug

We have a 3 node MQ cluster deployed on Azure AKS with MQ version 3.11.24
But repeatedly 2 out of 3 nodes goes into raft state as timeout and consumers stop receiving the data and effectively a downtime occurs.
This is occurring very frequently. Need guidance on how to troubleshoot.

Reproduction steps

Create a 3 node MQ cluster on AKS
I might not have steps to reproduce it as there is not specific pattern and it is happening randomly.

Expected behavior

Cluster should work well and raft state should not go into timeout state.

Additional context

No response

[michaelklishin]

RabbitMQ 3.11 is out of community support. There will be no guidance besides the absolute basics.

1. Please upgrade to 3.13.1
2. See logs from all nodes for clues
3. Consider how the nodes were added or removed. For example, the following upgrade strategy is very Raft-unfriendly, and requires extra care during the upgrade process (covered in the docs) when nodes leave the cluster

Node identity matters for Raft-based features, so if you (or AKS) remove a node, it must be done explicitly, simply nuking a pod is not good enough, or rather it eventually may have side effects on the rest of the cluster, similar to what you are observing: the replicas cannot elect a leader for one reason or another.
If this hypothesis turns out to be correct, then 3.13 will largely suffer from the same behavior eventually.

#10786 is one example of how removing nodes aggressively during a grow-then-shrink upgrade can lead to certain queues or streams not being able to elect a new leader because their original peers are gone without explicit removal.

Please take it from here.