RabbitMQ authentication failed after pod rescheduled to different node

[CassieZAHNG]

Hey, I have a cluster consist of 3 rabbitmq pods allocated at different node.

Those pods are managed by statefulset. I got an issue that, if a pod is rescheduled to a new node (e.g., previous node is down), then its peers (another 2 pods) will try to reset the new rescheduled pod, and then the rescheduled pod will keep restarting and throws connection authentication issue when other service trying to reconnect it.

error like:
Liveness probe failed: Get "http://10.200.36.128:3030/ping": read tcp 10.200.36.118:35532->10.200.36.128:3030: read: connection reset by peer

Do you have any idea about this?

[mkuratczyk]

/ping is not a RabbitMQ endpoint.

Hard to tell what happens exactly but:

1. do you use local storage? if a pod is restarted on a new node, does it start fresh or does it start with the state of the previous incarnation?
2. what partition handling strategy do you use? https://www.rabbitmq.com/partitions.html#automatic-handling