Rabbitmq cluster high availability problem of version 3.9.27

[yule-sun]

Describe the bug

I recently deployed the Yoga version of OpenStack, and I found that when I shut down a rabbitmq node, launch a new instance would be stuck at the build step, and the migration of the virtual machine would fail. I tried use the command "docker exec -it rabbitmq rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'" to adjust the rabbitmq cluster to mirror mode, but it didn't work. How can I adjust the rabbitmq cluster so that the cluster can still be used normally when I shut down a rabbitmq node? Looking forward to your reply.

Reproduction steps

1. deploy the yoga version openstack cluster with kolla-ansible
2. shutdown a rabbitmq node
3. create a new instance
   ...

Expected behavior

Shutting down a rabbitmq node does not affect the creation and migration of virtual machines

Additional context

No response

[michaelklishin]

I am afraid “deploy an OpenStack cluster” is not a reproduction step we can accept. There are hundreds of potential variables involved.

Start with collecting node logs for clues. Make sure you understand what happens when nodes are restarted and what that means for various health checks your system may be using for readiness probes.

RabbitMQ 3.9 is out of support and classic mirrored queues have been deprecated for a couple of years now.

[yule-sun]

Oh sorry sir, I may not have expressed my meaning clearly. I mean I have OpenStack environment on my side. I can reproduce this issue and provide logs. Could you please help me with some suggestions? Which cluster mode should I use in version RabbitMQ 3.9?

This is my cluster status, i shutdown a node named L12D08Ctrl34 in my cluster :

The logs on L12D07Yoga30:

The logs on L12D09Ctrl35:

[michaelklishin]

The only error in these log snippets is about missed heartbeats from a client. And RabbitMQ nodes are not responsible for those, and cannot do anything about them. It could be a proxy or a faulty client.

There are no messages that have anything to do with node startup or clustering. Consider uploading full logs as files, not as screenshots.

[yule-sun]

This is the log of the three nodes,thank you very much for your help.
Desktop.zip