Statuses of classic queues are reported incorrectly after server restart

[grishin-a]

Describe the bug

Hi,
I'm testing Rabbitmq HA with Openstack kolla-ansible classic installation.
Rabbitmq service is deployed in containers using Ubuntu 22.04.1, on 3 host cluster using quorum+classic queues.
I've noticed, when the Rabbitmq service is restarted on the host that owns the classic queue, the queue moves to(succesfully recreates on) any other running Rabbitmq host. Next, we restart the service on the next owner of the queue (after correct restoration of the cluster). In our configuration, the queue will essentially migrate between two of the three hosts due to the Rabbitmq queue_master_locator = min-masters setting
With several similar iterations of Rabbitmq service restart, Rabbitmq informs reconnecting clients(catched with Wireshark) that the queue exists, but in fact it does not exist - as a result, the clients does not recreate its queues when reconnecting.

Reproduction steps

1. Create 3 node cluster
2. Create classic queue
3. Restart Rabbitmq service on classic queue owner host
4. Wait for cluster recovery and classic queue moves to(succesfully recreates on) another host
5. Restart Rabbitmq service on new classic queue owner host

This simple script I'm using to automate this check:

#!/usr/bin/bash

connect_host="control-3"
host_list="control-2 control-1"
reply="reply_351f90757a75436287b313f0eb076e46"
iter=1
tries=20

while [[ "${iter}" != "${tries}" ]]; do
  for host in ${host_list}; do
    if ssh ${connect_host} "sudo docker exec -u0 rabbitmq rabbitmqctl list_consumers -sq --no-table-headers" | grep -wq ${reply}; then
      echo "iter:${iter} service restart on: ${host}"
      ssh ${host} "sudo docker restart rabbitmq" > /dev/null
      sleep 10
    else
      echo "Queue is lost - iter:${iter} service restart on: ${host}"
      exit 0
    fi
  done
((++iter))
done

in script variables:
connect_host - the host to which the queue does not move because queue_master_locator = min-masters, this host is used for control
host_list - hosts between which the queue moves
reply - classic queue being monitored

Expected behavior

Rabbitmq informs reconnecting clients about queues correctly

Additional context

Tested on:
RabbitMQ 3.12.0, 3.12.6
Erlang 26.1.1
oslo.messaging 14.0.1
amqp 5.1.1
kombu 5.2.4

just in case, cluster_partition_handling = pause_minority

[lukebakken]

Please provide your full RabbitMQ configuration as well as an export of the definitions from your cluster. ATTACH the output to your response (do not paste it into the comment box!)

[grishin-a]

extra.conf.txt
advanced.config.txt
definitions.json
enabled_plugins.txt
rabbitmq-env.conf.txt
rabbitmq.conf.txt

[michaelklishin]

It depends on how you define whether a queue "exists" or not. A queue can exist but have no elected leader: does it still exist in that case?

What I am getting at is that nodes can report existence of a queue in its schema database but the queue may or may not have an elected leader. Specifically classic mirrored queues can end up without an electable leader in certain scenarios.
This will be very visible in the logs, the message to look for is specific and well documented.

Classic mirrored queues have been deprecated for several years now. Sorry but our team won't spend any more time on classic mirrored queues: their design is unfixable. Quorum queues have been around for some five years now, and streams for a couple of years. They do not exibit this behavior as long as a majority of nodes is online (and thus a new leader can be elected).

[michaelklishin]

Relying on non-mirrored classic queue migration is also very nuanced and the solution is more or less the same:

• Assume that non-mirrored classic queues can become unavailable or "lost" during a rolling restart. Always re-declare them if you need to use a non-mirrored queue type, ro
• Use quorum queues that provide a well defined, efficient (as far as data transfers go in particular) leader election as long as a majority of nodes [hosting QQ replicas] are online

Nodes will report non-mirrored classic queues as existing even if no replica has been started
yet. Any client operation on a non-replicated queue will generally not be retried when the replica
is starting, restarting, migrating, etc.

On the contrary, operations on a quorum queue without a current leader are delayed by channels (connections) and executed when the leader election successfully finishes. This means that applications generally don't have to worry or know if and when an election happened. We are not aware of any widely reported issues with this QQ retry mechanism.

[grishin-a]

Thank you for your answer! But I'm not using mirrored classic queues, this happens with just a classic queues. About rolling restart - I don't think my restart is "rolling" - it is about 30sec-1min delay between service restarts and from Rabbitmq logs I can see, that restarted nodes enters cluster successfully. But I'll try to increase the delay between restarts in my tests.
How I see this behavior:

1. Client has a pool of RabbitMQ nodes to connect to - node1, node2, node3
2. Client connects to node1, creates a classic non mirrored queue, that appears on node3
3. We restart RabbitMQ service on node3
4. Client creates a queue again, that appears on node2
5. We restart RabbitMQ service on node2 after node3 enter cluster successfully
6. Client creates a queue again, that appears on node3

after some iterations like that, connected to node1 client gets an answer from RabbitMQ node1, that queue still exists but in fact it does not - and client is not redeclaring that queue.
May be there is a bug in oslo.messaging, I don't know yet - all that I find for now that client before redeclaring queue gets an answer from RabbitMQ that a queue is still exists - but in fact - it does not.

[michaelklishin]

We are guessing as to what specifically happened in your cluster since there are no logs or any examples of what the client tries to do.

Non-replicated queue types are meant to be used for queues transient in nature. Which means that applications can likely delete and re-declare them at will.

A "passive queue declaration" right now only checks for the queue's entry in the schema database, and does not try to verify whether the queue has a fully booted leader. Which should not really matter if you delete + re-declare your semantically transient queues.

And for all others, use quorum queues or streams.