RabbitMQ: Working classic queues in "stopped" state after cluster issues with network partitions

[motmot80]

Environment:

• RabbitMQ version: 3.9.13
• Erlang version: 24.2.1
• Cluster: yes
• Production system: yes

Reproduction

1. one VM of a rmq cluster node was paused for about 3 minutes
   => causing network partitions in cluster and permanent errors (saying: queue home node down or inaccessible)
2. round robin restart reboot of all nodes
   => errors went away
3. some classic-queues (non durable) now are in "stopped" state.

Analysis
Both Admin-UI and rabbitmqctl list_queues showing the queue as "stopped" and show no other information.
All RMQ nodes in the cluster are showing the same queue state. No PID is shown.
BUT these queues are working and permanently consuming and publish messages.
It seems like only metrics are not working for these queues.
rabbitmqctl list_unresponsive_queues and rabbitmqctl list_queues --offline do not show these queues.

Questions

• What does "stopped" state mean for classic non-mirrored non-durable queues?
• How to "restart" these queues without restarting the cluster (production impact)?

[michaelklishin]

"Stopped" means that that a queue process successfully terminated. For transient queues this probably involved migration to another node but due to node disconnection the state was not updated.

Since the queue is transient, sounds like deleting it and allowing the apps to re-declare/rebind them would be one option.