Classic mirrored queues ended up without an electable leader

[henrydn]

Describe the bug

We had an incident in production where a Mirrored queue stopped working and could not recover, but we cannot explain why. It seems as if the cluster just 'forgot' which node was the master for this queue.

Some background information:

• We are using the RabbitMQ Helm Charts from Bitnami deployed into an Azure AKS cluster
• RabbitMQ Version: (Version 3.9.13)
• RabbitMQ was scaled to 3 pods
• AKS scheduled all three pods onto the same physical node
• We managed to recover by deleting the queue and recreating it (No pods were restarted)
• During the whole incident, no pods restarted. All had an up time of 23 days.

Reproduction steps

1. Deploy RabbitMQ into AKS in a clustered setup with three nodes
2. Declare a Mirrored Classis Queue
3. ?
4. The queue stops working and publishing becomes impossible

Expected behavior

We expected that in a high availability setup, when one node encounters problems, another node would take over, and the queue would continue to function.

Additional context

This log entry coincides exactly with when we started experiencing issues with the queue:

2023-09-07 08:19:42.240695+00:00 [warn[] <0.27852.18> Mirrored queue '<REDACTED>' in vhost '/': Stopping all nodes on master shutdown since no synchronised mirror (replica) is available

But this message does not make sense because no pods were shut down/restarted. All were up for 23 days. There is also no sign of an actual restart in the logs (usually you would see a Starting broker... message upon restart).

Attempts to reconnect to the queue were met with timeouts:

2023-09-07 08:22:50.049288+00:00 [erro[] <0.15707.80> Channel error on connection <0.15689.80> (<REDACTED>:51620 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 2:
2023-09-07 08:22:50.049288+00:00 [erro[] <0.15707.80> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout

We eventually deleted and recreated the queue, and this solved the issue. This was shown in logs when queue was deleted:

2023-09-07 08:54:09.024129+00:00 [warn[] <0.28887.99> Queue <REDACTED> in vhost / has its master node is down and no mirrors available or eligible for promotion. Forcing queue deletion.

The logs for each instance is below. They are the only logs from the day of the incident.

Node 1:

2023-09-07 06:28:19.130744+00:00 [warn[] <0.198.0> <REDACTED>_sysmon_handler busy_dist_port <0.1098.0> [{name,delegate_management_0},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.23>,unknown}
2023-09-07 08:19:42.232443+00:00 [warn[] <0.669.49> closing AMQP connection <0.669.49> (<REDACTED>:44418 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:19:42.232443+00:00 [warn[] <0.669.49> client unexpectedly closed TCP connection
2023-09-07 08:21:43.848092+00:00 [info[] <0.15689.80> accepting AMQP connection <0.15689.80> (<REDACTED>:51620 -> <REDACTED>:5672)
2023-09-07 08:21:43.874937+00:00 [info[] <0.15689.80> Connection <0.15689.80> (<REDACTED>:51620 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:21:43.882158+00:00 [info[] <0.15689.80> connection <0.15689.80> (<REDACTED>:51620 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:22:22.042733+00:00 [warn[] <0.15689.80> closing AMQP connection <0.15689.80> (<REDACTED>:51620 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:22:22.042733+00:00 [warn[] <0.15689.80> client unexpectedly closed TCP connection
2023-09-07 08:22:50.049288+00:00 [erro[] <0.15707.80> Channel error on connection <0.15689.80> (<REDACTED>:51620 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 2:
2023-09-07 08:22:50.049288+00:00 [erro[] <0.15707.80> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:33:06.689397+00:00 [warn[] <0.667.49> closing AMQP connection <0.667.49> (<REDACTED>:35328 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:33:06.689397+00:00 [warn[] <0.667.49> client unexpectedly closed TCP connection
2023-09-07 08:39:54.179788+00:00 [info[] <0.17062.80> accepting AMQP connection <0.17062.80> (<REDACTED>:48428 -> <REDACTED>:5672)
2023-09-07 08:39:54.847130+00:00 [info[] <0.17062.80> Connection <0.17062.80> (<REDACTED>:48428 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:39:54.886559+00:00 [info[] <0.17062.80> connection <0.17062.80> (<REDACTED>:48428 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:40:32.083724+00:00 [warn[] <0.17062.80> closing AMQP connection <0.17062.80> (<REDACTED>:48428 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:40:32.083724+00:00 [warn[] <0.17062.80> client unexpectedly closed TCP connection
2023-09-07 08:41:04.827258+00:00 [erro[] <0.17079.80> Channel error on connection <0.17062.80> (<REDACTED>:48428 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 2:
2023-09-07 08:41:04.827258+00:00 [erro[] <0.17079.80> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:45:42.909009+00:00 [info[] <0.17515.80> accepting AMQP connection <0.17515.80> (<REDACTED>:46420 -> <REDACTED>:5672)
2023-09-07 08:45:44.369475+00:00 [info[] <0.17515.80> Connection <0.17515.80> (<REDACTED>:46420 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:45:44.373120+00:00 [info[] <0.17515.80> connection <0.17515.80> (<REDACTED>:46420 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:46:12.109121+00:00 [warn[] <0.17515.80> closing AMQP connection <0.17515.80> (<REDACTED>:46420 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:46:12.109121+00:00 [warn[] <0.17515.80> client unexpectedly closed TCP connection
2023-09-07 08:46:13.586190+00:00 [info[] <0.17591.80> accepting AMQP connection <0.17591.80> (<REDACTED>:33718 -> <REDACTED>:5672)
2023-09-07 08:46:14.343191+00:00 [info[] <0.17591.80> Connection <0.17591.80> (<REDACTED>:33718 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:46:14.373761+00:00 [info[] <0.17591.80> connection <0.17591.80> (<REDACTED>:33718 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:46:52.030381+00:00 [warn[] <0.17591.80> closing AMQP connection <0.17591.80> (<REDACTED>:33718 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:46:52.030381+00:00 [warn[] <0.17591.80> client unexpectedly closed TCP connection
2023-09-07 08:46:55.107346+00:00 [erro[] <0.17531.80> Channel error on connection <0.17515.80> (<REDACTED>:46420 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 2:
2023-09-07 08:46:55.107346+00:00 [erro[] <0.17531.80> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:47:22.832268+00:00 [erro[] <0.17601.80> Channel error on connection <0.17591.80> (<REDACTED>:33718 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:47:22.832268+00:00 [erro[] <0.17601.80> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout

Node 2:

2023-09-07 08:19:44.527819+00:00 [info[] <0.2040.91> accepting AMQP connection <0.2040.91> (<REDACTED>:33900 -> <REDACTED>:5672)
2023-09-07 08:19:46.500228+00:00 [info[] <0.2040.91> Connection <0.2040.91> (<REDACTED>:33900 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:19:46.504910+00:00 [info[] <0.2040.91> connection <0.2040.91> (<REDACTED>:33900 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:20:21.939264+00:00 [warn[] <0.2040.91> closing AMQP connection <0.2040.91> (<REDACTED>:33900 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:20:21.939264+00:00 [warn[] <0.2040.91> client unexpectedly closed TCP connection
2023-09-07 08:20:23.114000+00:00 [info[] <0.2159.91> accepting AMQP connection <0.2159.91> (<REDACTED>:40694 -> <REDACTED>:5672)
2023-09-07 08:20:25.043588+00:00 [info[] <0.2159.91> Connection <0.2159.91> (<REDACTED>:40694 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:20:25.047320+00:00 [info[] <0.2159.91> connection <0.2159.91> (<REDACTED>:40694 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:20:50.548709+00:00 [erro[] <0.2052.91> Channel error on connection <0.2040.91> (<REDACTED>:33900 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:20:50.548709+00:00 [erro[] <0.2052.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:21:01.936960+00:00 [warn[] <0.2159.91> closing AMQP connection <0.2159.91> (<REDACTED>:40694 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:21:01.936960+00:00 [warn[] <0.2159.91> client unexpectedly closed TCP connection
2023-09-07 08:21:30.475605+00:00 [erro[] <0.2181.91> Channel error on connection <0.2159.91> (<REDACTED>:40694 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:21:30.475605+00:00 [erro[] <0.2181.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:22:06.535639+00:00 [erro[] <0.2260.91> Channel error on connection <0.2159.91> (<REDACTED>:40694 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 4:
2023-09-07 08:22:06.535639+00:00 [erro[] <0.2260.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:33:23.407613+00:00 [info[] <0.3511.91> accepting AMQP connection <0.3511.91> (<REDACTED>:42534 -> <REDACTED>:5672)
2023-09-07 08:33:24.355329+00:00 [info[] <0.3511.91> Connection <0.3511.91> (<REDACTED>:42534 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:33:24.367901+00:00 [info[] <0.3511.91> connection <0.3511.91> (<REDACTED>:42534 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:34:02.027799+00:00 [warn[] <0.3511.91> closing AMQP connection <0.3511.91> (<REDACTED>:42534 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:34:02.027799+00:00 [warn[] <0.3511.91> client unexpectedly closed TCP connection
2023-09-07 08:34:31.142630+00:00 [erro[] <0.3531.91> Channel error on connection <0.3511.91> (<REDACTED>:42534 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:34:31.142630+00:00 [erro[] <0.3531.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:39:13.804427+00:00 [info[] <0.4160.91> accepting AMQP connection <0.4160.91> (<REDACTED>:47068 -> <REDACTED>:5672)
2023-09-07 08:39:14.266256+00:00 [info[] <0.4160.91> Connection <0.4160.91> (<REDACTED>:47068 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:39:14.279213+00:00 [info[] <0.4160.91> connection <0.4160.91> (<REDACTED>:47068 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:39:52.144022+00:00 [warn[] <0.4160.91> closing AMQP connection <0.4160.91> (<REDACTED>:47068 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:39:52.144022+00:00 [warn[] <0.4160.91> client unexpectedly closed TCP connection
2023-09-07 08:40:26.756624+00:00 [erro[] <0.4169.91> Channel error on connection <0.4160.91> (<REDACTED>:47068 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:40:26.756624+00:00 [erro[] <0.4169.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:52:02.829566+00:00 [info[] <0.5474.91> accepting AMQP connection <0.5474.91> (<REDACTED>:48286 -> <REDACTED>:5672)
2023-09-07 08:52:04.769243+00:00 [info[] <0.5474.91> Connection <0.5474.91> (<REDACTED>:48286 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:52:04.773120+00:00 [info[] <0.5474.91> connection <0.5474.91> (<REDACTED>:48286 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:52:32.154608+00:00 [warn[] <0.5474.91> closing AMQP connection <0.5474.91> (<REDACTED>:48286 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:52:32.154608+00:00 [warn[] <0.5474.91> client unexpectedly closed TCP connection
2023-09-07 08:52:33.672532+00:00 [info[] <0.5595.91> accepting AMQP connection <0.5595.91> (<REDACTED>:34462 -> <REDACTED>:5672)
2023-09-07 08:52:35.610404+00:00 [info[] <0.5595.91> Connection <0.5595.91> (<REDACTED>:34462 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:52:35.617705+00:00 [info[] <0.5595.91> connection <0.5595.91> (<REDACTED>:34462 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:53:11.079589+00:00 [erro[] <0.5483.91> Channel error on connection <0.5474.91> (<REDACTED>:48286 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:53:11.079589+00:00 [erro[] <0.5483.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:53:12.019771+00:00 [warn[] <0.5595.91> closing AMQP connection <0.5595.91> (<REDACTED>:34462 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:53:12.019771+00:00 [warn[] <0.5595.91> client unexpectedly closed TCP connection
2023-09-07 08:53:40.410653+00:00 [erro[] <0.5611.91> Channel error on connection <0.5595.91> (<REDACTED>:34462 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:53:40.410653+00:00 [erro[] <0.5611.91> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:54:28.926004+00:00 [info[] <0.5879.91> accepting AMQP connection <0.5879.91> (<REDACTED>:47458 -> <REDACTED>:5672)
2023-09-07 08:54:29.457340+00:00 [info[] <0.5879.91> Connection <0.5879.91> (<REDACTED>:47458 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:54:29.469992+00:00 [info[] <0.5879.91> connection <0.5879.91> (<REDACTED>:47458 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'

Node 3:

2023-09-07 08:19:42.240695+00:00 [warn[] <0.27852.18> Mirrored queue '<REDACTED>' in vhost '/': Stopping all nodes on master shutdown since no synchronised mirror (replica) is available
2023-09-07 08:21:03.207677+00:00 [info[] <0.6772.99> accepting AMQP connection <0.6772.99> (<REDACTED>:53686 -> <REDACTED>:5672)
2023-09-07 08:21:03.247079+00:00 [info[] <0.6772.99> Connection <0.6772.99> (<REDACTED>:53686 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:21:03.385081+00:00 [info[] <0.6772.99> connection <0.6772.99> (<REDACTED>:53686 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:21:42.044812+00:00 [warn[] <0.6772.99> closing AMQP connection <0.6772.99> (<REDACTED>:53686 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:21:42.044812+00:00 [warn[] <0.6772.99> client unexpectedly closed TCP connection
2023-09-07 08:22:08.912824+00:00 [erro[] <0.6796.99> Channel error on connection <0.6772.99> (<REDACTED>:53686 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:22:08.912824+00:00 [erro[] <0.6796.99> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:22:44.307871+00:00 [erro[] <0.8915.99> Channel error on connection <0.6772.99> (<REDACTED>:53686 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 4:
2023-09-07 08:22:44.307871+00:00 [erro[] <0.8915.99> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:23:45.867017+00:00 [info[] <0.11876.99> accepting AMQP connection <0.11876.99> (<REDACTED>:59216 -> <REDACTED>:5672)
2023-09-07 08:23:47.780297+00:00 [info[] <0.11876.99> Connection <0.11876.99> (<REDACTED>:59216 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:23:47.796661+00:00 [info[] <0.11876.99> connection <0.11876.99> (<REDACTED>:59216 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:24:22.037805+00:00 [warn[] <0.11876.99> closing AMQP connection <0.11876.99> (<REDACTED>:59216 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:24:22.037805+00:00 [warn[] <0.11876.99> client unexpectedly closed TCP connection
2023-09-07 08:24:51.551842+00:00 [erro[] <0.11891.99> Channel error on connection <0.11876.99> (<REDACTED>:59216 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 1:
2023-09-07 08:24:51.551842+00:00 [erro[] <0.11891.99> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:27:11.804083+00:00 [info[] <0.12233.99> accepting AMQP connection <0.12233.99> (<REDACTED>:42138 -> <REDACTED>:5672)
2023-09-07 08:27:13.249914+00:00 [info[] <0.12233.99> Connection <0.12233.99> (<REDACTED>:42138 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:27:13.273456+00:00 [info[] <0.12233.99> connection <0.12233.99> (<REDACTED>:42138 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:27:42.131443+00:00 [warn[] <0.12233.99> closing AMQP connection <0.12233.99> (<REDACTED>:42138 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:27:42.131443+00:00 [warn[] <0.12233.99> client unexpectedly closed TCP connection
2023-09-07 08:28:16.919880+00:00 [erro[] <0.12248.99> Channel error on connection <0.12233.99> (<REDACTED>:42138 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 2:
2023-09-07 08:28:16.919880+00:00 [erro[] <0.12248.99> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:32:46.821546+00:00 [info[] <0.12785.99> accepting AMQP connection <0.12785.99> (<REDACTED>:45064 -> <REDACTED>:5672)
2023-09-07 08:32:47.782181+00:00 [info[] <0.12785.99> Connection <0.12785.99> (<REDACTED>:45064 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:32:48.793259+00:00 [info[] <0.12785.99> connection <0.12785.99> (<REDACTED>:45064 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:33:11.309442+00:00 [info[] <0.12848.99> accepting AMQP connection <0.12848.99> (<REDACTED>:57802 -> <REDACTED>:5672)
2023-09-07 08:33:11.337742+00:00 [info[] <0.12848.99> Connection <0.12848.99> (<REDACTED>:57802 -> <REDACTED>:5672) has a client-provided name: <REDACTED>
2023-09-07 08:33:11.408447+00:00 [info[] <0.12848.99> connection <0.12848.99> (<REDACTED>:57802 -> <REDACTED>:5672 - <REDACTED>): user '<REDACTED>' authenticated and granted access to vhost '/'
2023-09-07 08:33:22.070376+00:00 [warn[] <0.12785.99> closing AMQP connection <0.12785.99> (<REDACTED>:45064 -> <REDACTED>:5672 - <REDACTED>, vhost: '/', user: '<REDACTED>'):
2023-09-07 08:33:22.070376+00:00 [warn[] <0.12785.99> client unexpectedly closed TCP connection
2023-09-07 08:33:52.788210+00:00 [erro[] <0.12793.99> Channel error on connection <0.12785.99> (<REDACTED>:45064 -> <REDACTED>:5672, vhost: '/', user: '<REDACTED>'), channel 2:
2023-09-07 08:33:52.788210+00:00 [erro[] <0.12793.99> operation queue.declare caused a channel exception not_found: failed to perform operation on queue '<REDACTED>' in vhost '/' due to timeout
2023-09-07 08:54:09.024129+00:00 [warn[] <0.28887.99> Queue <REDACTED> in vhost / has its master node is down and no mirrors available or eligible for promotion. Forcing queue deletion.

[michaelklishin]

RabbitMQ 3.9 has reached end of life. Classic mirrored queues have been deprecated for a couple of years now, scheduled to be removed in 4.0, which is expected to come out in Q1-Q2 next year. You must switch to quorum queues and/or streams (and upgrade from 3.9).

Queue in vhost / has its master node is down and no mirrors available or eligible for promotion. Forcing queue deletion.

is the line you are looking for. This scenario has a dedicated documentation section.