RabbitMQ 3.9.22 / Erlang 24.3.4.5 - Failed to publish messages after outage

[bbuniazet]

Hello,

We're running a RabbitMQ cluster with 9 nodes version RabbitMQ 3.9.22 and using a third party application to create/consume channel/queues/consumers, publish messages, binding/unbinding routing keys.
Recently we've had some outage on some nodes that resulted with some queues not being mirrored correctly on to the second mirror
(ha-params:2) - see picture attached.
When this happens, RabbitMQ is unable to let the third party application publish any messages with a routing key (a) that should be delivered to the queue "broken queue" matching the routing key (a).

Only way to recover is either to delete the key (usually doesn't work through the management UI - works only using this command sudo docker exec -it rabbit rabbitmqctl eval "{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<\"/\">>, queue, <<\"rabbitQUEUE\">>)), rabbit_amqqueue:delete_crashed(Q)." and/or remove the routing key.

Is this an expected behaviour ?
Is this fixed in a newer version ?
Should we have the third party managing automatically the cleanup of broken queues ?

I'm sorry if I don't use all the proper terminology, I'm not a developer.
Appreciate all the help I can have,

Baptiste

[lukebakken]

Thanks for using RabbitMQ.

Unfortunately, we do not have the resources to answer questions or do troubleshooting for unsupported versions of RabbitMQ. Please upgrade, and please start using quorum queues.