Undeletable queues after version upgrade

[narrowtux]

After upgrading from 3.11.1 to 3.11.2, all or nearly all of the queues with the auto-delete: true param kind of broke (unsynchronized, not responding to sync request). Immediately after we noticed this, we were able to delete 2 of those queues, which our application redeclared and these new ones were fine. However, we didn't delete all of those queues just then and now we're unable to drop them.

note the +1 telling us there are unsynchronized mirrors

note that we're not seeing any queued messages/message rate graphs

When clicking "Delete" in the management interface, it takes about 30 to 45 seconds until the request bounces back with a 504 error:

These queues have bindings to a topic exchange, should a message in that exchange be routed to one of the broken queues, a publisher confirm will never be sent, further impacting our application, because at some point it will stop publishing messages to that exchange because of back-pressure mechanisms.

We were able to resolve the last issue by simply dropping the bindings, because those are luckily not managed by the queue process.

[narrowtux]

I've connected to the shell using rabbitmq-diagnostics remote_shell and tried to find out what's happening with the queue process:

erl> R2 = rabbit_misc:r(<<"/">>, queue, <<"ws_api_34b32502-148c-4f2b-a4fe-31c74ee23994">>).
erl> {ok, Q} = rabbit_amqqueue:lookup(R2).
erl> Pid = amqqueue:get_pid(Q).
<0.7318.493>
erl> erlang:is_process_alive(Pid).
false

seems like it died at some point.

I was able to delete them all with amqqueue:delete_crashed.

[lukebakken]

Thank you for following up, and well done to figure out the solution.