Unable to delete dead binding in dynamic queue

[FalconerTC]

Using Rabbitmq 3.9.13
Deployed via Helm in GKE

We have exchanges with dynamic queues (autoDelete) set to true and we're seeing scenarios where bindings get stuck after the queue is deleted. It seems to happen after a rabbitmq node shutdown and comes back up.

Here's an exchange that throws a 404 when attempting to delete a binding

From the CLI I can confirm that the queue doesn't exist but the binding does, and I am unable to remove the binding

root@fcarter-tools:/# ./rabbitmqadmin list queues | grep amq.gen-0Kp32y_22H1zbaQ4-KBlBQ
root@fcarter-tools:/# ./rabbitmqadmin list bindings | grep amq.gen-0Kp32y_22H1zbaQ4-KBlBQ
| allogator-logical-rack-count              | amq.gen-0Kp32y_22H1zbaQ4-KBlBQ                                        | allogator-logical-rack-count                                                    |
root@fcarter-tools:/# ./rabbitmqadmin delete binding destination=amq.gen-0Kp32y_22H1zbaQ4-KBlBQ source=allogator-logical-rack-count destination_type=queue
*** Not found: /api/bindings/%2F/e/allogator-logical-rack-count/q/amq.gen-0Kp32y_22H1zbaQ4-KBlBQ/%7E

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

The 404 very likely means that the queue is no longer there, and as such, no bindings on it can be found.

Your queue is not just auto-delete, it is server-named. You cannot assume that the name of that queue is stable, e.g. reconnecting clients will generate a new name every time to avoid a race condition with anything RabbitMQ nodes do. This is a fundamental race condition
on auto-delete, server-named queues and client activity, there is no solution or workaround besides: do not use auto-delete queues
if you assume that anything about them should be stable/"recoverable". Use a queue with a well-known queue name, and maybe
queue TTL over auto-deletion.

[michaelklishin]

I see the rabbitmqadmin lines. So what would be a set of steps we can use to reprodue?

[FalconerTC]

To clarify some more, we publish only to the exchange so the queue being server-named is fine. The problem is that once we get into this state we find that some messages seem to be routed using the broken bindings to non-existent queues so we are unable to consume them.

I will try and isolate this and find a consistent way to recreate it.