Message not routed to all bound queues

[narrowtux]

At some point during running, rabbitmq seems to forget about most bindings using a particular routing key.

rabbitmq-diagnostics list_bindings | grep packet_added.device.\\* lists these 2 queues:

z_global_events exchange        z_element_find_rule_broadway            queue   packet_added.device.*   []
z_global_events exchange        z_element_packet_histogram_broadway     queue   packet_added.device.*   []

however, when asking rabbit_exchange:route/2 where it would route a message with the routing key packet_added.device.abc, it only returns one queue:

rabbit_exchange:route(LExchange, Delivery).
[{resource,<<"/">>,queue,
           <<"z_element_packet_histogram_broadway">>}]

Is there a reason why such a binding could be lost during runtime? The only thing that works to re-enable the binding is to delete it and re-create it (i.e. via the management UI). I've done it for this queue last friday, which fixed it temporarily, but today we noticed that messages were again not routed.

In case it's interesting, here's the code I jotted down in rabbitmq-diagnostics remote_console:

LExchange = rabbit_exchange:lookup(rabbit_misc:r(<<"/">>, exchange, <<"z_global_events">>)).
PBasic2 = {'P_basic', <<"txt/plain">>, <<"utf8">>, [], 2, 0, 0, none, 0, 3, 168872738, message, 1, 1, 1}.
Content5 = {content, 7, PBasic2, none, none, [<<"hello">>]}.
{ok, Message2} = rabbit_basic:message(<<"z_global_events">>, <<"packet_added.device.auisdhasuiuhdsa">>, Content5).
Delivery = {delivery,  true,    true,      self(),       Message2,      1,   true}.
rabbit_exchange:route(LExchange, Delivery).

additional context information

RabbitMQ Version: 3.11.2
Erlang/OTP: 25

[narrowtux]

Interesting: as soon as I "fix" that problem in one queue, the same binding will stop working in the other:

rabbit_exchange_type_topic:route(LExchange, Delivery).
[{resource,<<"/">>,queue,
           <<"z_element_find_rule_broadway">>}]

[mkuratczyk]

Are there any transient (non-durable) entities involved (queues/exchanges)?

[lukebakken]

How are the queues, exchanges and bindings normally created (and deleted, if they ever are)? By an application, by importing definitions, manually, etc?

[narrowtux]

Initially everything is created by the application, but should a new rabbit node join the cluster, it will copy the definitions from one of the other nodes. This happens every 1-2 weeks when upgrading rabbitmq or due to server issues 😬

[lukebakken]

Does anything DELETE exchanges, queues or binding?

[narrowtux]

All of the transient queues are auto-deleted or have timeouts. the 2 queues where the bug is happening are not deleted, neither are their bindings to this exchange.

In the past, the setup was more involved, where there were lots of exchanges and bindings mapping to a folder structure in the application that was kept up to date using deletes of bindings and exchanges, but we don't do that any more. I remember we experienced similar bugs back then.

[narrowtux]

There is still a feature that creates and deletes exchanges and bindings though. These exchanges all bind to that global_events_exchange using different routing keys to filter the messages.