Fanout exchange does not route messages to bound *running* quorum queues when there are other queues without a leader

[devkits]

Hi Everyone,
Wanted to confirm the following behaviour as a bug and best practices to troubleshoot/RCA or gracefully recover from the following issue/bug(?). Let me know if the format of this message is not to community's satisfaction and I will do better next time.

Environment: RMQ 3.9.13 on EKS in AWS on a large 7-node cluster

Issue: Fanout exchange does not route messages to any of the bound queues if there's at least one bound queue that is not in the "running" state. Queues that are not "running" have a number of errors like this one in the logs and their status does not change:

...
queue 'salt_minion_command_queue_ip-10-0-98-95_ip-10-0-98-95-ms-08' in vhost '/': leader call - leader not known. Command will be forwarded once leader is known.
...

Thanks.

[michaelklishin]

Individual quorum queues are separate Raft clusters. Without a way to reproduce, it is hard to believe that this kind of interference is possible.

[devkits]

I have a live repro in a public cloud if you'd like to see.

[devkits]

Just want to update this thread in case someone may find this interesting.

When some quorum queues are in this state, the above issue is also reproducible when attempting to send a message to the exchange from RMQ Management UI. @kjnilsson provided this comment:

"""...if it is the first message the channel is publishing to these queues it may well not reach any of them as it first needs to set up a session with the queue and if it is unavailable this will take a lot of time to time out"""

[taju20910]

Hi @devkits
I am also facing the same issue. Did you get any solution from the last time.

Thanks.

[lukebakken]

@devkits @taju20910 - we don't know how queues get into this state in the first place so reproducing this issue and fixing it is not really possible.

[taju20910]

@lukebakken FYI ^^

[lukebakken]

Yes, I saw that you attached log files. I am emailed for every response to a question or issue in github.com/rabbitmq. As you can imagine, that is a LOT. I'll try to find some time to follow up here but really, like I said, without a way to reproduce this issue reliably we can't spend hours of time getting to the bottom of it.

cc @kjnilsson

[taju20910]

@lukebakken sure thanks. appreciate your efforts.
We are also trying on our end to reproduce. Will share the steps in case we are able to.

[lukebakken]

Thank you. I am away from work until April 10th, just FYI.

[taju20910]

@lukebakken Okay.