Federation status is empty however federation links are still running

[oyviaase]

After migrating our rabbitmq clusters have we experienced a weird issue in one of our clusters. After removing the federation upstreams and the policy for federation from the cluster are we still seeing the federation: * queues and exchanges in the web interface at the same time are there no mention of federation in any of the policies nor are any upstreams visible in the federation status pages. As shown in these images:

I've tried fully disabeling the federation plugin across all the nodes at the same time and that didn't fix the issue, however I did notice that if I restart the federation plugin on only one node a lot of messages like this is spit out into the logs:

12:08:34.988 [warn]  Federation link for queue 'queue \'test\' in vhost \'XXXXXXX-XXX\'' was already started

12:08:35.000 [warn]  Federation link for queue 'queue \'federation%3A%20amq.topic%20-%3E%20XXXXXX%40XXX-XXX-XXXXXXXX-XX-0.XXX-XXX-XXXXXXXX-XX-XXXXXXXXX.XXX.XXX.XXXXXXX.XXXXX\' in vhost \'XXXXXXXX-X-XXXXXXXX\'' was already started

12:08:35.016 [warn]  Federation link for queue 'queue \'federation%3A%20outgoing_XXXXXX-XXXXXXXXXX-XXXXXXXXXXX%20-%3E%20rabbit%40XXX-XXX-XXXXXXXX-XX-0.XXX-XXX-XXXXXXXX-XX-XXXXXXXXX.XXX.XXX.XXXXXXX.XXXXX\' in vhost \'XXXXXXXX-X-XXXXXXXX\' in vhost \'XXXXXX-XXXXXXXXXX-XXXXXXXXXXX\'' was already started
The following plugins have been enabled:
  rabbitmq_federation

Which makes it seem that the federations are still up and running even when it should not be.

We're running version 3.9.13 in case that has anything to do with it. I have not been able to figure out what made this happen and therefore don't have any way of replicating this.

Does anybody have a way to disable the federations? I'm considering restarting the cluster one node at a time to see if that fixes the issue.

[michaelklishin]

We have seen with only some versions affected but not most recent releases. @lukebakken do you remember what they were?

[oyviaase]

Is the best way to fix this to just manually delete the queue's and hope that the issue goes away the next time we restart, upgrade or migrate the cluster?

[michaelklishin]

Disabling and re-enabling the plugin would do.

[oyviaase]

I've already tried that without success:

I've tried fully disabling the federation plugin across all the nodes at the same time and that didn't fix the issue, however I did notice that if I restart the federation plugin on only one node a lot of messages like this is spit out into the logs:

It did provide some interesting things in the logs though which I have already posted in the original post.