Exchange federation does not propagate exchange correctly

[Swierkowski]

Describe the bug

I have 4 instances which connect each other using rabbitmq federation.

All of them have the same version:
RabbitMQ 3.9.11
Erlang 23.3.4.4

When I create the new exchange on 2 of them, using web console, it is correct, i.e. I see the exchange with created name and correct binding on all other instances.

When I create the same exchange on two other instances it is strange, because I do not see the exchange with exactly the same name on other instances. Instead of that I see a new exchange which name is just "?" and which has the created by me binding inside.

What could be the reason on such strange behavior when exchange "?" is created automatically? The fact that it is reproducable only for half of instances makes is weird.

Reproduction steps

1. Configure federation upstreams
2. Configure policy applying to exchanges with federation-upstream-set
3. Create new exchange with name matching policy pattern
4. Exchange with name "?" is created on other servers.

Expected behavior

Instead of exchange named "?" I should see exchange with exactly the same name as in original instance.

Additional context

I provide definitions for both environments as requested by @lukebakken

DEV_definitions.json.txt
ZT4_definitions.json.txt

When I create new exchange on DEV env it is propagated as expected.

When I create new exchange on ZT4 env it is propagated to other environments with "?" name.

[lukebakken]

Right now you're asking us how to guess how to reproduce your issue. Please provide exact steps to reproduce the issue. This means providing the exact values that you have used to set up your vhosts, exchanges and federation.

Even better would be a script we can run to create the same environment. I suggest using the rabbitmqadmin utility.

Or, you can export the definitions from both of your environments and ATTACH the files to your response.

[michaelklishin]

Federation links propagate the federated entity if necessary, and then try to keep bindings in sync on the other end. This is done continuously.

You can create an exchange with the same name on both ends of the federation before setting up federation itself.

Enabling debug logging or taking a traffic capture will help you see what the plugin does.

RabbitMQ 3.9 is out of community support and goes completely EOL in less than a week. Consider upgrading to 3.10.25, then 3.11.20, the. 3.12.x. See release notes and the guide on Upgrades to learn more.

[lukebakken]

@Swierkowski I can see the ? name in your ZT4 definitions file. Unfortunately, this doesn't show how the exchange was created.

I agree with @michaelklishin's comment that you should upgrade first, then try to reproduce again. If you can, you need to provide us an exact sequence of steps that you're taking to create these federated exchanges. You can either...

• Provide step-by-step instructions on what you're doing in the management UI. A video would be acceptable.
• Provide a script that uses rabbitmqadmin to create your exchanges and federation links.

[michaelklishin]

RabbitMQ does not use "?" for entity names. Either it is a formatting output issue (I doubt it) or something else sets that exchange up (possibly mishandling input encoding?)

In any case, the right way to go debugging distributed systems is not to guess but to collect relevant data. Such as these internal events extremely useful for audit.

Also note that for exchange federation, the upstream exchange name can be overridden as part of the upstream definition. And since those are often provided as a single URI, a stray "?" can end up as an exchange name.

[Swierkowski]

Our administrator did a reconfiguration from scratch. It started working as expected.

Thank you for your assistance @lukebakken @michaelklishin