RabbitMQ federation exchanges are unhealthy

[xkovacsp1]

Describe the bug

Hello, we are using self-hosted rabbitmq clusters in Amazon EKS. Currently, we are trying to create a multi-region deployment for our rabbitmq clusters (see 1. image below ). So far we have had a federated exchange connection between two rabbitmq clusters (between Central and EU region ), and all our federation links were healthy. Now we have added the second region (USA region) and wee see in the federation status admin panel that the federation links are unhealthy in this newly added region (they have an error state, see 2. image below). We have compared the configurations between the old region and the newly added region (USA), but everything is the same, even though the federation links are unhealthy in the USA region. We are using an AWS load balancer to connect to the central rabbitmq cluster. We are using this rabbitmq version: 3.11.16. And the Erlang version is: 25.3.2
I have two questions:

1. in this multiregion setup do we need to set the Max hop variable to 2?
2. is it okay to have the policy and federation upstream names the same for every region (EU, US), or these names must be unique?

Reproduction steps

1. Create a federation upstream which connects to central rabbitmq in the USA region
2. Create a policy for federation exchange
3. Check the federation status in the admin panel
4. Federation links are failing

Expected behavior

All federation links are healthy, they have running state

Additional context

No response

[michaelklishin]

Consider using GitHub Discussions for questions in repositories that have them enabled. They are no longer a new and unknown GitHub feature.

We cannot suggest anything without logs from both sides and a list of policies and upstreams (feel free to edit out credentials and hostnames).

Assuming that Central is a downstream that has two upstreams, US and EU, you don't need to set max-hops because there is no risk of ending up with a message transfer cycle. If US and EU are federated between them, that would introduce a cycle in the connection graph. max-hops affects messages, not connections.

In a given cluster, every upstream must have a unique name. If they don't, you won't be able to add, delete or update both. Then to set up exchange federation from both upstreams, use the special predefined upstream set named "all".

See node logs on both sides for clues as to why the connections do not succeed. We do not guess in this community.

Troublshooting Network Connectivity and Troubleshooting TLS explain the methodology we recommend to narrow the problem down.

If you use default credentials in connection URIs, they will be rejected for non-local clients (including federation links) by default for obvious security reasons.

[xkovacsp1]

Hi, thanks for the prompt response :). In our case, we have 1 upstream (central) and 2 downstream (USA region, EU region), please see the image below. The arrows on this image represent the flows of the messages and also you can see the federation upstream name :)

The federation upstream names are the same for the EU region and USA region please see the image below:

Also, the federation policies are the same for the EU region and the USA region please see the image below:

Also, we have a federation user defined in the central cluster with these configurations:

Below you can see the logs for the central rabbitmq cluster:

srsng-infra-rabbitmq-0_central.log

And there are the logs for the USA region cluster:

srsng-infra-rabbitmq-0_usa_region.log

All sensitive information in these logs I replaced with characters: XXXXX

Thanks for your help again :)

[michaelklishin]

There are two types of exceptions in the US cluster:

One

2023-06-01 08:46:30.617107+00:00 [error] <0.18668.9> Federation link could not open a disposable (one-off) connection, reason: {error,
2023-06-01 08:46:30.617107+00:00 [error] <0.18668.9>                                                                            timeout}

Exchange federation periodically opens new connections to the upstream to check if the source exchange is still there (otherwise it will stop). At least some of that time, that
connection timed out (most likely its underlying TCP connection timed out).

Two

2023-06-01 08:46:30.758798+00:00 [warning] <0.18668.9> Federation exchange 'XXXXXX.XXX' in vhost 'test' did not connect to exchange 'XXXXXX.XXX' in vhost 'test' on amqp://a56feb4602aa74888bf87381fd52f311-a38d6a49fa58f75c.elb.eu-west-1.amazonaws.com:5672. Server has closed the connection due to an error, code: 404, message: NOT_FOUND - no exchange 'federation: XXXXXX.XXX -> rabbit@test-infra-rabbitmq-0.test-infra-rabbitmq-headless.XXX.svc.cluster.local A' in vhost 'test'

This means that the link is configured to use a specific upstream exchange but it does not exist.
So the link would not be able to bind an internally used queue to it and transfer anything.

[xkovacsp1]

Regarding the second exception, I can say that the exchange exists in upstream since in the EU region cluster every federation links are healthy (they do not have an error state and they are not restarting) :)

[michaelklishin]

On an unrelated note, you use non-TLS connections to go over the public Internet. We highly recommend using TLS for inter-cluster communication and client connections (and federation is just an Erlang-based client as far as the upstream is concerned).

[xkovacsp1]

In the EU region we are using the non-TLS connection and every federation link is healthy. I wonder whether this architecture is allowed (see the image below)

Could you please tell me whether it is possible that two downstreams are connecting to the same upstream? :)