Join an external cluster from POD always attempt to connect rabbitmq-headless.rabbitmq.svc.cluster.local

[vjwilson1987]

Describe the bug

Basically, I installed the RabbitMQ on K8s using the bitnami/rabbitmq helm chart and reported this issue already here but they said it would be better to report here regardless of the helm chart but might be a configuration behavior of RabbitMQ on K8s.

The RabbitMQ app and Bitnami chart version I use:

App version: 3.11.9
Chart version: 11.10.0

After deploying it on the K8s, I tried to have it join an external RabbitMQ cluster running on EC2 instances with HA mode enabled. The purpose of doing it is that I am preparing a migration of RabbitMQ from EC2 to Kubernetes given here

The RabbitMQ node name on the source ( EC2 instance ) is rabbit@rabbitmqec2 ( using a short node name by default ) and its public IP is, let's say 1.2.3.4

Command running on the POD to join the external RabbitMQ EC2 node:

I have no name!@rabbitmq-0:/$ rabbitmqctl stop_app && rabbitmqctl join_cluster rabbit@rabbitmqec2 && rabbitmqctl start_app

But it fails and shows the following error always:

attempted to contact: ['[email protected]']

[email protected]:
  * unable to connect to epmd (port 4369) on rabbitmqec2.rabbitmq-headless.rabbitmq.svc.cluster.local: nxdomain

To make the IP 1.2.3.4 known to the RabbitMQ on K8s to connect to rabbit@rabbitmqec2 since the short node name rabbitmqec2 does not resolve globally, I tried the following methods on K8s:

1. Pod hostAliases /etc/hosts

Using the hostAliases option by adding the following in the helm chart so that the POD /etc/hosts will have an entry

hostAliases:
- ip: "1.2.3.4"
  hostnames:
  - "rabbitmqec2"

2. Externalname

I tried creating an external service name ( within the same namespace rabbitmq ) with the short node name pointing to the DNS resolving long node name, say rabbitmqec2.example.com ( where the long node name points to the EC2 IP 1.2.3.4 )

apiVersion: v1
kind: Service
metadata:
  name: rabbitmqec2
  namespace: rabbitmq
spec:
  type: ExternalName
  externalName: rabbitmqec2.example.com

3. Coredns entry

I tried the following entry in CoreDNS confirmap

        hosts custom.hosts rabbitmqec2 {
            1.2.3.4 rabbitmqec2
            fallthrough
        }

Even though I tried all the above methods, the command rabbitmqctl join_cluster rabbit@rabbitmqec2 still attempts to connect to [email protected] which I don't understand why.

Reproduction steps

1. Deploy RabbitMQ on K8s using the bitnami/rabbitmq helm chart.

2. Try to join an external RMQ cluster with a short node name ( not a DNS resolving or not a FQDN node name ) and you will get this error even if you try the above 3 methods.

Expected behavior

The following command should work successfully on the POD and it should connect to the source RMQ node using any of the 3 methods.

I have no name!@rabbitmq-0:/$ rabbitmqctl stop_app && rabbitmqctl join_cluster rabbit@rabbitmqec2 && rabbitmqctl start_app

Additional context

No response

[michaelklishin]

Joining nodes from two separate clusters is a curious migration strategy that the core team never recommends anywhere.

Blue/Green deployment is what should be done instead.

Our team does not maintain any charts but I assume it uses Kubernetes peer discovery. While you
can form a cluster using peer discovery and manually add nodes, this is very rarely done.

rabbitmqctl join_cluster [existing cluster member]

by default will connect to a local node. Use --node if you want to tell a different node to connect to target cluster member. Before you do so, the node must be reset, as in, ALL of its data
must be wiped out so that it can sync from [existing cluster member]. Which is why this cluster migration strategy is unlikely to be of much use.

The command will resolve the hostname of the target (--node) and attempt to contact it. If
successful, that node will resolve the hostname of [existing cluster member] and attempt to contact it.

[michaelklishin]

To reiterate:

• CLI tools and nodes resolve hostnames the way one would expect (unless an erl_inetrc file is used, which is very unlikely on Kubernetes)
• rabbitmqctl join_cluster accepts an existing cluster member but also a --node that allows it to tell a non-local node to join that existing member
• Before node in cluster A can join cluster B it MUST be reset, with all of its data removed. This is not particularly useful for cluster migrations
• Blue/Green deployment is how migration from cluster A to cluster B should be done. This strategy has been around for a while, has the lowest risk (the original cluster is still there if/when you need it) and only requires enough connectivity between clusters for Federation links to work

[vjwilson1987]

Thanks, @michaelklishin

Regarding the Blue/Green deployment, it says the following

First define the upstream on "green" and point it to "blue":

rabbitmqctl set_parameter federation-upstream blue \
  '{"uri":"amqp://node-in-blue-cluster"}'

If the source RMQ node name is rabbit@rabbitmqec2 where rabbitmqec2 is short name and non resolvable from the new cluster, how does it connect?

Or can we simply use the source cluster node IP like amqp://1.2.3.4:5672?

[michaelklishin]

Federation connects the same way clients do. It can use IP addresses or hostnames that do resolve.

[vjwilson1987]

I will look into that and see if everything works fine. Thanks, @michaelklishin

[vjwilson1987]

@michaelklishin the Blue/Green deployment method works great. I am marking this discussion as solved.