Peer Discovery with DNS record wont cluster

[womblep]

Describe the bug

I am upgrading from 3.11 to 3.13 and my working peer discovery wont work in the new version.
It discovers the A records but wont do anything with them.

| 1716362235833 | {"time":"2024-05-22 07:17:15.833049+00:00","level":"info","msg":"Addresses discovered via A records of peernodes.mq-nonpci.apse2.lab.securecall.local: 10.10.17.226, 10.10.23.236","pid":"<0.253.0>","domain":"rabbitmq"}
| 1716362235834 | {"time":"2024-05-22 07:17:15.834162+00:00","level":"info","msg":"Addresses discovered via AAAA records of peernodes.mq-nonpci.apse2.lab.securecall.local: ","pid":"<0.253.0>","domain":"rabbitmq"}
| 1716362237520 | =PROGRESS REPORT==== 22-May-2024::07:17:17.511731 ===
| 1716362237520 | supervisor: {local,inet_gethost_native_sup}
| 1716362237520 | started: [{pid,<0.97.0>},{mfa,{inet_gethost_native,init,[[]]}}]
| 1716362237521 | =PROGRESS REPORT==== 22-May-2024::07:17:17.520454 ===
| 1716362237521 | supervisor: {local,kernel_safe_sup}
| 1716362237521 | started: [{pid,<0.96.0>},
| 1716362237521 | {id,inet_gethost_native_sup},
| 1716362237521 | {mfargs,{inet_gethost_native,start_link,[]}},
| 1716362237521 | {restart_type,temporary},
| 1716362237521 | {significant,false},
| 1716362237521 | {shutdown,1000},
| 1716362237521 | {child_type,worker}]
| 1716362237609 | {"time":"2024-05-22 07:17:17.608862+00:00","level":"error","msg":"Peer discovery: could not discover and join another node; proceeding as a standalone node","line":245,"pid":"<0.253.0>","file":"rabbit_peer_discovery.erl","domain":"rabbitmq.peer_discovery","mfa":["rabbit_peer_discovery","retry_sync_desired_cluster",3]}

Reproduction steps

cluster_formation.peer_discovery_backend = dns
cluster_formation.dns.hostname =
cluster_formation.discovery_retry_interval = 500
cluster_formation.discovery_retry_limit = 10
cluster_partition_handling = autoheal

Expected behavior

The IP addresses found would be used as peers

Additional context

This was working in 3.11
I restarted the whole cluster thinking it was an upgrade issue but they still wont find each other

[womblep]

I have turned up debugging and I think it is to do with my load balancer health check not bringing the node up (and putting it into DNS) until it can ping the traffic port.
The new check of "is the node's IP address in the DNS list?" is tripping me up.

[womblep]

Yep, AWS Network Load Balancer is expecting some port available to send TCP requests to for the health check before it will put the IP address in the DNS entry. The traffic port is not available until the clustering has happened so I am stuck in a catch-22.

Is there a port which RabbitMQ has open when it starts? I tried 4369 but that doesnt seem to work either.

Otherwise we need a config item to bypass this check:

    ThisNode = node(),
    ThisNodeIsIncluded = lists:member(ThisNode, Nodes),
    case ThisNodeIsIncluded of
        true ->
            ok;
        false ->
            ?LOG_DEBUG(
               "Peer discovery: not satisfyied with discovered peers: the "
               "list does not contain this node",
               #{domain => ?RMQLOG_DOMAIN_PEER_DISC})
    end,

which is a good check but in this instance is not required or desired

[lukebakken]

Is there a port which RabbitMQ has open when it starts?

Port 25672, which is used for Erlang distribution.

[michaelklishin]

As well as other ports but they are opened late during node boot.

The inter-node communication port can be overridden but that's relatively rarely done.

[womblep]

The other ports are not that useful because they get opened after the peer discovery.
The issue is the peer discovery wont work with AWS Network Load Balancer due to the 3.13 change that requires the node's IP to be returned in the DNS A peers. NLB wont add a node to the DNS until it has passed it's health checks. So I cant use those other ports.
I will try 25672 and see if that opens early enough to be useful.

[lukebakken]

NLB wont add a node to the DNS until it has passed it's health checks

I suggest seeing if that behavior can be changed.

[womblep]

Unfortunately, it cant, you cant even turn off the health check.

[womblep]

I was able to get this working by adjusting the healthcheck to port 25672 as answered above as well as adjusting healthcheck timers.
Route53 wouldn't return the node's IP as part of the peer DNS hostname until the container and NLB healthchecks both passed.
It wasnt a DNS caching issue with RabbitMQ or Erlang, just the healthchecks interacting with Route53.

[lukebakken]

Thank you for following up.

[lukebakken]

@womblep just curious, why aren't you using the AWS peer discovery plugin instead? https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-aws

[womblep]

@lukebakken I would love to but I am using containers not EC2. I don't think it would be hard to modify the AWS plugin to handle them as well. The metadata is on a different address but not too dissimilar.
However I know nothing about Erlang so I don't have the skills to do a patch for it.