DNS peer discovery stopped working after upgrade to 3.13

[KonradMikulaDTIQ]

Hello,

After upgrading RabbitMQ to 3.13 DNS peer discovery seems to stop working. I'm trying to set up a new cluster using the same config as in 3.12; during init phase RabbitMQ sees other nodes just fine, but logs says that node Peer discovery: not satisfyied with discovered peers: the list does not contain this node. This is a full log (one of the retry):

Addresses discovered via A records of rabbitmq.xxxx.com: 10.112.4.34
{ok,{['rabbit@ip-10-112-4-34'],disc}}
Peer discovery: backend returned the following configuration:
Peer discovery: using temporary hidden node 'rabbit-850-17@ip-10-112-5-197' to query discovered peers properties
{'rabbit@ip-10-112-4-34',['rabbit@ip-10-112-4-34'],1709543211659002,true}
Peer discovery: sorted list of nodes and their properties considered to create/sync the cluster:
Peer discovery: retrying to create/sync cluster in 1000 ms (7 attempts left)
Peer discovery: not satisfyied with discovered peers: the list does not contain this node

[mkuratczyk]

Please share full logs from all nodes

[dumbbell]

The DNS A record should resolve to IP addresses of all nodes that are part of the cluster.

In your situation, it looks like the DNS A record resolves to a single IP address. I'm not sure if this corresponds to a (still) valid node, based on your "from old deployment" comment, but it should resolve to the other IP addresses too.

[KonradMikulaDTIQ]

@mkuratczyk Sorry for the confusion. I'll try to explain on a new deployment:

The approach I'm taking right now is to start nodes one by one - 1st node will obviously be standalone node because no additional nodes has not started yet; From logs you can see that node 2 can reach node 1 but will fail to form cluster with it. The same goes when I will add node 3; it will discover node 1 and 2 but fail to form cluster with them.

Node 1 (10.112.4.87):
node1.log

Node 2 (10.112.3.123):
node2.log

Node 3 (10.112.5.42):
node3.log

Important thing: When I replicate this steps for a new deployment but based on rabbit 3.12 it works as expected - node 2 will form a cluster with node 1 and later, node 3 will join this cluster.

Example on RabbitMQ 3.12 (the same config, the same infrastructure, only version changed):

Node 1 (10.112.5.37):
3-12-node1.log

Node 2 (10.112.3.5):
3-12-node2.log

Node 3 (10.112.4.78):
3-12-node3.log

Result on 3.12: Formed cluster of 3 nodes, as expected:

@dumbbell
As you can see your answer is not a solution since the same setup works just fine on RabbitMQ version 3.12.

@michaelklishin
This is not an issue of DNS installation; as proved, 3.12 works just fine with the same setup, so in 3.13 some breaking change might have been introduced which results in issue with forming clusters using DNS peer discovery.

[dumbbell]

How are the A records populated? It looks like they are dynamically added because the DNS query returns different results over time.

[KonradMikulaDTIQ]

Hi, I found a solution, a rather trivial one... It looks like startup in RabbitMQ 3.13 takes a little bit longer for node to become ready to be clustered. I just added

cluster_formation.discovery_retry_limit = 50
cluster_formation.discovery_retry_interval = 10000

and after some time it started clustering properly. Thanks for the help!

[dumbbell]

Let me expand my answer a bit. The peer discovery logic changed significantly in 3.13 (see #9797). In particular, it now requires that the list of nodes returned by the peer discovery backend contains the node that queried it. That’s why I’m asking how the discovered nodes list grows.

[dumbbell]

@KonradMikulaDTIQ: Could you please share the logs from the deployment that worked? I would like to see what differs in the behavior.

Also, could you please describe how the A records are added in your case? There is another user who sees a regression with the same configuration (though they are using Consul).

[KonradMikulaDTIQ]

@dumbbell That particular PR pushed me to change these settings exactly. I'm running a cluster using AWS ECS Fargate along with CloudMap Service Discovery - This handles adding A records.

This is a log of successful join to a cluster from node 3 (although in INFO level, I disabled debug already but if you need it in DEBUG let me know):
node3-new.log

[KonradMikulaDTIQ]

I think the issue might be in "the list of nodes returned by the peer discovery backend contains the node that queried it" - from my logs you can see that node who was trying to join/form a cluster is not appearing on this list in a default amount of time of cluster formation - when I expanded it to the values I pasted above, it eventually showed up on this list and cluster formation was successful. Maybe there is some background init happening which takes longer time now than in 3.12?

[michaelklishin]

Our team does not have the cycles to troubleshoot DNS installations of non-paying customers. You were provided evidence of a discrepancy between your node's IP addresses and what the A/AAAA records return.

Please take it from here.