Classic peer discovery - is computer name required?

[lukebakken]

cc @michaelklishin @dumbbell

• RabbitMQ 3.13.0
• Erlang 26.2.3

I am testing classic peer discovery on Windows. Here is the relevant configuration:

cluster_formation.peer_discovery_backend = classic_config
cluster_formation.classic_config.nodes.0 = rmq0
cluster_formation.classic_config.nodes.1 = rmq1
cluster_formation.classic_config.nodes.2 = rmq2

I can start up three nodes, all on localhost, but peer discovery fails when using RabbitMQ 3.13.0. It succeeds using 3.12.13. The following is from the prelaunch debug log of a 3.13.0 node:

2024-03-18 12:24:30.393000-07:00 [debug] <0.248.0> Peer discovery: backend returned the following configuration:
2024-03-18 12:24:30.393000-07:00 [debug] <0.248.0>   {ok,{[rmq2@PROKOFIEV,rmq2,rmq1,rmq0],disc}}
2024-03-18 12:24:30.393000-07:00 [debug] <0.248.0> Peer discovery: peer node arguments: #{args =>
2024-03-18 12:24:30.393000-07:00 [debug] <0.248.0>                                            ["-boot","start_sasl","-hidden"],
2024-03-18 12:24:30.393000-07:00 [debug] <0.248.0>                                        name => "rmq2-1091-11676"}
2024-03-18 12:24:30.566000-07:00 [debug] <0.248.0> Peer discovery: using temporary hidden node 'rmq2-1091-11676@PROKOFIEV' to query discovered peers properties
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: failed to query cluster members of node 'rmq2': {error,{erpc,noconnection}}
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: node 'rmq2' excluded from the discovered nodes
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: failed to query cluster members of node 'rmq1': {error,{erpc,noconnection}}
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: node 'rmq1' excluded from the discovered nodes
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: failed to query cluster members of node 'rmq0': {error,{erpc,noconnection}}
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: node 'rmq0' excluded from the discovered nodes
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0> Peer discovery: sorted list of nodes and their properties considered to create/sync the cluster:
2024-03-18 12:24:30.566000-07:00 [debug] <33244.99.0>   - {rmq2@PROKOFIEV,[rmq2@PROKOFIEV],1710789850112000,true}
2024-03-18 12:24:30.566000-07:00 [debug] <0.248.0> Peer discovery: not satisfyied with discovered peers: the list should contain at least two nodes with a configured cluster size hint of 4 nodes
2024-03-18 12:24:30.566000-07:00 [debug] <0.248.0> Peer discovery: retrying to create/sync cluster in 1000 ms (18 attempts left)
2024-03-18 12:24:31.586000-07:00 [debug] <0.248.0> Peer discovery: backend returned the following configuration:
2024-03-18 12:24:31.586000-07:00 [debug] <0.248.0>   {ok,{[rmq2@PROKOFIEV,rmq2,rmq1,rmq0],disc}}
2024-03-18 12:24:31.586000-07:00 [debug] <0.248.0> Peer discovery: peer node arguments: #{args =>
2024-03-18 12:24:31.586000-07:00 [debug] <0.248.0>                                            ["-boot","start_sasl","-hidden"],
2024-03-18 12:24:31.586000-07:00 [debug] <0.248.0>                                        name => "rmq2-1223-11676"}
2024-03-18 12:24:31.743000-07:00 [debug] <0.248.0> Peer discovery: using temporary hidden node 'rmq2-1223-11676@PROKOFIEV' to query discovered peers properties
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: failed to query cluster members of node 'rmq2': {error,{erpc,noconnection}}
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: node 'rmq2' excluded from the discovered nodes
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: failed to query cluster members of node 'rmq1': {error,{erpc,noconnection}}
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: node 'rmq1' excluded from the discovered nodes
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: failed to query cluster members of node 'rmq0': {error,{erpc,noconnection}}
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: node 'rmq0' excluded from the discovered nodes
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0> Peer discovery: sorted list of nodes and their properties considered to create/sync the cluster:
2024-03-18 12:24:31.759000-07:00 [debug] <33245.99.0>   - {rmq2@PROKOFIEV,[rmq2@PROKOFIEV],1710789850112000,true}
2024-03-18 12:24:31.759000-07:00 [debug] <0.248.0> Peer discovery: not satisfyied with discovered peers: the list should contain at least two nodes with a configured cluster size hint of 4 nodes

If I change the configuration to include the hostname, peer discovery succeeds:

cluster_formation.peer_discovery_backend = classic_config
cluster_formation.classic_config.nodes.0 = rmq0@PROKOFIEV
cluster_formation.classic_config.nodes.1 = rmq1@PROKOFIEV
cluster_formation.classic_config.nodes.2 = rmq2@PROKOFIEV

The only reason I bring this up is that most other parts of RabbitMQ allow leaving off the @HOSTNAME part of a node name (the -n CLI argument, for instance). I don't think that it is the end of the world in this case, but perhaps it should be documented?

[lukebakken]

This appears to be relevant prelaunch debug logging from the startup of a 3.12.13 node:

2024-03-18 12:53:01.620000-07:00 [debug] <0.135.0>     - cluster_partition_handling = pause_minority
2024-03-18 12:53:01.620000-07:00 [debug] <0.135.0>     - cluster_formation = [{peer_discovery_backend,
2024-03-18 12:53:01.620000-07:00 [debug] <0.135.0>                                rabbit_peer_discovery_classic_config}]
2024-03-18 12:53:01.620000-07:00 [debug] <0.135.0>     - cluster_nodes = {[rmq2@PROKOFIEV,rmq1@PROKOFIEV,rmq0@PROKOFIEV],disc}
...
...
...
2024-03-18 12:53:14.853000-07:00 [debug] <0.235.0> rabbit_peer_discovery:lock returned {ok,
2024-03-18 12:53:14.853000-07:00 [debug] <0.235.0>                                      {{"WNW/o1HF7ZYSWb5klAO5rg==",
2024-03-18 12:53:14.853000-07:00 [debug] <0.235.0>                                        rmq0@PROKOFIEV},
2024-03-18 12:53:14.853000-07:00 [debug] <0.235.0>                                       [rmq2@PROKOFIEV,rmq1@PROKOFIEV,
2024-03-18 12:53:14.853000-07:00 [debug] <0.235.0>                                        rmq0@PROKOFIEV]}}
2024-03-18 12:53:14.853000-07:00 [info] <0.235.0> All discovered existing cluster peers: rmq0@PROKOFIEV, rmq1@PROKOFIEV, rmq2@PROKOFIEV
2024-03-18 12:53:14.853000-07:00 [info] <0.235.0> Peer nodes we can cluster with: rmq1@PROKOFIEV, rmq2@PROKOFIEV

[dumbbell]

Are you sure you didn't keep the second configuration file (the one with @PROKIFIEV specified) for the test with 3.12.13?

The following log line suggests me that the configuration already had the hostname:

2024-03-18 12:53:01.620000-07:00 [debug] <0.135.0>     - cluster_nodes = {[rmq2@PROKOFIEV,rmq1@PROKOFIEV,rmq0@PROKOFIEV],disc}

[dumbbell]

I can't reproduce the problem: with the v3.12.x branch, I get the same behavior as the main branch:

2024-03-19 12:22:27.708645+01:00 [info] <0.239.0> All discovered existing cluster peers: rabbit-1, rabbit-2, rabbit-3                                                                                                                          
2024-03-19 12:22:27.708690+01:00 [info] <0.239.0> Peer nodes we can cluster with: rabbit-1, rabbit-2, rabbit-3                                                                                                                                 
2024-03-19 12:22:27.708748+01:00 [warning] <0.239.0> Could not auto-cluster with node rabbit-1: {badrpc,nodedown}                                                                                                                              
2024-03-19 12:22:27.708784+01:00 [warning] <0.239.0> Could not auto-cluster with node rabbit-2: {badrpc,nodedown}                                                                                                                              
2024-03-19 12:22:27.708824+01:00 [warning] <0.239.0> Could not auto-cluster with node rabbit-3: {badrpc,nodedown}

The configuration is:

cluster_formation.peer_discovery_backend = classic_config
cluster_formation.classic_config.nodes.0 = rabbit-1
cluster_formation.classic_config.nodes.1 = rabbit-2
cluster_formation.classic_config.nodes.2 = rabbit-3

[lukebakken]

You're correct, it fails in 3.12.13. I must have used the wrong config files like you noted.

Sorry about the wild goose chase!

[dumbbell]

Thank you for the test and report!

Indeed, this scenario shouldn't break with the internal change of rabbit_peer_discovery. I'm looking for the place where the nodenames are made uniform and will get back to you.

[dumbbell]

A side comment about this configuration: when I see it without looking at any documentation, I don't know if I should interpret it as rmq1@$HOSTNAME or rabbit@rmq1.

If we want to allow a user to use partial nodenames to make it flexible (and I think this is a good thing), I'm wondering if the syntax should be something like @rmq1 and rmq1@ so that RabbitMQ can determine which part is specified and which one should be "guessed".

That said, as you said, the CLI's -n rmq1 is the same as -n rmq1@$HOSTNAME, so perhaps a user of RabbitMQ is not as confused as I am.

[lukebakken]

It should be interpreted as rmq1@HOSTNAME.

[dumbbell]

Can I convert this discussion to an issue/feature request?

[lukebakken]

Sure, if you think it's useful. I'm pretty sure I'm the only person on the planet that has noticed this behavior.