BOOT FAILED issue on rabbitmq cluster using consul

[t0k4rt]

I've got a weird issue with rabbitmq.
I'm trying to setup a 3 servers rabbitmq cluster. Rabbitmq is deployed and launched through docker.

I'm using an hybrid cloud with baremetal and virtual machines provided by openstack.
All the servers can communicate on a private network with a specific vlan.
Consul is correctly setup and all my servers appears alive on the consul member list.
All nodes in the rabbitmq cluster can communicate between them.

When the cluster is launched on vm only, it launches without issues (3 vms with a rabbitmq server each).

But when I try to add one baremetal server in the cluster (2 vms, 1 baremetal server with a rabbitmq server each) I get this error:

BOOT FAILED
===========
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0> BOOT FAILED
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0> ===========
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0> Exception during startup:
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0> error:{badmatch,{error,not_found}}
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery_consul:service_address/4, line 441
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery_consul:service_id/0, line 449
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery_consul:registration_body_add_id/0, line 295
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery_consul:build_registration_body/0, line 285
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery_consul:registration_body/0, line 267
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery_consul:register/0, line 94
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery:register/0, line 163
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>     rabbit_peer_discovery:maybe_register/0, line 126
2022-03-09 13:05:22.439215+00:00 [erro] <0.228.0>

2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>   crasher:
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     initial call: application_master:init/4
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     pid: <0.227.0>
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     registered_name: []
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     exception exit: {{badmatch,{error,not_found}},{rabbit,start,[normal,[]]}}
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>       in function  application_master:init/4 (application_master.erl, line 142)
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     ancestors: [<0.226.0>]
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     message_queue_len: 1
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     messages: [{'EXIT',<0.228.0>,normal}]
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     links: [<0.226.0>,<0.44.0>]
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     dictionary: []
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     trap_exit: true
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     status: running
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     heap_size: 2586
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     stack_size: 29
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>     reductions: 209
2022-03-09 13:05:23.441327+00:00 [erro] <0.227.0>   neighbours:

The error happens only on the baremetal server, it starts properly on the 2 other vms.

I suspect a network issue since I think it's the only big difference between the vms and the baremetal server.

I really don't know why it is happening. Any help or guidance to debug this would be much appreciated.

[lukebakken]

What version of RabbitMQ and Erlang are you using?

[lukebakken]

https://github.com/rabbitmq/rabbitmq-server/blob/v3.9.13/deps/rabbitmq_peer_discovery_consul/src/rabbit_peer_discovery_consul.erl#L449

https://github.com/rabbitmq/rabbitmq-server/blob/v3.9.13/deps/rabbitmq_peer_discovery_consul/src/rabbit_peer_discovery_consul.erl#L441

What's happening at this point is that the value of consul_svc_addr_nic is being used to look up a corresponding IPv4 address and it's not being found.

I'd start by checking your configuration.

[t0k4rt]

Thanks a lot ! It's very helpful.

@lukebakken I've got a crash dump, I want to check the value of consul_svc_addr_nic.
but I cannot find how to open it on osx. I installed erlang, launched erl and typed crashdump_viewer:start/0 but nothings happen ...
Sorry to ask this but do you have any idea how to launch the crashdump viewer on osx ?

[lukebakken]

Please attach your complete, un-edited configuration files for consul and RabbitMQ.

[t0k4rt]

Ok I found the culprit,

cluster_formation.consul.svc_addr_nic is not the good one ...
ens4 is available on the each of my vms but on the dedicated server it's not the same interface name ...
Thanks a lot you made me realize it !

loopback_users.guest = false
listeners.tcp.default = 5672
hipe_compile = false
management.listener.port = 15672
management.listener.ssl = false
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_consul
cluster_formation.consul.svc_addr_use_nodename = false
cluster_formation.consul.svc_addr_nic = enp3s0f1.20
cluster_formation.consul.svc_addr_auto = false
cluster_formation.consul.use_longname = true
cluster_formation.consul.host = consul.service.consul
# 8500 is used by default
cluster_formation.consul.port = 8500
# http is used by default
cluster_formation.consul.scheme = http

[lukebakken]

Great! Thank you for letting us know!