RabbitMQ unable to join cluster using consul peer discovery #10920

ghimisradu · 2024-04-04T15:23:26Z

ghimisradu
Apr 4, 2024

Fresh install of RabbitMQ 3.13.1 Erlang 26.2.3
Configured all the nodes as following(the single difference is the IP address):

default_vhost = /
default_user = guest
default_pass = guest
default_user_tags.administrator= true
default_permissions.configure = .*
default_permissions.read = .*
default_permissions.write = .*
loopback_users.guest= true
log.file.level = debug
management.tcp.ip   = 192.168.169.2
#consul setup
cluster_formation.peer_discovery_backend = consul
cluster_formation.consul.host=192.168.169.2
# 8500 is used by default
cluster_formation.consul.port = 8500
# http is used by default
cluster_formation.consul.scheme = http
# rabbitmq is used by default
cluster_formation.consul.svc = rabbitmq
# do compute service address
cluster_formation.consul.svc_addr_auto = false
# compute service address using the IP address of a NIC, eno2
#cluster_formation.consul.svc_addr_nic = eno2
#cluster_formation.consul.svc_addr_use_nodename = false
cluster_formation.consul.svc_addr=192.168.169.2
# use long RabbitMQ node names?
#cluster_formation.consul.use_longname = false
# 5672 is used by default
cluster_formation.consul.svc_port = 5672
# health check interval (node TTL) in seconds
cluster_formation.consul.svc_ttl = 300
# how soon should nodes that fail their health checks be unregistered by Consul?
# this value is in seconds and must not be lower than 60 (a Consul requirement)
cluster_formation.consul.deregister_after = 90
# include node in the warning state into discovery result set
cluster_formation.consul.include_nodes_with_warnings = true
#some service tags

Environment (rabbitmq-env.conf), again on all nodes the same except the IP address used:

export RABBITMQ_USE_LONGNAME=true
export NODENAME[email protected]
export NODE_IP_ADDRESS=192.168.169.2
export ERL_EPMD_ADDRESS=192.168.169.2

Advanced config(advanced.config):

[
 {kernel,[{inet_dist_use_interface,{  192,168,169,2  }}]}
].

I stopped all three nodes and enabled the consul peer discovery plugin via rabbitmq-plugins --offline enable rabbitmq_peer_discovery_consul then I started first node and copied the .erlang.cookie from it to the other two nodes.

Now when I start the second node, it tries to resolve the peers from consul, but fails. From debug logs I can see Peer discovery: not satisfyied with discovered peers: the list does not contain this node then the node starts in standalone and registers itself in consul.

I really don't know what I'm missing, I attached all the debug logs from start until the first healthcheck it's done.
[email protected]

Below it's one of the attempts logs.

Peer discovery: translated cluster formation configuration: [{peer_discovery_consul,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                               [{include_nodes_with_warnings,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 true},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_host,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 "192.168.169.2"},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_port,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 8500},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_scheme,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 "http"},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_svc,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 "rabbitmq"},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_svc_addr,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 "192.168.169.2"},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_svc_addr_auto,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 false},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_include_nodes_with_warnings,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 true},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_svc_port,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 5672},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_svc_ttl,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 300},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                {consul_deregister_after,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                                 90}]},
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                              {peer_discovery_backend,
2024-04-04 15:15:13.674537+00:00 [debug] <0.250.0>                                                               rabbit_peer_discovery_consul}]
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0> Peer discovery: cluster formation backend configuration: [{include_nodes_with_warnings,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            true},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_host,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            "192.168.169.2"},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_port,8500},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_scheme,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            "http"},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_svc,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            "rabbitmq"},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_svc_addr,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            "192.168.169.2"},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_svc_addr_auto,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            false},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_include_nodes_with_warnings,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            true},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_svc_port,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            5672},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_svc_ttl,300},
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                           {consul_deregister_after,
2024-04-04 15:15:13.675280+00:00 [debug] <0.250.0>                                                            90}]
2024-04-04 15:15:13.676038+00:00 [debug] <0.250.0> GET http://192.168.169.2:8500/v1/health/service/rabbitmq
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0> Response: {ok,{{"HTTP/1.1",200,"OK"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                [{"date","Thu, 04 Apr 2024 15:15:13 GMT"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"vary","Accept-Encoding"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"content-length","3515"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"content-type","application/json"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"access-control-expose-headers",
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                  "x-consul-default-acl-policy"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"x-consul-default-acl-policy","allow"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"x-consul-effective-consistency","leader"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"x-consul-index","43011859"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"x-consul-knownleader","true"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"x-consul-lastcontact","0"},
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                 {"x-consul-query-backend","blocking-query"}],
2024-04-04 15:15:13.677558+00:00 [debug] <0.250.0>                "[{\"Node\":{\"ID\":\"\",\"Node\":\"consul-node-1\",\"Address\":\"192.168.169.1\",\"Datacenter\":\"europe\",\"TaggedAddresses\":{\"lan\":\"192.168.169.1\",\"lan_ipv4\":\"192.168.169.1\",\"wan\":\"192.168.169.1\",\"wan_ipv4\":\"192.168.169.1\"},\"Meta\":{\"consul-network-segment\":\"\",\"consul-version\":\"1.18.1\"},\"CreateIndex\":31994063,\"ModifyIndex\":43011859},\"Service\":{\"ID\":\"rabbitmq:192.168.169.1\",\"Service\":\"rabbitmq\",\"Tags\":[\"traefik.enable=true\",\"traefik.http.routers.rabbitmq.service=rabbitmq\",\"traefik.http.routers.rabbitmq.middlewares=httpsredirect\",\"traefik.http.routers.rabbitmq.rule=Host(`rabbitmq.localhost.com`)\",\"traefik.http.routers.rabbitmq.entryPoints=web\",\"traefik.http.routers.rabbitmq_tls.service=rabbitmq\",\"traefik.http.routers.rabbitmq_tls.rule=Host(`rabbitmq.localhost.com`)\",\"traefik.http.routers.rabbitmq_tls.tls.certResolver=acmetsl\",\"traefik.http.routers.rabbitmq_tls.tls=true\",\"traefik.http.routers.rabbitmq_tls.entryPoints=web-secure\",\"traefik.http.services.rabbitmq.loadbalancer.sticky.cookie.name=serverUUID\",\"traefik.http.services.rabbitmq.loadbalancer.sticky.cookie.secure=true\",\"traefik.http.services.rabbitmq.loadbalancer.sticky.cookie.httponly=true\",\"traefik.http.services.rabbitmq.loadbalancer.sticky=true\",\"traefik.http.middlewares.httpsredirect.redirectScheme.permanent=true\",\"traefik.http.middlewares.httpsredirect.redirectScheme.scheme=https\",\"traefik.http.services.rabbitmq.loadbalancer.server.port=15672\"],\"Address\":\"192.168.169.1\",\"TaggedAddresses\":{\"lan_ipv4\":{\"Address\":\"192.168.169.1\",\"Port\":5672},\"wan_ipv4\":{\"Address\":\"192.168.169.1\",\"Port\":5672}},\"Meta\":null,\"Port\":5672,\"Weights\":{\"Passing\":1,\"Warning\":1},\"EnableTagOverride\":false,\"Proxy\":{\"Mode\":\"\",\"MeshGateway\":{},\"Expose\":{}},\"Connect\":{},\"PeerName\":\"\",\"CreateIndex\":43003685,\"ModifyIndex\":43003685},\"Checks\":[{\"Node\":\"consul-node-1\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"ServiceTags\":[],\"Type\":\"\",\"Interval\":\"\",\"Timeout\":\"\",\"ExposedPort\":0,\"Definition\":{},\"CreateIndex\":31994063,\"ModifyIndex\":31994063},{\"Node\":\"consul-node-1\",\"CheckID\":\"service:rabbitmq:192.168.169.1\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Consul-based peer discovery plugin TTL check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq:192.168.169.1\",\"ServiceName\":\"rabbitmq\",\"ServiceTags\":[\"traefik.enable=true\",\"traefik.http.routers.rabbitmq.service=rabbitmq\",\"traefik.http.routers.rabbitmq.middlewares=httpsredirect\",\"traefik.http.routers.rabbitmq.rule=Host(`rabbitmq.localhost.com`)\",\"traefik.http.routers.rabbitmq.entryPoints=web\",\"traefik.http.routers.rabbitmq_tls.service=rabbitmq\",\"traefik.http.routers.rabbitmq_tls.rule=Host(`rabbitmq.localhost.com`)\",\"traefik.http.routers.rabbitmq_tls.tls.certResolver=acmetsl\",\"traefik.http.routers.rabbitmq_tls.tls=true\",\"traefik.http.routers.rabbitmq_tls.entryPoints=web-secure\",\"traefik.http.services.rabbitmq.loadbalancer.sticky.cookie.name=serverUUID\",\"traefik.http.services.rabbitmq.loadbalancer.sticky.cookie.secure=true\",\"traefik.http.services.rabbitmq.loadbalancer.sticky.cookie.httponly=true\",\"traefik.http.services.rabbitmq.loadbalancer.sticky=true\",\"traefik.http.middlewares.httpsredirect.redirectScheme.permanent=true\",\"traefik.http.middlewares.httpsredirect.redirectScheme.scheme=https\",\"traefik.http.services.rabbitmq.loadbalancer.server.port=15672\"],\"Type\":\"ttl\",\"Interval\":\"\",\"Timeout\":\"\",\"ExposedPort\":0,\"Definition\":{},\"CreateIndex\":43003685,\"ModifyIndex\":43003685}]}]"}}
2024-04-04 15:15:13.679370+00:00 [debug] <0.250.0> Peer discovery: backend returned the following configuration:
2024-04-04 15:15:13.679370+00:00 [debug] <0.250.0>   {ok,{['[email protected]'],disc}}
2024-04-04 15:15:13.679686+00:00 [debug] <0.250.0> Peer discovery: peer node arguments: #{args =>
2024-04-04 15:15:13.679686+00:00 [debug] <0.250.0>                                            ["-boot","start_sasl","-hidden"],
2024-04-04 15:15:13.679686+00:00 [debug] <0.250.0>                                        name => "rabbit-1539-325412",
2024-04-04 15:15:13.679686+00:00 [debug] <0.250.0>                                        host => "192.168.169.2",
2024-04-04 15:15:13.679686+00:00 [debug] <0.250.0>                                        longnames => true}
2024-04-04 15:15:13.816823+00:00 [debug] <0.250.0> Peer discovery: using temporary hidden node '[email protected]' to query discovered peers properties
2024-04-04 15:15:13.927399+00:00 [debug] <35900.100.0> Peer discovery: sorted list of nodes and their properties considered to create/sync the cluster:
2024-04-04 15:15:13.927399+00:00 [debug] <35900.100.0>   - {'[email protected]',['[email protected]'],1712233679806063,true}
2024-04-04 15:15:13.928946+00:00 [debug] <0.250.0> Peer discovery: not satisfyied with discovered peers: the list does not contain this node
2024-04-04 15:15:13.929079+00:00 [error] <0.250.0> Peer discovery: could not discover and join another node; proceeding as a standalone node

Answered by mkuratczyk

Apr 4, 2024

#10760

View full answer

mkuratczyk · 2024-04-04T15:57:59Z

mkuratczyk
Apr 4, 2024
Maintainer

#10760

1 reply

ghimisradu Apr 4, 2024
Author

Thanks, that saved me a lot of time, I'll stick with 3.12.x until consul discovery works again.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RabbitMQ unable to join cluster using consul peer discovery #10920

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

RabbitMQ unable to join cluster using consul peer discovery #10920

Uh oh!

ghimisradu Apr 4, 2024

Replies: 1 comment · 1 reply

Uh oh!

mkuratczyk Apr 4, 2024 Maintainer

Uh oh!

ghimisradu Apr 4, 2024 Author

ghimisradu
Apr 4, 2024

Replies: 1 comment 1 reply

mkuratczyk
Apr 4, 2024
Maintainer

ghimisradu Apr 4, 2024
Author