rabbitmq_peer_discovery_aws - node unable to join cluster

[limjf85]

Describe the bug

I am trying to set up a cluster of 2 nodes using rabbitmq_peer_discovery_aws plugin following the documentation.
I have been trying the entire day without success.

I have verified the erlang cookies match, that the hosts are able to talk to each other (ping, telnet, AWS Reachability analyzer), the nodes can talk to each other, the config file is really simple and straight forward however I keep getting the following error

... [info] <0.236.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@ip-10-102-9-34 is empty. Assuming we need to join an existingcluster or initialise from scratch...
... [info] <0.236.0> Configured peer discovery backend: rabbit_peer_discovery_aws
... [info] <0.236.0> Will try to lock with peer discovery backend rabbit_peer_discovery_aws
... [info] <0.236.0> Attempting to obtain EC2 IMDSv2 token from "http://169.254.169.254/latest/api/token" ...
... [info] <0.236.0> Will try to lock connecting to nodes ['rabbit@ip-10-102-9-217',
... [info] <0.236.0>                                       'rabbit@ip-10-102-9-34']
... [info] <0.236.0> All discovered existing cluster peers: rabbit@ip-10-102-9-217, rabbit@ip-10-102-9-34
... [info] <0.236.0> Peer nodes we can cluster with: rabbit@ip-10-102-9-217
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
... [warning] <0.236.0> Could not auto-cluster with node rabbit@ip-10-102-9-217: {badrpc,nodedown}
... [info] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
... [info] <0.236.0> Could not successfully contact any node of: rabbit@ip-10-102-9-217 (as in Erlang distribution). Starting as a blank standalone node...

Details about the system

OS PID: 16296
OS: Linux
Uptime (seconds): 1086
Is under maintenance?: false
RabbitMQ version: 3.12.2
RabbitMQ release series support status: supported
Node name: rabbit@ip-10-102-9-34
Erlang configuration: Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.0.8 7 Feb 2023
Erlang processes: 374 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Config file /etc/rabbitmq/rabbitmq.conf

# server config
cluster_name = arx-us-east-1-qa
listeners.tcp.default = 5672
cluster_partition_handling = pause_minority
# management ui
management.tcp.port = 15672
management.http_log_dir = /var/log/rabbitmq/http-access.log
# net configs
tcp_listen_options.backlog = 128
tcp_listen_options.nodelay = true
tcp_listen_options.linger.on = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.exit_on_close = false
tcp_listen_options.keepalive = false
# cluster auto-discovery settings
cluster_formation.peer_discovery_backend = aws
cluster_formation.aws.region = us-east-1
cluster_formation.aws.instance_tags.app = arx
cluster_formation.aws.instance_tags.service = rabbitmq
cluster_formation.aws.instance_tags.environment = qa
cluster_formation.node_cleanup.only_log_warning = true
# Log config
log.file.level = info

Enabled plugins /etc/rabbitmq/enabled_plugin

[rabbitmq_management,rabbitmq_peer_discovery_aws].

Reproduction steps

1. Install RabbitMQ on a new Amazon Linux 2023 instance
2. configure Rabbitmq
3. Start rabbitmq

Expected behavior

Should be able to see cluster formed and nodes joined

Additional context

I have verified the security groups for the EC2 instances, inbound is open to all port on the entire VPC CIDR
I can see the process listening on the expected ports using Netstat

I even tried the AWS Reachability analyzer tools and get success on all ports 4369,5672,25672

I can ping an instance from the other using the instance hostname, I can Telnet on port 5672

I do not find any useful info on Stackoverflow or web serched.

Setting debug level does not return any useful info

[michaelklishin]

Please use Discussions for questions.

Peer discovery mechanisms log a lot of details, including HTTP API requests the perform, at debug log level.

"nodedown" indicates that nodes cannot contact each other. Again, see logs on both nodes, due to a network connectivity issue or a shared secret mismatch.

Cluster formation guide explains the prerequisites for every mechanism, including AWS. ping is next to useless when it comes to troubleshooting network connectivity because it does not use TCP (and RabbitMQ does).

By the way, two node clusters are an explicitly unsupported configuration. That is, the cluster will form but… it's all covered in the doc section.

[limjf85]

I understand 2 nodes are not recommended, this is a POC for an IAC pipeline for a larger cluster with odd number of nodes, HA configs, etc.

I did try connectivity via TCP with the AWS Reachability Analyzer tools to the default ports and is successful.
I can also confirm the cookies match, I can see matching cookie hashes in both server logs, these servers are created using IAC so they are basically Identical.

The OS firewall is disabled.

I will share with you the logs in debug mode, maybe you spot something I am missing.

Thanks for your time, I really appreciate all the help

[michaelklishin]

Since you already have existing cluster members, two more things are worth mentioning:

• Logs from the node this newly joining one would be more useful. For example, they will log errors when the peer tries to connect with a mismatching shared secret
• Existing cluster members do not perform peer discovery. This is not the case with this node according to the last log snippet line but it can be confusing
• When autoscaling groups are used, previous ASG members potentially have to be cleaned up

[limjf85]

Launched the cluster using a barebones AWS Linux 2023 AMI
Cluster formed as expected

There is some OS config we are messing up in our images when hardening the instance according to CIS.
Will update for future reference

[michaelklishin]

Can someone potentially prevent the process from opening outgoing [TCP] connections?

[limjf85]

Is possible, we are digging through the code that build the AMI

[limjf85]

Found the root cause of our issue
we were applying CIS 3.1.1 and 3.1.2 and this was breaking cluster communication

Linux 3.1.1	Network tuning policy recommendations	Ensure that IP forwarding is disabled.	/etc/security/pscxpert/bin/adapter /etc/security/pscxpert/bin/network-tuning-options-rules
Arguments: ipforwarding false

Linux 3.1.2	Network tuning policy recommendations	Ensure that packet redirect sending is disabled.	/etc/security/pscxpert/bin/adapter /etc/security/pscxpert/bin/network-tuning-options-rules
Arguments: send_redirects false