CLI tools fail to connect to a node using inter-node TLS on Erlang 26

[Abraxos]

Describe the bug

I've run into a fairly serious issue. Namely that I use TLS for inter-node communication and ever since I upgraded some of the nodes in my cluster to RabbitMQ 3.12.0 and Erlang 26, the nodes cannot cluster together. Further investigation demonstrated that despite following instructions almost to the letter (see my notes below on where I differ) the issue persists even in he most trivial case of using certificates with rabbitmqctl.

I would appreciate any help in diagnosing this issue.

Thank you.

Reproduction steps

Please follow the instructions below on any Ubuntu 22.04 machine, I am confident that they should play out basically the same way, however, I did it on a VM with the hostname rabbitmq-test-vm-0 which is an important tidbit for some of the configuration.

Set up a single machine

Pre-requisites

apt-get update --fix-missing;
apt-get upgrade -y;
apt-get dist-upgrade -y;
apt install -y tmux emacs-nox apt-transport-https socat logrotate init-system-helpers adduser;

RabbitMQ & Erlang

A copy of the instructions provided here: https://www.rabbitmq.com/install-debian.html#apt-cloudsmith except executed as root from the get-go

apt-get install curl gnupg apt-transport-https -y

curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | gpg --dearmor | tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
curl -1sLf https://ppa1.novemberain.com/gpg.E495BB49CC4BBE5B.key | gpg --dearmor | tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
curl -1sLf https://ppa1.novemberain.com/gpg.9F4587F226208342.key | gpg --dearmor | tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null

tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
deb [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main

deb [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
EOF

apt-get update -y

apt-get install -y erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

apt-get install rabbitmq-server -y --fix-missing

Verify that everything seems OK

root@rabbitmq-test-vm-0:/home/eugene# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-test-vm-0 ...
Basics

Cluster name: rabbit@rabbitmq-test-vm-0
Total CPU cores available cluster-wide: 4

Disk Nodes

rabbit@rabbitmq-test-vm-0

Running Nodes

rabbit@rabbitmq-test-vm-0

Versions

rabbit@rabbitmq-test-vm-0: RabbitMQ 3.12.0 on Erlang 26.0.1

CPU Cores

Node: rabbit@rabbitmq-test-vm-0, available CPU cores: 4

Maintenance status

Node: rabbit@rabbitmq-test-vm-0, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rabbitmq-test-vm-0, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-test-vm-0, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Configure TLS

Generate Certificates

Install mkcert

The easiest thing, from my perspective is to generate certs using mkcrt

apt install -y libnss3-tools;
curl -JLO "https://dl.filippo.io/mkcert/latest?for=linux/amd64";
chmod +x mkcert-v*-linux-amd64;
mv mkcert-v*-linux-amd64 /usr/local/bin/mkcert;

We can check that it works by running mkcert --help

Generate Relevant Certificates

Note that I am assuming that the hostname of the VM is rabbitmq-test-vm-0, if you use a different name, then please use that in the incantations below accordingly. It is important that you be root while generating these certificates, otherwise the locations and permissions may get messed up.

cd /etc/rabbitmq;
mkcert rabbitmq-test-vm-0 localhost 127.0.0.1 ::1;
cp /root/.local/share/mkcert/rootCA.pem /etc/rabbitmq/ca.crt;
chown rabbitmq:rabbitmq /etc/rabbitmq/*

Verify That Certs are Valid

You should have the following in /etc/rabbitmq

root@rabbitmq-test-vm-0:/etc/rabbitmq# ls -l /etc/rabbitmq
total 12
-rw-r--r-- 1 rabbitmq rabbitmq 1671 Jun 13 22:48 ca.crt
-rw------- 1 rabbitmq rabbitmq 1704 Jun 13 22:47 rabbitmq-test-vm-0+3-key.pem
-rw-r--r-- 1 rabbitmq rabbitmq 1549 Jun 13 22:47 rabbitmq-test-vm-0+3.pem

We can check that the certificate identifies the hostname rabbitmq-test-vm-0 with the following incantation:

openssl verify -CAfile /etc/rabbitmq/ca.crt -verify_hostname rabbitmq-test-vm-0 /etc/rabbitmq/rabbitmq-test-vm-0+3.pem

The output should be something like: /etc/rabbitmq/rabbitmq-test-vm-0+3.pem: OK however, if you use google.com instead of rabbitmq-test-vm-0, then you should see error 62 at 0 depth lookup: hostname mismatch.

Configure RabbitMQ for TLS with Inter-node Communication

TLS Settings

Write the following to /etc/rabbitmq/inter_node_tls.config. Note that this is taken directly from https://www.rabbitmq.com/clustering-ssl.html#strategy-two-flags except that the key password line is removed because the key we generated above does not have a password.

[
  {server, [
    {cacertfile, "/etc/rabbitmq/ca.crt"},
    {certfile,   "/etc/rabbitmq/rabbitmq-test-vm-0+3.pem"},
    {keyfile,    "/etc/rabbitmq/rabbitmq-test-vm-0+3-key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
  ]},
  {client, [
    {cacertfile, "/etc/rabbitmq/ca.crt"},
    {certfile,   "/etc/rabbitmq/rabbitmq-test-vm-0+3.pem"},
    {keyfile,    "/etc/rabbitmq/rabbitmq-test-vm-0+3-key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
  ]}
].

Then we write the following to /etc/rabbitmq/rabbitmq-env.conf. This is also taken verbatim from the page mentioned above, except that ERL_SSL_PATH is found in /usr/lib/erlang/lib/ssl-11.0.1/ebin

ERL_SSL_PATH="/usr/lib/erlang/lib/ssl-11.0.1/ebin"
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"
RABBITMQ_CTL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"

Hostname Settings

In order to make sure the Hostnames are not screwing with us, its important to check that the following line is present in the /etc/hosts file:

127.0.0.1 rabbitmq-test-vm-0

Configure RabbitMQ

Now we write the following to /etc/rabbitmq/rabbitmq.conf as per the instructions https://www.rabbitmq.com/ssl.html:

# disables non-TLS listeners, only TLS-enabled clients will be able to connect
listeners.tcp = none

listeners.ssl.default = 5671

ssl_options.cacertfile = /etc/rabbitmq/ca.crt
ssl_options.certfile   = /etc/rabbitmq/rabbitmq-test-vm-0+3.pem
ssl_options.keyfile    = /etc/rabbitmq/rabbitmq-test-vm-0+3-key.pem
ssl_options.verify     = verify_peer
ssl_options.fail_if_no_peer_cert = true

and restart rabbitmq with: service rabbitmq-server restart, checking the status with service rabbitmq-server status should produce something like this:

root@rabbitmq-test-vm-0:/etc/rabbitmq# service rabbitmq-server status
● rabbitmq-server.service - RabbitMQ broker
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-06-13 23:13:53 UTC; 5s ago
   Main PID: 16885 (beam.smp)
      Tasks: 29 (limit: 4571)
     Memory: 113.3M
        CPU: 2.030s
     CGroup: /system.slice/rabbitmq-server.service
             ├─16885 /usr/lib/erlang/erts-14.0.1/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -pc unicode -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -- -root /usr/lib/erlang -bindir /usr/lib/erlang/erts-14.0.1/bin -progname erl -- -home /var/lib/rabbitmq -- -pa "" -noshell -noinput -s rabbit boot -boot start_sasl -pa /usr/lib/erlang/lib/ssl-11.0.1/ebin -proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config -syslog logger "[]" -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false -enable-feature maybe_expr
             ├─16895 erl_child_setup 32768
             ├─16928 /usr/lib/erlang/erts-14.0.1/bin/epmd -daemon
             ├─16955 /usr/lib/erlang/erts-14.0.1/bin/inet_gethost 4
             ├─16956 /usr/lib/erlang/erts-14.0.1/bin/inet_gethost 4
             └─16959 /bin/sh -s rabbit_disk_monitor

Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Release series support status: supported
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Doc guides:  https://rabbitmq.com/documentation.html
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Support:     https://rabbitmq.com/contact.html
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Tutorials:   https://rabbitmq.com/getstarted.html
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Monitoring:  https://rabbitmq.com/monitoring.html
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Logs: /var/log/rabbitmq/[email protected]
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:         <stdout>
Jun 13 23:13:52 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Config file(s): /etc/rabbitmq/rabbitmq.conf
Jun 13 23:13:53 rabbitmq-test-vm-0 rabbitmq-server[16885]:   Starting broker... completed with 0 plugins.
Jun 13 23:13:53 rabbitmq-test-vm-0 systemd[1]: Started RabbitMQ broker.

The Problem

And now the problem should present itself in that even rabbitmqctl cannot talk to RabbitMQ on the same machine. I discovered this issue when my nodes could not connect to each other, however, I suspect that whatever is preventing rabbitmqctl from connecting would also prevent the nodes talking to each other, so this seems like a good way to rule out networking and firewall issues.

root@rabbitmq-test-vm-0:/etc/rabbitmq# rabbitmqctl cluster_status
Error: unable to perform an operation on node 'rabbit@rabbitmq-test-vm-0'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@rabbitmq-test-vm-0
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-test-vm-0']

rabbit@rabbitmq-test-vm-0:
  * connected to epmd (port 4369) on rabbitmq-test-vm-0
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 
  * suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more


Current node details:
 * node name: 'rabbitmqcli-777-rabbit@rabbitmq-test-vm-0'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Qv4zds2Y1MAbPXvyY70t4g==

Please note that its important to check that the cookie used by rabbitmqctl is the same the server, but this should be easy enough with:

Current node details:
 * node name: 'rabbitmqcli-527-rabbit@rabbitmq-test-vm-0'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Qv4zds2Y1MAbPXvyY70t4g==

root@rabbitmq-test-vm-0:/etc/rabbitmq# cat /var/log/rabbitmq/rabbit\@rabbitmq-test-vm-0.log | grep cookie
2023-06-13 22:39:52.487504+00:00 [info] <0.237.0>  cookie hash    : Qv4zds2Y1MAbPXvyY70t4g==
2023-06-13 23:09:31.924262+00:00 [info] <0.237.0>  cookie hash    : Qv4zds2Y1MAbPXvyY70t4g==
2023-06-13 23:13:50.194523+00:00 [info] <0.259.0>  cookie hash    : Qv4zds2Y1MAbPXvyY70t4g==
2023-06-13 23:13:52.719066+00:00 [info] <0.259.0>  cookie hash    : Qv4zds2Y1MAbPXvyY70t4g==

Expected behavior

I would expect the output of the command rabbitmqctl cluster_status to be exactly the same as it was in the section labeled Verify that everything seems OK above (modulo some TLS details).

Additional context

I spent some time working through this issue with @mkuratczyk on Discord, but we were unable to resolve the issue. I would like to believe that its somewhere in my configuration, but for the life of me, I cannot figure out where. This is especially worrisome as for the purposes of filing this bug, I created a completely fresh setup with a configuration using only snippets from the documentation (thus avoiding any potential issues with the actual configuration I used which originally experienced this issue).

Thank you for your help in advance.

[michaelklishin]

Inter-node communication is implemented by the runtime, not RabbitMQ. You can try Erlang 25.3.x to compare.

Erlang 26 is one month old, neither the core team nor the community have a lot of experience with it.

[Abraxos]

and when I try to install after changing the /etc/apt/preferences.d/erlang file, it still installs 26.0 rather than 25.3

[michaelklishin]

apt supports specifying version numbers together with package name:

# remove any currently installed erlang-* packages
sudo apt-get remove -y erlang*;
# install 25.3.2.2
sudo apt-get install -y erlang-base=1:25.3.2.2-1 erlang-asn1=1:25.3.2.2-1 erlang-crypto=1:25.3.2.2-1 erlang-eldap=1:25.3.2.2-1 erlang-ftp=1:25.3.2.2-1 erlang-inets=1:25.3.2.2-1                         erlang-mnesia=1:25.3.2.2-1 erlang-os-mon=1:25.3.2.2-1 erlang-parsetools=1:25.3.2.2-1 erlang-public-key=1:25.3.2.2-1 erlang-runtime-tools=1:25.3.2.2-1 erlang-snmp=1:25.3.2.2-1 erlang-ssl=1:25.3.2.2-1 erlang-syntax-tools=1:25.3.2.2-1 erlang-tftp=1:25.3.2.2-1 erlang-tools=1:25.3.2.2-1 erlang-xmerl=1:25.3.2.2-1

[michaelklishin]

I think I know what the issue is in the example. Debian versions are convoluted and include package versions (the "epoch") as well as the version of the tool.

So the version in your example should be 1:25.3.2.2-1 (or, well, 1:25.3-1), and the -1 is missing from the docs. Not any more.

You can inspect effective policy for a specific package like so

sudo apt-cache policy erlang-base

[Abraxos]

OK, i installed erlang using the incantation you provided and then installed rabbitmq-server. I also had to change /etc/rabbitmq/rabbitmq-env.conf to use /usr/lib/erlang/lib/ssl-10.9.1/ebin/ as the SSL path. I confirmed that it is running:

root@rabbitmq-test-vm-0:/etc/apt/preferences.d# service rabbitmq-server status
● rabbitmq-server.service - RabbitMQ broker
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-06-14 00:21:13 UTC; 1min 32s ago
   Main PID: 20635 (beam.smp)
      Tasks: 29 (limit: 4571)
     Memory: 104.5M
        CPU: 2.445s
     CGroup: /system.slice/rabbitmq-server.service
             ├─20635 /usr/lib/erlang/erts-13.2.2.1/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -pc unicode -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -- -root /usr/lib/erlang -bindir /usr/lib/erlang/erts-13.2.2.1/bin -progname erl -- -home /var/lib/rabbitmq -- -pa "" -noshell -noinput -s rabbit boot -boot start_sasl -pa /usr/lib/erlang/lib/ssl-11.0.1/ebin -proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config -syslog logger "[]" -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false -enable-featu
re maybe_expr
             ├─20645 erl_child_setup 32768
             ├─20678 /usr/lib/erlang/erts-13.2.2.1/bin/epmd -daemon
             ├─20705 /usr/lib/erlang/erts-13.2.2.1/bin/inet_gethost 4
             ├─20706 /usr/lib/erlang/erts-13.2.2.1/bin/inet_gethost 4
             └─20709 /bin/sh -s rabbit_disk_monitor

Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Release series support status: supported
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Doc guides:  https://rabbitmq.com/documentation.html
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Support:     https://rabbitmq.com/contact.html
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Tutorials:   https://rabbitmq.com/getstarted.html
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Monitoring:  https://rabbitmq.com/monitoring.html
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Logs: /var/log/rabbitmq/[email protected]
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:         <stdout>
Jun 14 00:21:12 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Config file(s): /etc/rabbitmq/rabbitmq.conf
Jun 14 00:21:13 rabbitmq-test-vm-0 rabbitmq-server[20635]:   Starting broker... completed with 0 plugins.
Jun 14 00:21:13 rabbitmq-test-vm-0 systemd[1]: Started RabbitMQ broker.

However, even with Erlang 25.3 the issue persists. I verified through the server logs that I am running the version you suggested: 2023-06-14 00:26:50.017328+00:00 [info] <0.254.0> Starting RabbitMQ 3.12.0 on Erlang 25.3.2.2 [jit]

root@rabbitmq-test-vm-0:/etc/apt/preferences.d# rabbitmqctl cluster_status
Error: unable to perform an operation on node 'rabbit@rabbitmq-test-vm-0'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@rabbitmq-test-vm-0
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-test-vm-0']

rabbit@rabbitmq-test-vm-0:
  * connected to epmd (port 4369) on rabbitmq-test-vm-0
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 
  * suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more


Current node details:
 * node name: 'rabbitmqcli-308-rabbit@rabbitmq-test-vm-0'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Qv4zds2Y1MAbPXvyY70t4g==

[Abraxos]

I think I know what the issue is in the example. Debian versions are convoluted and include package versions (the "epoch") as well as the version of the tool.

So the version in your example should be 1:25.3.2.2-1 (or, well, 1:25.3-1), and the -1 is missing from the docs. Not any more.

You can inspect effective policy for a specific package like so

sudo apt-cache policy erlang-base

Yup, this worked to get the policy set up properly (and I doubly appreciate you updating the docs), however, I managed to install erlang 25.3 with the incantation you posted, and it was no better =/

[michaelklishin]

I upgraded some of the nodes in my cluster to RabbitMQ 3.12.0 and Erlang 26, the nodes cannot cluster together

There is even less experience with mixed Erlang 25 and 26 clusters. Consider upgrading all nodes to 26 or using 25.3.x on all nodes.

[Abraxos]

I believe the problem is pretty narrow. I spent a lot of time creating copy-paste commands that can be used to reproduce the issue specifically to make it as easy to figure this out as possible, and I narrowed it down to the smallest possible version of the problem. I can worry about the actual clustering issue later, for now, I cannot even use rabbitmqctl on a single node, which is a pre-requisite for working on clustering.

[michaelklishin]

We will try to reproduce this as time permits. Your question states that you have upgraded "some of your nodes", I did not realize that they were single node clusters.

[Abraxos]

I only mentioned upgrading "some of [my] nodes" to demonstrate the provenance of the problem. Since then, as I noted in my writeup, I narrowed down the problem to the example where rabbitmqctl doesn't work.

Further investigation demonstrated that despite following instructions almost to the letter (see my notes below on where I differ) the issue persists even in he most trivial case of using certificates with rabbitmqctl.

[Abraxos]

Please take your time to investigate, I am not in a hurry, and I can take the time to run various tests you suggest to help diagnose. However, I do advise following my writeup to see if you get the same issue. If you do not, it may be warranted for me to send you an image of some kind.

[michaelklishin]

We will, thanks for all the details. We will first follow our own docs on Erlang 25.3 and then 26, though.

[michaelklishin]

Erlang 26 release notes mention at least two changes around the distribution protocol, and one specifically around distribution over TLS.

Therefore intentional and unintentional breaking changes are likely.

When/if there is an isolated way to reproduce a behavior on 26 that is not present on 25 with the same
configuration, we can try to narrow it down further and report it to the Erlang/OTP team.

Nothing around how Erlang distribution is used by RabbitMQ has changed in 3.12.0 (in fact, in several years).

[michaelklishin]

Here is an example of what kind of changes can affect existing installations that use #8548. Could be something similar in the case of TLS for inter-node and CLI tools communication.

[Abraxos]

How can I use this information?

[lukebakken]

Hello,

Please use openssl s_client to verify that the TLS handshake succeeds when connecting to the following ports:

• 25672
• 5671

When you run the command, capture the full command and full output into a file, and attach the file to your response. Please do NOT paste a giant wall of text.

Based on the information you provided, these are the commands you should run:

openssl s_client -connect localhost:25672 -CAfile /etc/rabbitmq/ca.crt -cert /etc/rabbitmq/rabbitmq-test-vm-0+3.pem -key /etc/rabbitmq/rabbitmq-test-vm-0+3-key.pem -verify_hostname rabbitmq-test-vm-0 -verify_depth 8

openssl s_client -connect localhost:5671 -CAfile /etc/rabbitmq/ca.crt -cert /etc/rabbitmq/rabbitmq-test-vm-0+3.pem -key /etc/rabbitmq/rabbitmq-test-vm-0+3-key.pem -verify_hostname rabbitmq-test-vm-0 -verify_depth 8

Thanks!

[Abraxos]

Done, here you go

5671.txt
25672.txt

[lukebakken]

Thanks, nothing looks out of the ordinary. We'll try to find time to take a look.

[Abraxos]

Understood, thank you for looking into it

[lukebakken]

@Abraxos - please take a look at this project's tls branch:

https://github.com/lukebakken/docker-rabbitmq-cluster/tree/tls

Everything works correctly using inter-node TLS. Note that the docker image is currently using Erlang 25.3.2.2.

I'm guessing that, in your case, there may be something that Erlang doesn't like about the certs created by mkcrt. The above project uses our tls-gen project which we know creates certificates that can be used by Erlang.

Start by comparing my project and its configuration files to yours, as well as the X509 certs themselves. Use this command to see a text representation of the certs:

openssl x509 -text -in rmq0/server_rmq0.local_certificate.pem

Be sure to check the X509v3 extensions values - compare between our generated certs and yours.

[Abraxos]

I will check this out, but I won't have time to do it until tomorrow.

One potentially important thing to note: I am not using mkcert in my production environment where I first found this issue, I use openssl incantations in a python script. I only opted to include mkcert here for the purposes of creating an easy process by which the issue could be replicated, and mkcert happened to replicate the issue. That being said, it will be important to note the differences between mkcert and tls-gen if this does turn out to be the cause of the problem.

[Abraxos]

Alright, I ran the test with tls-gen and at first, it did not work, however, I did notice that tls-gen generates both server and client certificates, so I decided to give it a go by giving rabbitmqctl the client certs instead, and that DID work. So it seems like the change is that now, in order to connect with rabbitmqctl one must have a client-specific certificate, rather than just re-using the server cert for both as I used to do.

[Abraxos]

Just as a side note, I went and checked on my production system, and I have a certificate there that is extended key usage for both client and server authentication. I suppose I can change my system to separate this certificate into two, but is there a documented reason why one ought not to use a certificate as both a client and server that justifies RabbitMQ/Erlang rejecting this kind of cert?

[lukebakken]

I'm glad that fixed your issue.

You would have to ask the Erlang team at Ericsson about Erlang's TLS implementation. Erlang's TLS implementation is pretty picky about the values in X509v3 extensions. RabbitMQ of course has no control over that.

[michaelklishin]

x.509 extensions separate different usage scenarios. Some implementations may be stricter than others in enforcing them for clients and servers.

This is covered in RFC5280 in case someone really cares to know more.

A single certificate can list most usages, as most of them do not conflict. Why this is not recommended from the security (peer verification) perspective? For the same reason why using
the same pair of password-based credentials is not. In some environments, it'd be a reasonable
decision to make.

FWIW, tls-gen produces at least two pairs of certificates and keys with differentiated usage extensions (which you also can customize).

[michaelklishin]

One of my first hypotheses was that tls-gen (that we've developed and use across the board) produces certificates with a different set of usage extensions or something like that, from mkcrt.
But I don't have any reasons to doubt mkcrt per se, so I figured it'd be too early to bring it up.