Can't enable TLS on with RabbitMQ v3.9.13, Erlang 24.2 on Ubuntu 18.04

[yue-myob]

Hi all,

I try to enable TLS on my rabbitmq-server v3.9.13, it doesn't have any error and the connection be closed by mq-server.

I use this way to test the port 5671:

> openssl s_client -connect localhost:5671

CONNECTED(00000005)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 311 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Here are mq server logs:

2022-02-11 12:15:41.361606+00:00 [info] <0.613.0> Supervisor {<0.613.0>,tls_dyn_connection_sup}: child sender started (<0.614.0>): {tls_sender,start_link,[]}
2022-02-11 12:15:41.361845+00:00 [info] <0.613.0> Supervisor {<0.613.0>,tls_dyn_connection_sup}: child receiver started (<0.615.0>): {ssl_gen_statem,start_link,[server,<0.614.0>,"localhost",5671,#Port<0.48>,{#{fallback => undefined,next_protocols_advertised => undefined,server_name_indication => undefined,middlebox_comp_mode => true,password => [],sni_hosts => [],versions => [{3,3}],renegotiate_at => 268435456,eccs => {elliptic_curves,[{1,3,132,0,39},{1,3,132,0,38},{1,3,132,0,35},{1,3,36,3,3,2,8,1,1,13},{1,3,132,0,36},{1,3,132,0,37},{1,3,36,3,3,2,8,1,1,11},{1,3,132,0,34},{1,3,132,0,16},{1,3,132,0,17},{1,3,36,3,3,2,8,1,1,7},{1,3,132,0,10},{1,2,840,10045,3,1,7},{1,3,132,0,3},{1,3,132,0,26},{1,3,132,0,27},{1,3,132,0,32},{1,3,132,0,33},{1,3,132,0,24},{1,3,132,0,25},{1,3,132,0,31},{1,2,840,10045,3,1,1},{1,3,132,0,1},{1,3,132,0,2},{1,3,132,0,15},{1,3,132,0,9},{1,3,132,0,8},{1,3,132,0,30}]},sni_fun => undefined,padding_check => true,cacerts => undefined,customize_hostname_check => [],ocsp_responder_certs => [],reuse_session => #Fun<ssl.13.32502971>,supported_groups => {supported_groups,[x25519,x448,secp256r1,secp384r1]},client_renegotiation => true,key => undefined,erl_dist => false,srp_identity => undefined,reuse_sessions => true,signature_algs => [{sha512,ecdsa},{sha512,rsa},{sha384,ecdsa},{sha384,rsa},{sha256,ecdsa},{sha256,rsa},{sha224,ecdsa},{sha224,rsa},{sha,ecdsa},{sha,rsa},{sha,dsa}],crl_cache => {ssl_crl_cache,{internal,[]}},depth => 10,keep_secrets => false,psk_identity => undefined,cookie => true,ocsp_stapling => false,user_lookup_fun => undefined,use_ticket => undefined,anti_replay => undefined,handshake => full,dhfile => undefined,cert => undefined,partial_chain => #Fun<ssl.11.32502971>,alpn_preferred_protocols => undefined,key_update_at => 388736063997,early_data => undefined,cacertfile => <<"/etc/rabbitmq/ssl/ca_certificate.pem">>,verify => verify_none,certfile => <<"/etc/rabbitmq/ssl/server_certificate.pem">>,keyfile => <<"/etc/rabbitmq/ssl/server_key.pem">>,fail_if_no_peer_cert => false,honor_cipher_order => true,honor_ecc_order => true,max_fragment_length => undefined,signature_algs_cert => undefined,ocsp_nonce => true,dh => undefined,max_handshake_size => 262144,verify_fun => {#Fun<ssl.12.32502971>,[]},ciphers => [<<"À,">>,<<"À0">>,<<"À$">>,<<"À(">>,<<"À.">>,<<"À2">>,<<"À&">>,<<"À*">>,<<0,159>>,<<0,163>>,<<0,107>>,<<0,106>>,<<"À+">>,<<"À/">>,<<"À#">>,<<"À'">>,<<"À-">>,<<"À1">>,<<"À%">>,<<"À)">>,<<0,158>>,<<0,162>>,<<0,103>>,<<0,64>>,<<"À\n">>,<<192,20>>,<<0,57>>,<<0,56>>,<<192,5>>,<<192,15>>,<<"À\t">>,<<192,19>>,<<0,51>>,<<0,50>>,<<192,4>>,<<192,14>>],secure_renegotiate => true,next_protocol_selector => undefined,beast_mitigation => one_n_minus_one,alpn_advertised_protocols => undefined,session_tickets => disabled,crl_check => false,log_level => notice,hibernate_after => infinity,protocol => tls},{socket_options,binary,raw,0,0,false},[{option_tracker,<0.579.0>},{session_tickets_tracker,disabled},{session_id_tracker,<0.580.0>}]},<0.581.0>,{gen_tcp,tcp,tcp_closed,tcp_error,tcp_passive}]}

Information:

• 18.04.6 LTS (Bionic Beaver)
• rabbitmq-server v3.9.13
• erl v24.2.1

The certs are generated by tls-gen, it can be verified by openssl.

openssl s_server -accept 8443 -cert server_certificate.pem -key server_key.pem -CAfile ca_certificate.pem
openssl s_client -connect localhost:8443 -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem -verify 8

My rabbitmq config is:

listeners.ssl.default = 5671
handshake_timeout = 10000
listeners.ssl.1 = 5671
ssl_options.cacertfile           = /etc/rabbitmq/ssl/ca_certificate.pem
ssl_options.certfile             = /etc/rabbitmq/ssl/server_certificate.pem
ssl_options.keyfile              = /etc/rabbitmq/ssl/server_key.pem
ssl_options.honor_cipher_order   = true
ssl_options.honor_ecc_order      = true
ssl_options.versions.1 = tlsv1.2
ssl_options.ciphers.1  = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.2  = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3  = ECDHE-ECDSA-AES256-SHA384
ssl_options.ciphers.4  = ECDHE-RSA-AES256-SHA384
ssl_options.ciphers.5  = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.6  = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.7  = ECDH-ECDSA-AES256-SHA384
ssl_options.ciphers.8  = ECDH-RSA-AES256-SHA384
ssl_options.ciphers.9  = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.10 = DHE-DSS-AES256-GCM-SHA384
ssl_options.ciphers.11 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.12 = DHE-DSS-AES256-SHA256
ssl_options.ciphers.13 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.14 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.15 = ECDHE-ECDSA-AES128-SHA256
ssl_options.ciphers.16 = ECDHE-RSA-AES128-SHA256
ssl_options.ciphers.17 = ECDH-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.18 = ECDH-RSA-AES128-GCM-SHA256
ssl_options.ciphers.19 = ECDH-ECDSA-AES128-SHA256
ssl_options.ciphers.20 = ECDH-RSA-AES128-SHA256
ssl_options.ciphers.21 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.22 = DHE-DSS-AES128-GCM-SHA256
ssl_options.ciphers.23 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.24 = DHE-DSS-AES128-SHA256
ssl_options.ciphers.25 = ECDHE-ECDSA-AES256-SHA
ssl_options.ciphers.26 = ECDHE-RSA-AES256-SHA
ssl_options.ciphers.27 = DHE-RSA-AES256-SHA
ssl_options.ciphers.28 = DHE-DSS-AES256-SHA
ssl_options.ciphers.29 = ECDH-ECDSA-AES256-SHA
ssl_options.ciphers.30 = ECDH-RSA-AES256-SHA
ssl_options.ciphers.31 = ECDHE-ECDSA-AES128-SHA
ssl_options.ciphers.32 = ECDHE-RSA-AES128-SHA
ssl_options.ciphers.33 = DHE-RSA-AES128-SHA
ssl_options.ciphers.34 = DHE-DSS-AES128-SHA
ssl_options.ciphers.35 = ECDH-ECDSA-AES128-SHA
ssl_options.ciphers.36 = ECDH-RSA-AES128-SHA
ssl_handshake_timeout = 5000

log.file.level = debug

Check ports on the mq server:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      4468/systemd-resolv
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      914/sshd
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN      26984/beam.smp
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      26984/beam.smp
tcp6       0      0 :::4369                 :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      914/sshd
tcp6       0      0 :::5671                 :::*                    LISTEN      26984/beam.smp
tcp6       0      0 :::5672                 :::*                    LISTEN      26984/beam.smp

Use rabbitmq-diagnostics listeners to verify that TLS has been enabled on the node:

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS

> rabbitmq-diagnostics --silent tls_versions
tlsv1.3
tlsv1.2
tlsv1.1
tlsv1

Does any have any idea about enable TLS on my rabbitmq-server v3.9.13, what's wrong on the config?

[anbland]

Did you create the certificates with a password? if so, try adding the password to the config file as a ssl_option.
ssl_options.password = xyz

[caijingwen-vincent]

Did you create the certificates with a password? if so, try adding the password to the config file as a ssl_option. ssl_options.password = xyz

oh my god , I found the whole internet , finally I got the answer from you.

[michaelklishin]

Providing Private Key Passwords. IIRC modern Erlang versions log something when the password is incorrect. RabbitMQ does not implement TLS or PKI, so we cannot improve logging on our side.

[caijingwen-vincent]

Amazing speed of update, salute

[lukebakken]

Try again without any of the TLS cipher settings.

[michaelklishin]

RabbitMQ does not implement TLS, Erlang/OTP and OpenSSL do in combination (the latter implements all the cipher suites and crypto functions). In case of TLS 1.3, OpenSSL 1.1.x is a practical requirement.

TLS 1.3 has no cipher suite overlap with earlier versions. Two things immediately stand out in this setup:

• It does not use the 1.3 cipher suites our docs list
• It runs on an four year old Ubuntu distribution which only provides OpenSSL 1.1 via updates/backports repositories

When connecting with an actual client, you also need to make sure that the client's runtime implements TLS 1.3 and its support is not considered experimental, and that the client is configured to use TLS 1.3.

TLS 1.3 is an all-or-nothing proposition by design. You cannot mix-it-and-match with earlier versions
since there is no supported cipher suite overlap.

[michaelklishin]

Oh, and this line

ssl_options.versions.1 = tlsv1.2

instructs RabbitMQ to use TLS 1.2 only when negotiating a suitable connection. See the TLS 1.3 configuration example in the RabbitMQ TLS guide.