OpenSSL 3 Issues connecting to RMQ

[SpaceLineCrash]

Hello RMQ,

I have a new environment using RabbitMQ 3.12.4 with Erlang 26. Our Ubuntu 22 clients have been unable to connect to this 3 node cluster due to an SSL error. "ERROR [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)". Clients are connecting through a HAProxy load balancer but we also attempted to connect directly which failed.

The Ubuntu 20 clients do NOT have this issue above. The only major difference I am aware of is Openssl v1.1.1 on Ub20 to Openssl v3.0.2 on Ub22. From documentation, ERL 26 and RMQ 3.12 state they support Openssl V3.

What configuration might I look into for my server or my clients to troubleshoot this?

Haproxy Config:
frontend f-client1fronend-5671
mode tcp
maxconn 2500
bind 1.1.1.4:5671 name 1.1.1.4:5671
timeout client 500000
tcp-request connection track-sc1 src
default_backend b-upqueue2x2p-5671
stick-table type ip size 1048576 expire 300000 store gpc0,conn_rate(1m),bytes_in_rate(10s),bytes_out_rate(10s),conn_cnt,sess_rate(10s)
backend b-clientbackendserver-5671
mode tcp
balance roundrobin
timeout server 500000
server clientbackendserver1_5671 1.1.1.1:5671 check fall 3 inter 2000 rise 5 weight 10
server clientbackendserver2_5671 1.1.1.2:5671 check fall 3 inter 2000 rise 5 weight 10
server clientbackendserver3_5671 1.1.1.3:5671 check fall 3 inter 2000 rise 5 weight 10

RabbitMQ Failing Client: Ubuntu 22, Openssl v3.0.2, Python3.10.12, pyOpenSSL 21.0.0

RabbitMQ Server Config below:

RabbitMQ General Settings

heartbeat = 600
proxy_protocol = true
log.file.level = info
log.dir = /var/log/rabbitmq
log.file = rabbit.log
log.exchange = true
log.exchange.level = info

RabbitMQ Connection Settings

listeners.tcp = none
listeners.ssl.1 = 5671
auth_mechanisms.1 = EXTERNAL
auth_mechanisms.2 = PLAIN
auth_mechanisms.3 = AMQPLAIN
cluster_name = Client1-Prod
cluster_formation.peer_discovery_backend = dns
ssl_cert_login_from = common_name
ssl_options.cacertfile = /etc/ssl/lst/client1.cacert.pem
ssl_options.certfile = /etc/ssl/lst/client1.cert.pem
ssl_options.keyfile = /etc/ssl/lst/client1.key.pem
ssl_options.verify = verify_peer
ssl_options.versions.1 = tlsv1.3
ssl_options.versions.2 = tlsv1.2
ssl_options.ciphers.1 = TLS_AES_128_GCM_SHA256
ssl_options.ciphers.2 = TLS_AES_256_GCM_SHA384
ssl_options.ciphers.3 = TLS_CHACHA20_POLY1305_SHA256
ssl_options.ciphers.4 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.6 = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.8 = ECDHE-ECDSA-CHACHA20-POLY1305
ssl_options.ciphers.9 = ECDHE-RSA-CHACHA20-POLY1305
ssl_options.ciphers.10 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.11 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.12 = DHE-RSA-CHACHA20-POLY1305
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.fail_if_no_peer_cert = true

RabbitMQ Management Settings

management.ssl.port = 12345
management.ssl.versions.1 = tlsv1.2
management.ssl.cacertfile = /etc/ssl/lst/gdig2.crt
management.ssl.certfile = /etc/ssl/lst/client1.pem
management.ssl.keyfile = /etc/ssl/lst/client1.key

[michaelklishin]

RabbitMQ does not implement TLS or use OpenSSL (besides password hashing).

Your settings restrict supported versions and cipher suites. In addition to the TLS troubleshooting methodology suggested in the docs, try removing those
restrictions and re-introduce them one by one.

Specifically TLS 1.3 support is an all-or-nothing proposition
because it by design does not share any cipher suites with earlier versions.

[lukebakken]

@SpaceLineCrash

I have a new environment using RabbitMQ 3.12.4 with Erlang 26

Which packages are you using to install RabbitMQ and Erlang, and on what operating system?

Can you please provide a git repository that has the following in it:

• Complete Python test client code. All this has to do is open an AMQP connection using TLS / proxy protocol in the same way your real code is doing
• Complete HAproxy configuration
• Complete RabbitMQ configuration

I am asking for a git repo because then I know I'm using the same files as you.

What I can do is take the above information and set up an environment (probably using docker compose) to reproduce your issue. I am not going to set up a RabbitMQ cluster because that is not what is causing the issue here, I believe.

[SpaceLineCrash]

Hey Luke, we have a centralized HAProxy which carries most of our production environment. I will need some time to sanitize these items for you. Thank you for the reply and I will get this back to you shortly.

[SpaceLineCrash]

Hey Luke, I have a test consumer being created for me and I can update this ticket tomorrow with your requested information.

As an alternative and given this is already setup in my environment, would you be open to having a quick call to discuss/review configuration in our current environment? This would eliminate the need for you to build the vlab and we can do testing on my "production" environment.

[SpaceLineCrash]

Hello Luke, I added a private repo and added you and Michael. If there are any additional users, please let me know.

https://github.com/SpaceLineCrash/RabbitMQ-TestBed.git

[lukebakken]

Thank you. I will investigate this as time allows.