TLSv1.2 connections to an LDAP server fail on Erlang 27.3 but succeed on 26.2.x

[stankeB]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.7

Erlang version used

27.2.x

Operating system (distribution) used

RHEL 8.10

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

$ rabbitmq-diagnostics cipher_suites --format openssl --all
Listing available cipher suites in OpenSSL format
TLS_AES_256_GCM_SHA384
TLS_AES_128_GCM_SHA256
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_CCM_SHA256
TLS_AES_128_CCM_8_SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-CCM
ECDHE-ECDSA-AES256-CCM8
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-CCM
ECDHE-ECDSA-AES128-CCM8
ECDH-ECDSA-AES256-GCM-SHA384
ECDH-RSA-AES256-GCM-SHA384
ECDH-ECDSA-AES128-GCM-SHA256
ECDH-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-DSS-AES256-GCM-SHA384
DHE-RSA-AES128-GCM-SHA256
DHE-DSS-AES128-GCM-SHA256
DHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
ECDH-ECDSA-AES256-SHA384
ECDH-RSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256
ECDH-ECDSA-AES128-SHA256
ECDH-RSA-AES128-SHA256
DHE-RSA-AES256-SHA256
DHE-DSS-AES256-SHA256
DHE-RSA-AES128-SHA256
DHE-DSS-AES128-SHA256
RSA-PSK-AES256-GCM-SHA384
RSA-PSK-AES256-CBC-SHA384
RSA-PSK-AES128-GCM-SHA256
RSA-PSK-AES128-CBC-SHA256
RSA-PSK-AES256-CBC-SHA
RSA-PSK-AES128-CBC-SHA
RSA-PSK-RC4-SHA
SRP-RSA-AES-256-CBC-SHA
SRP-DSS-AES256-CBC-SHA
SRP-RSA-AES-128-CBC-SHA
SRP-DSS-AES128-CBC-SHA
AES256-GCM-SHA384
AES256-SHA256
AES128-GCM-SHA256
AES128-SHA256
AES256-SHA
AES128-SHA
DES-CBC3-SHA
ECDHE-ECDSA-DES-CBC3-SHA
ECDHE-RSA-DES-CBC3-SHA
EDH-RSA-DES-CBC3-SHA
DHE-DSS-DES-CBC3-SHA
ECDH-ECDSA-DES-CBC3-SHA
ECDH-RSA-DES-CBC3-SHA
EDH-RSA-DES-CBC-SHA
DES-CBC-SHA
ECDHE-ECDSA-RC4-SHA
ECDHE-RSA-RC4-SHA
ECDH-ECDSA-RC4-SHA
ECDH-RSA-RC4-SHA
RC4-SHA
RC4-MD5
ECDHE-ECDSA-AES256-SHA
ECDHE-RSA-AES256-SHA
ECDH-ECDSA-AES256-SHA
ECDH-RSA-AES256-SHA
ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES128-SHA
ECDH-ECDSA-AES128-SHA
ECDH-RSA-AES128-SHA
DHE-RSA-AES256-SHA
DHE-DSS-AES256-SHA
DHE-RSA-AES128-SHA
DHE-DSS-AES128-SHA
$ rabbitmq-diagnostics erlang_version -q
Erlang/OTP 27.3
$ rabbitmq-diagnostics server_version -q
4.0.7
$ rabbitmq-diagnostics tls_versions -q
tlsv1.3tlsv1.2tlsv1.1tlsv1
$ rabbitmq-diagnostics list_feature_flags -q
name    state
classic_mirrored_queue_version  enabled
classic_queue_type_delivery_support     enabled
detailed_queues_endpoint        enabled
direct_exchange_routing_v2      enabled
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
feature_flags_v2        enabled
implicit_default_bindings       enabled
khepri_db       disabled
listener_records_in_ets enabled
maintenance_mode_status enabled
message_containers      enabled
message_containers_deaths_v2    enabled
quorum_queue    enabled
quorum_queue_non_voters enabled
rabbit_exchange_type_local_random       enabled
rabbitmq_4.0.0  enabled
restart_streams enabled
stream_filtering        enabled
stream_queue    enabled
stream_sac_coordinator_unblock_group    enabled
stream_single_active_consumer   enabled
stream_update_config_command    enabled
tracking_records_in_ets enabled
user_limits     enabled
virtual_host_metadata   enabled

Logs from node 1 (with sensitive values edited out)

{"time":"2025-03-24 17:45:15.847708+01:00","level":"info","msg":"LDAP CHECK: login for testuser","pid":"<0.1127.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.847746+01:00","level":"info","msg":"    LDAP connecting to servers: [\"my.ad\"]","pid":"<0.1127.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.877062+01:00","level":"info","msg":"    LDAP network traffic: Connect: \"my.ad\" failed {error,closed}\n","pid":"<0.1128.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.877226+01:00","level":"info","msg":"    LDAP connect error: {error,\"connect failed\"}","pid":"<0.462.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.877307+01:00","level":"info","msg":"    LDAP connecting to servers: [\"my.ad\"]","pid":"<0.1127.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.879843+01:00","level":"info","msg":"    LDAP network traffic: Connect: \"my.ad\" failed {error,closed}\n","pid":"<0.1133.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.879967+01:00","level":"info","msg":"    LDAP connect error: {error,\"connect failed\"}","pid":"<0.462.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.880046+01:00","level":"info","msg":"LDAP DECISION: login for testuser: {error,ldap_connect_error}","pid":"<0.1127.0>","domain":"rabbitmq.ldap"}
{"time":"2025-03-24 17:45:15.880090+01:00","level":"debug","msg":"User 'testuser' failed authentication by backend rabbit_auth_backend_ldap","pid":"<0.1127.0>","domain":"rabbitmq"}
{"time":"2025-03-24 17:45:15.880173+01:00","level":"warning","msg":"HTTP access denied: rabbit_auth_backend_ldap failed authenticating testuser: ldap_connect_error","pid":"<0.1127.0>","domain":"rabbitmq"}

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

listeners.ssl.default = 5671
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.cacertfile = /etc/rabbitmq/ssl/ca.pem
ssl_options.certfile = /etc/rabbitmq/ssl/cert.pem
ssl_options.keyfile = /etc/rabbitmq/ssl/key.pem
default_vhost = /
default_user = ***
default_pass = ***
management.listener.port = 15672
management.listener.ip   = 127.0.0.1
management.listener.ssl  = true
management.ssl.port = 15671
management.ssl.cacertfile = /etc/rabbitmq/ssl/ca.pem
management.ssl.certfile = /etc/rabbitmq/ssl/cert.pem
management.ssl.keyfile = /etc/rabbitmq/ssl/key.pem
log.file.formatter = json
classic_queue.default_version = 2
quorum_queue.property_equivalence.relaxed_checks_on_redeclaration = true
default_queue_type = quorum
max_message_size = 33554432

Steps to deploy RabbitMQ cluster

Three node cluster deployed. After update from Erlang 26.2.5.8 to 27.3 no login via rabbitmq_auth_backend_ldap is possible. Logs appear like pasted here.

Steps to reproduce the behavior in question

Configure RabbitMQ 4.0.7 (Erlang 27.3) to use rabbitmq_auth_backend_ldap against a MS AD. Changing rabbitmq_auth_backend_ldap.ssl_options.versions.1 to tlsv1.2.

advanced.config

[
 {rabbit, [
  {auth_backends, [rabbit_auth_backend_internal, rabbit_auth_backend_ldap]},
  {vm_memory_high_watermark, 0.8}
 ]},
  {rabbitmq_auth_backend_ldap, [
    {servers, ["my.ad"]},
    {port, 636},
    {use_ssl, true},
    {ssl_options, [{cacertfile, "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"},
                   {verify, verify_peer},
                   {versions, ['tlsv1.2']}]
    },
    {log, network},
    {timeout, 15000},
    {idle_timeout, 300000},
    {other_bind, {"CN=testbind,OU=Service,OU=Admins,DC=my,DC=ad", "***T"}},
    {dn_lookup_attribute,   "userPrincipalName"},
    {dn_lookup_base,        "dc=my,dc=ad"},
    {user_dn_pattern,       "${username}@my.ad"},
    {tag_queries, [
        {administrator, {in_group, "CN=testgroup,OU=Admin,OU=Groups,OU=Admins,DC=my,DC=ad"}}
        {management, {constant, true}}
    ]}
  ]}
].

Application code

no application involved

Kubernetes deployment file

no kubernetes deployment

What problem are you trying to solve?

Get LDAP authentication back to work under Erlang 27.3. without degrading security. With Erlang 26.2.5.8 it was working.

If we set rabbitmq_auth_backend_ldap.ssl_options.versions.1 to tlsv1.1, then it works with Erlang 27.3 as well. But that is less secure.

Openssl tells that tlsv1.2 handshake against MS AD LDAP port is working:

openssl s_client -verifyCAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem -connect my.ad:636 -tls1_2
CONNECTED(00000003)
depth=2 C = DE, O = MyCompany, CN = Root-CA
verify return:1
depth=1 DC = ad, DC = my, CN = Sub-CA
verify return:1
depth=0
verify return:1
---
Certificate chain
 0 s:
   i:DC = ad, DC = my, CN = Sub-CA
 1 s:DC = local, DC = ext, CN = Sub-CA
   i:C = DE, O = MyCompany, CN = Root-CA
---
Server certificate
-----BEGIN CERTIFICATE-----
***
-----END CERTIFICATE-----
subject=

issuer=DC = ad, DC = my, CN = Sub-CA

---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1:RSA+SHA512:ECDSA+SHA512
Shared Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:RSA+SHA512:ECDSA+SHA512
---
SSL handshake has read 3839 bytes and written 836 bytes
Verification: OK
---
New, TLSv1.2, Cipher is AES128-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES128-SHA256
    Session-ID: C21100007800BCF49830A21F5A2ADB037B6AEFE0E1F22C93755503E0BA84E0AE
    Session-ID-ctx:
    Master-Key: C39EA390137F63549D46EE8D2D11D9B4E91D517561D648F665E21EE4B612C9507CE0BDBED7CA0689F0AD65D6F411B424
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1742886148
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

And we have other applications that bind to MS AD LDAP port via TLSv1.2.

Thanks in advance for any help.

[michaelklishin]

As our community support policy very clearly states, we will not troubleshoot TLS for non-paying users.

RabbitMQ does not implement TLS or the LDAP client we use, both are parts of erlang/otp. Searching for issued that mention TLS or LDAP would be a good first step.

You already use the recommended tooling from the Troubleshooting TLS guide, so the key problem is putting together a minimal reproducible case for the Erlang maintainers to agree to investigate. Using openssl s_server, for example.

[lukebakken]

@stankeB you are missing this very important information in your issue report:

• Windows version and patch level
• Active Directory version

I realize that this used to work with an older version of Erlang, but it's still necessary info.

[lukebakken]

@stankeB - for what it's worth, I can't reproduce the issue you report using Erlang 27.3. Please see this repo which details what I tried:

https://github.com/lukebakken/rabbitmq-server-13610

[lukebakken]

I suggest seeing what version of OpenSSL ships with RHEL 8.10. The version of OpenSSL I'm using is 3.1.0. It doesn't matter that I'm running Erlang on Windows, because Erlang on Windows uses OpenSSL, just like Erlang on Linux.

[lukebakken]

Finally, I'm willing to assist a little more just to figure out what is going on. If you'd like to continue, start a discussion here:

https://github.com/lukebakken/rabbitmq-server-13610/discussions