RabbitMQ 3.13.0 changes default for LDAP SSL peer verification with no mention in the breaking changes

[evolvedlight]

Describe the bug

I use LDAP with SSL to authenticate users. I updated to 3.13.0 and couldn't login any more; only when I disabled the SSL peer verification it failed to startup and the error message is not too useful.

Reproduction steps

1. Have SSL secured SSL server. Have a custom CA on the SSL server (sorry, this is hard)
2. Connect to it from machine without that CA (ie, docker)
3. RabbitMQ connection fails

Expected behavior

Either it works as it did before, or something is put in the release notes. Either way, the error message could be better?

Additional context

Logs where it doesn't work

{"time":"2024-03-31 10:55:10.136895+00:00","level":"info","msg":"LDAP CHECK: login for aduser","pid":"<0.7277.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 10:55:10.137120+00:00","level":"info","msg":" LDAP filling template \"${username}@example.com\" with\n [{username,<<\"v002827\">>}]","pid":"<0.7277.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 10:55:10.137394+00:00","level":"info","msg":" LDAP template result: \"[email protected]\"","pid":"<0.7277.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 10:55:10.137688+00:00","level":"info","msg":" LDAP connect error: {error,\"connect failed\"}","pid":"<0.607.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 10:55:10.137809+00:00","level":"info","msg":"LDAP DECISION: login for aduser: {error,ldap_connect_error}","pid":"<0.7277.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 10:55:10.137905+00:00","level":"warning","msg":"HTTP access denied: rabbit_auth_backend_ldap failed authenticating aduser: ldap_connect_error","pid":"<0.7277.0>","domain":"rabbitmq"}

After increasing logging level:

{"time":"2024-03-31 15:07:55.176074+00:00","level":"info","msg":"LDAP CHECK: login for aduser","pid":"<0.8005.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.176297+00:00","level":"info","msg":" LDAP filling template \"${username}@vontobel.com\" with\n [{username,<<\"v002827\">>}]","pid":"<0.8005.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.176525+00:00","level":"info","msg":" LDAP template result: \"[email protected]\"","pid":"<0.8005.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.176569+00:00","level":"info","msg":" LDAP connecting to servers: [\"ads.example.com\"]","pid":"<0.8005.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.176953+00:00","level":"info","msg":" LDAP network traffic: Connect: \"ads.example.com\" failed {error,\n {options,incompatible,\n [{verify,verify_peer},\n {cacerts,\n undefined}]}}\n","pid":"<0.8006.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.177150+00:00","level":"info","msg":" LDAP connect error: {error,\"connect failed\"}","pid":"<0.571.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.177229+00:00","level":"info","msg":"LDAP DECISION: login for aduser: {error,ldap_connect_error}","pid":"<0.8005.0>","domain":"rabbitmq.ldap"}
{"time":"2024-03-31 15:07:55.177317+00:00","level":"warning","msg":"HTTP access denied: rabbit_auth_backend_ldap failed authenticating aduser: ldap_connect_error","pid":"<0.8005.0>","domain":"rabbitmq"}

The temporary settings I use:

 auth_backends.1 = internal
 auth_backends.2 = ldap
 auth_ldap.servers.1 = ads.vt.ch
 auth_ldap.port = 636 

 auth_ldap.use_ssl   = true
 auth_ldap.ssl_options.hostname_verification = none

 # Temp
 auth_ldap.ssl_options.verify = verify_none
 auth_ldap.ssl_options.fail_if_no_peer_cert = false

 auth_ldap.log = network

[lukebakken]

This is actually due to a change in Erlang:

https://www.erlang.org/news/164#ssl

Yes, we should document this better. Although, this change in Erlang's behavior hasn't resulted in nearly as many reports from RabbitMQ users as I expected it would.

cc @michaelklishin @pstack2021

[michaelklishin]

I'm editing the LDAP guide to mention the new default.

[michaelklishin]

RabbitMQ does not implement TLS, Erlang/OTP does. Indeed starting with Erlang 26, some TLS defaults have changed. Whether that's a bug or the previous behavior was a bug, depends on who you ask.

In some cases we override the default (e.g. #8547) because setting up peer verification may or may not be practical in some contexts where TLS is used.

I'm not sure why you consider these settings to be "temporary", if you choose to not enable peer verification, that's your call. Some may argue that the default should be "peer verification enabled."

[michaelklishin]

Both 3.13.0 release notes and LDAP client TLS settings now mention the default change.

FWIW most of the industry believe this is a better default, regardless of whether it drives any meaningful change in actual peer verification adoption. So in most places in RabbitMQ it will likely remain exactly as it is in the runtime.

[evolvedlight]

Thanks very much! I for sure agree that the new default makes much more sense.