Query with rabbitmq configuration files while moving from version 3.11.x to 3.13.2

[devfinwiz]

1. I am moving to rabbit 3.13.2 and erlang 26.2.3.
2. While I do so, I am facing the following error:

47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0> EXTERNAL login refused: rabbit_auth_backend_http failed authenticating certmaster.0: {failed_connect,"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                               [{to_address,"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                 {""certauth"","
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                  9000}},"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                {inet,"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                 [inet],"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                 {option,"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                  server_only,"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.942940964+10:00  2024-07-18 11:32:03.942694+00:00 [error] <0.58810.0>                                                                  fail_if_no_peer_cert}}]}"
47bc8eaba2053a41b45995be785dd90a622b50c421b5441fb83bf1a1ded084da: "2024-07-18T21:32:03.944056755+10:00  2024-07-18 11:32:03.943800+00:00 [info] <0.58810.0> closing AMQP connection <0.58810.0> ([fd00::1:8:3b8]:38498 -> [fd00::1:8:3b7]:5671)"

3. Here is my rabbitmq.conf:

listeners.tcp                           = none
listeners.ssl.default                   = 5671

mqtt.allow_anonymous                    = false

ssl_options.client_renegotiation        = false
ssl_options.cacertfile                  = /rabbitmq/keys/cacert.pem
ssl_options.certfile                    = /rabbitmq/keys/server.cert.pem
ssl_options.keyfile                     = /rabbitmq/keys/server.key.pem
ssl_options.verify                      = verify_peer
ssl_options.fail_if_no_peer_cert        = true
ssl_options.versions.1                  = tlsv1.2
ssl_options.ciphers.1                   = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.2                   = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3                   = ECDHE-ECDSA-AES256-SHA384
ssl_options.ciphers.4                   = ECDHE-RSA-AES256-SHA384
ssl_options.ciphers.5                   = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.6                   = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.7                   = ECDHE-ECDSA-AES128-SHA256
ssl_options.ciphers.8                   = ECDHE-RSA-AES128-SHA256
ssl_options.honor_cipher_order          = true
ssl_options.honor_ecc_order             = true
ssl_handshake_timeout                   = 30000

ssl_options.client_renegotiation        = false
ssl_options.secure_renegotiate          = true

auth_mechanisms.1                       = PLAIN
auth_mechanisms.2                       = EXTERNAL

heartbeat                               = 120

vm_memory_high_watermark.relative       = 0.75
max_message_size                        = 209715200
ssl_cert_login_from                     = common_name

auth_backends.1                         = http

log.console                             = true
log.file                                = false
consumer_timeout                        = 25200000

4. Here is my advanced.config:

%% advanced.config
[
  {rabbitmq_auth_backend_http, [
    {ssl_options, [
      {verify, verify_peer},
      {fail_if_no_peer_cert, true},
      {cacertfile, "/rabbitmq/keys/cacert.pem"},
      {certfile, "/rabbitmq/keys/server.cert.pem"},
      {keyfile, "/rabbitmq/keys/server.key.pem"}
    ]},
    {http_method, post},
    {user_path, <<"https://certauth:9000/auth/user">>},
    {vhost_path, <<"https://certauth:9000/auth/vhost">>},
    {resource_path, <<"https://certauth:9000/auth/resource">>},
    {topic_path, <<"https://certauth:9000/auth/topic">>}
  ]},
  {rabbit, [
    {consumer_timeout, undefined}
  ]}
].

5. What have I done to debug this issue?

• Checked my certificates (their content/role) and the path to my certificates mentioned in conf files.
• certauth and rabbit have a common CA.
• There is no issue on the certificates end as manually hitting the /auth/user endpoint produces appropriate response. This confirms that no issue lies with certauth and it actually works. I made sure to pass the appropriate certificates in the curl command.
• Logs (outbound connection where rabbitmq is the client and certauth is the server):

curl --cert /rabbitmq/keys/server.cert.pem --key /rabbitmq/keys/server.key.pem --cacert /rabbitmq/keys/cacert.pem https://certauth:9000/auth/user?username=certmaster.0

allow (this is the response produced)

6. The problem is that /auth/user is not being hit on it's own when authentication is to be carried out. I have confirmed the same from certauth container logs. It indicated absence of request to /auth/user endpoint. I am suspecting some issue with my configurations itself.

Extra information:

1. When we have verify_none as ssl_options (verify) instead of verify_peer in advanced.config, everything works fine without any issues. Hence, the issue lies when rabbitmq is the client and certauth is the server i.e during an outbound connection.

2. RabbitMQ is supposed to hit endpoints (mentioned in advanced.config) to authenticate itself and check permissions for resources and actions.

3. These endpoints are hit during connection setup, resource access, and message operations when RabbitMQ is acting as a client.

[michaelklishin]

Erlang 26's TLS implementation enforces peer verification by default, including for the HTTP client.

If you don't use peer certificate chain verification with the HTTP authN/authZ plugin, you need to disabled using the settings introduced in #11344. I suggest waiting for 3.13.5 to ship in the next few days.

[michaelklishin]

The problem is that /auth/user is not being hit on it's own when authentication is to be carried out

yes it is being hit, and the connection fails before any HTTP traffic is transferred because TLS peer certificate chain verification is enabled by default and it fails in your environmet.

Take a tcpdump capture and inspect it with Wireshark instead of container logs. Container logs won't tell you anything about inbound TCP connections in all likelihood.

[gomoripeti]

the bellow error

{option, server_only, fail_if_no_peer_cert}

hints at that fail_if_no_peer_cert is a server-side only TLS option and since Erlang 26 it does not allow it as a client-side option (earlier it was just ignored)

try removing it from your advanced config's rabbitmq_auth_backend_http section

(see for example rabbitmq/rabbitmq-website#1933)

[devfinwiz]

So, verify ssl option will have verify_peer as the value but fail_if_no_peer_cert should be removed from advanced.config and things should be working fine? Did I interpret your response correctly?

[devfinwiz]

@gomoripeti By any chance, could you provide me the reference to Erlang 26 source code snippet where this setting (fail_if_no_peer_cert is no more allowed) was altered?

[gomoripeti]

don't take my response offensive, but I don't know why you ask for this information

• if you know Erlang language you can search for keywords server_only and fail_if_no_peer_cert in the erlang/otp git repo
• if not (which is fine) then I don't know why source code is relevant for you

Either way

• I searched for the terms in the erlang repo https://github.com/search?q=repo%3Aerlang%2Fotp+fail_if_no_peer_cert+server_only&type=code
• that is how I found ssl.erl - then I switched to blame view
• this is how I found line 1724 in ssl.erl in commit cd00692e which is a big refactoring of ssl option handling that includes this change as well. As Michail also referred to it, Erlang 26 included many improvements in the TLS implementation which made defaults more secure and option handling more strict.

[devfinwiz]

don't take my response offensive, but I don't know why you ask for this information

• if you know Erlang language you can search for keywords server_only and fail_if_no_peer_cert in the erlang/otp git repo
• if not (which is fine) then I don't know why source code is relevant for you

Either way

• I searched for the terms in the erlang repo https://github.com/search?q=repo%3Aerlang%2Fotp+fail_if_no_peer_cert+server_only&type=code
• that is how I found ssl.erl - then I switched to blame view
• this is how I found line 1724 in ssl.erl in commit cd00692e which is a big refactoring of ssl option handling that includes this change as well. As Michail also referred to it, Erlang 26 included many improvements in the TLS implementation which made defaults more secure and option handling more strict.

I am a total newbie in this area and as my first ever task to handle rabbitmq configuration, I was supposed to upgrade it.
I could gather all that's related and restricted to only rabbitmq's configuration files but since I am relatively new to this stuff, I am not very familiar, comfortable with erlang yet.

Also, no offense intended but it would have been decent and kind of you to simply drop a response even if you did not want to offer entended help. I was away from my laptop due to CrowdStrike global outage and thought of saving some time by using Github mobile to gather necessary information via this forum itself since you guys are total experts with these repositories.

Anyways, thanks!

[devfinwiz]

Thank you so much @gomoripeti for letting me know about this. Issue resolved.