RabbitMQ with oAuth2 fails to connect with bad username and password

[sandhraprakash]

RabbitMQ as MQTT broker with oAuth2 plugin is not accepting token being passed. https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_auth_backend_oauth2

The client is spring boot based. We use paho client to establish connection.

We are running RabbitMQ broker on Kubernetes using cluster operator. We are trying to have jwt token based access control along with client certificate based authentication and topic permission based authorization.

following is the yaml to configure and deploy RabbitMQ:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: definition
spec:
  image: rabbitmq:3.9.13
  rabbitmq:
    additionalPlugins:
      - rabbitmq_mqtt
      - rabbitmq_management
      - rabbitmq_auth_mechanism_ssl
      - rabbitmq_auth_backend_oauth2
    additionalConfig: |
      ssl_options.fail_if_no_peer_cert = true
      ssl_options.verify = verify_peer
      log.connection.level = debug
      mqtt.listeners.ssl.default = 8883
      mqtt.listeners.tcp.default = 1883
      auth_mechanisms.1 = EXTERNAL
      ssl_cert_login_from = common_name
      mqtt.ssl_cert_login = true
      auth_backends.1 = rabbit_auth_backend_oauth2
      auth_backends.2 = rabbit_auth_backend_internal
      auth_oauth2.https.peer_verification = verify_peer
      auth_oauth2.https.depth = 1
      auth_oauth2.https.fail_if_no_peer_cert = true
      auth_oauth2.https.hostname_verification = wildcard
      auth_oauth2.algorithms.1 = HS256
      auth_oauth2.algorithms.2 = RS256
    advancedConfig: |
      [
        {rabbitmq_auth_backend_oauth2, [
          {resource_server_id, <<"rabbitmq">>},
          {key_config, [
          {jwks_url, <<"http://localhost:8080/realms/myrealm/protocol/openid-connect/certs">>}
          ]}
        ]} 
      ].    
  tls:
    disableNonTLSListeners: false
    secretName: tls-secret
    caSecretName: ca-secret

2. In Keycloak following was done

• Created a realm
• Created a client in the realm
• Set Access type of client as confidential to obtain clientId and client secret
• Set the Authorization Enabled to true
• Set Root URL
• Switch to Keys tab to set Use JWKS URL to true and paste the jwks uri obtained from the realm endpoints.
• Create Client Scopes in the format

   rabbitmq.configure:*/*/*
   rabbitmq.read:*/*/* 
   rabbitmq.write:*/*/*

• Add the above client scopes to the client
• Switch to mapper to create a new mapper. Select mapper type as audience. Set name as rabbitmq-aud. and set Included Custom Audience as <resource_server_id>. Enable Add to access token
• Create a User
• Obtain access token using the keycloak endpoint and validated its contents as per document in jwt.io.
• payload part of token decodes as below

  "iat": 1647343597,
  "jti": "9980b9bb-7491-43b3-98c2-0bf55a15fc11",
  "iss": "http://localhost:8090/auth/realms/myrealm",
  "aud": [
    "rabbitmq",
    "account"
  ],
  "sub": "9cf08a3e-e5cb-4d41-ad06-09ef07ad8af2",
  "typ": "Bearer",
  "azp": "subscriber-client",
  "session_state": "5f6ac687-164e-4310-9199-f87e45af2d4f",
  "acr": "1",
  "allowed-origins": [
    "http://localhost:9999/"
  ],
  "realm_access": {
    "roles": [
      "default-roles-subscriber",
      "offline_access",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid rabbitmq.read:*/*/* profile email rabbitmq.write:*/*/* rabbitmq.configure:*/*/*",
  "sid": "5f6ac687-164e-4310-9199-f87e45af2d4f",
  "email_verified": false,
  "preferred_username": "testuser"
}

3. In Client App, token was obtained through the keycloak api. And passed to the paho client in place of password. we were trying to establish using non tls approach. Token was passed in place of password. we also tried appending Bearer before token. Tried setting existing username, non existing username and by not setting username.
   Result
   In all the above cases the MQTTException stating Bad username and password was thrown from connect api.
   Logs of RabbitMQ revealed that the token passed was checked against password field of the username passed. if a username not registered with RabbitMQ was passed it would throw the same exception. if username was not set the default username guest was taken internally and log would state that.
   Please Help! we have tried everything as per documentation. and still getting the same result
   And also how to achieve the same with SSL?

[lukebakken]

Thank you for a pretty comprehensive set of information. It's smart to not try to also use TLS at this point in time. In order for us to assist you, we will need debug level logs from RabbitMQ. To do that, add the following to additionalConfig:

log.console.level = debug

In addition, remove the following from additionalConfig for now:

auth_mechanisms.1 = EXTERNAL
ssl_cert_login_from = common_name
mqtt.ssl_cert_login = tru

Please do the above and re-try your scenario. Afterward, attach your complete log files from all cluster nodes to your response. Thanks.

[sandhraprakash]

I have done the config changes as you suggested. and we set user that exists in Rabbitmq as username. attached is the log file.

Thankyou!
rabbitmqlog.txt

[lukebakken]

Please see the following error in your log file:

Authentication using an OAuth 2/JWT token failed: {error,
                                                   {failed_connect,
                                                    [{to_address,
                                                      {"localhost",8080}},
                                                     {inet,
                                                      [inet],
                                                      econnrefused}]}}

Your OAuth2 server is not available at localhost:8080. "Connection refused" means that there is no process listening on that port, or there may be something blocking the connection like a firewall.

[sandhraprakash]

rabbitmq is trying to use the jwks uri we provided and connection to that is failing?
We were able generate the token from localhost:8080 (Keycloak) programmatically.
Keycloak is running in the docker. Rabbitmq in the kubernetes cluster and client app is running locally.

[lukebakken]

rabbitmq is trying to use the jwks uri we provided and connection to that is failing?

Yes, the log file clearly shows that RabbitMQ can't connect to localhost:8080, most likely because nothing is listening there.

[sandhraprakash]

Thanks for helping us figure it out. We deployed keycloak with https and then it was able to connect

Our final yaml looks like this

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: definition
spec:
  image: rabbitmq:3.9.13
  rabbitmq:
    additionalPlugins:
      - rabbitmq_mqtt
      - rabbitmq_management
      - rabbitmq_auth_mechanism_ssl
      - rabbitmq_auth_backend_oauth2
    additionalConfig: |
      ssl_options.fail_if_no_peer_cert = true
      ssl_options.verify = verify_peer
      log.connection.level = debug
      mqtt.listeners.ssl.default = 8883
      mqtt.listeners.tcp.default = 1883
      log.console.level = debug
      auth_backends.1 = rabbit_auth_backend_oauth2
      auth_backends.2 = rabbit_auth_backend_internal
      ssl_cert_login_from = common_name
      mqtt.ssl_cert_login = true
      auth_oauth2.https.peer_verification = verify_peer
      auth_oauth2.https.depth = 1
      auth_oauth2.https.fail_if_no_peer_cert = true
      auth_oauth2.https.hostname_verification = wildcard
    advancedConfig: |
      [
        {rabbitmq_auth_backend_oauth2, [
          {resource_server_id, <<"rabbitmq">>},
          {key_config, [
          {jwks_url, <<"https://keycloak.develop.com/auth/realms/subscriber/protocol/openid-connect/certs">>}
          ]}
        ]} 
      ].    
  tls:
    disableNonTLSListeners: true
    secretName: tls-secret
    caSecretName: ca-secret

[mahadevikodavati]

Hallo Team,

I need small help..i would like authenticate rabbitmq with oauth2.0 with jwks_url. i am getting below error

My configuration is like below

advanced.config

[
{rabbit, [
{auth_backends, [rabbit_auth_backend_oauth2]},
{log, [
{console, [
{level, debug},
{enabled, true}
]}
]}
]},
{rabbitmq_management, [
{oauth_enabled, true},
{oauth_client_id, "rabbitmq-client-code"},
{oauth_client_secret, "X2ZqbZYKHq7WP8vsStr23jx6mtukt14g"},
{oauth_provider_url, "https://xxxxxxx.xxxx"}
]},
{rabbitmq_auth_backend_oauth2, [
{resource_server_id, <<"rabbitmq">>},
{key_config, [
{jwks_url, <<"https://rxxxxxxxx/openapi/v3/auth/realms/RMV/protocol/openid-connect/certs">>}
]}

]}
].

rabbitmq.conf

loopback_users.guest = false
log.console = true
load_definitions = /etc/rabbitmq/definitions.json
auth_oauth2.resource_server_id = rabbitmq
auth_oauth2.jwks_url = https://XXXXXXX/openapi/v3/auth/realms/RMV/protocol/openid-connect/certs

auth_oauth2.https.peer_verification = verify_peer
auth_oauth2.https.depth = 5
auth_oauth2.https.fail_if_no_peer_cert = true
auth_oauth2.https.hostname_verification = wildcard
auth_oauth2.algorithms.1 = RS256

ssl_options.fail_if_no_peer_cert = true
ssl_options.verify = verify_peer
log.connection.level = debug

log.console.level = debug
auth_backends.1 = rabbit_auth_backend_oauth2

Docker file
FROM rabbitmq:3.11.0-management-alpine
COPY definitions.json /etc/rabbitmq
COPY rabbitmq.conf /etc/rabbitmq
COPY advanced.config /etc/rabbitmq

COPY enabled_plugins /etc/rabbitmq
COPY conf /conf

RUN ls /etc/rabbitmq
RUN cat /etc/rabbitmq/rabbitmq.conf
RUN cat /etc/rabbitmq/advanced.config
RUN cat /etc/rabbitmq/definitions.json
RUN cat /etc/rabbitmq/enabled_plugins

Kindly help me

[sudhanshujoshi23]

Were you able to find the solution for the issue that you are facing? I am currently going through the same issue and it would be really helpful if you can post the root cause and the suggested approach to fix this.

Thanks

[MarcialRosales]

If you are getting the error message shown in the attached management ui screenshot please review your configured oauth_provider_url. Make sure that <oauth_provider_url>/.well-known/openid-configuration returns a json payload. It has to be a valid OIDC Discovery endpoint.