Management UI OAuth 2.0 authentication not possible after enabling Khepri

[bast42]

Describe the bug

After enabling Khepri in RabbitMQ (version 4.1) via the Management UI, I am redirected to the login page displaying the message "Not_Authorized". The previously working OAuth 2.0 login option no longer works (it redirects to a blank page) and disappears when the login page is reloaded. The basic authentication option still works as expected after the migration to Khepri.

Restarting RabbitMQ — either by restarting the container or via rabbitmqctl stop_app and rabbitmqctl start_app— resolves the issue, and the OAuth 2.0 option on the login page becomes available and functional again.

Note: When using RabbitMQ 4.2 RC (container image tag 4.2-rc-management) with Khepri as the default metadata store, the OAuth 2.0 login option is not shown after the initial startup. Again, a restart is required to resolve the issue.

Reproduction steps

1. Set up authentication via auth0.com as it is described in this example.
2. Start RabbitMQ container with the corresponding rabbitmq.conf (see additional context):

podman run -d --rm \
  --hostname rabbitmq \
  --name rabbitmq \
  -p 5672:5672 -p 15672:15672 \
  -v rabbitmq-data:/var/lib/rabbitmq \
  -v ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf \
  rabbitmq:4.1-management

3. Open RabbitMQ Management UI (http://localhost:15672/) and log in via OAuth2.
4. Naviage to Admin > Feature Flags and enable Khepri.

After a few seconds, you are redirected to the login page. The OAuth2 login option is still visible but no longer works.

To resolve the problem restart RabbitMQ (podman restart rabbitmq or rabbitmqctl stop_app & rabbitmqctl start_app).

Expected behavior

After a successful migration to Khepri, the OAuth2 login option remains available on the login page and works as expected.

Additional context

rabbitmq.conf:

auth_backends.1 = rabbit_auth_backend_oauth2
auth_backends.2 = rabbit_auth_backend_internal

log.console.level = debug

management.oauth_enabled = true
management.oauth_client_id = {Client ID}
management.oauth_scopes = openid profile rabbitmq.tag:administrator
management.oauth_authorization_endpoint_params.audience = rabbitmq
management.oauth_token_endpoint_params.audience = rabbitmq
management.oauth_disable_basic_auth = false

auth_oauth2.resource_server_id = rabbitmq
auth_oauth2.issuer = {Domain}
auth_oauth2.https.hostname_verification = wildcard

[michaelklishin]

@bast42 running plugins won't be notified of the metadata store change, so yes, a node restart may be necessary depending on what a plugin does.

We have several Selenium-based test suites that were originally introduced for testing certain OAuth 2 workflows. They suggest that a new 4.2 beta (there is no such thing as an 4.2 RC at the moment, regardless of how the community Docker image names its tags) is fine.

@MarcialRosales FYI.

[bast42]

@michaelklishin thanks for the answer. I would suggest adding a hint to the migration guide indicating that some plugins may require a node restart after Khepri is enabled.

The container image rabbitmq:4.2-rc-management from Docker Hub corresponds to RabbitMQ version 4.2.0-beta.1.

In this version, the OAuth2 option wasn't available for me on the login page after the initial startup, but restarting the node resolved the issue.

[MH-17]

@michaelklishin with the RabbitMQ 4.2 beta, the Khepri metadata store is enabled by default. If I configure OAuth 2.0 and start RabbitMQ 4.2 beta for the first time, the OAuth 2.0 login won't work initially but only after restarting RabbitMQ. In this example, RabbitMQ uses Khepri from the very start. Therefore, I would expect that a login with OAuth 2.0 is possible also on the very first start to support automated deployments (e.g., Helm) with OAuth 2.0 configurations.

[michaelklishin]

@MH-17 then why hasn't anyone else, including someone who works on OAuth 2 most of their time on our team, hasn't noticed it before? We will try to reproduce.

[michaelklishin]

@bast42 @MH-17 a member of our team cannot reproduce any issues with OAuth 2 logins on a freshly booted 4.2 (main) node that enables Khepri automatically.

Next up: testing an upgrade from 4.1.

[MarcialRosales]

I have followed the reproduction steps against v4.1.x branch (build: RabbitMQ 4.1.3+32.g7413939.dirty) and i have not seen any issues. I have tested using keycloak. I start a brand new server. I successfully log in with rabbit_admin user using oauth2. I enable feature flag khepri_db. I am still logged in, no problems. I log out. I am redirected to the initial login page with the single button. I click on the button, I am redirected to keycloak to log in again. I am back again to RabbitMQ successfully logged in. Regardless, I am currently adding this scenario to our selenium test suite for v4.1.x branch.
btw, I am always testing against RabbitMQ server built from source not from a docker image although I do not expect any difference. In any case, our selenium tests cases run against locally built docker images.

[MH-17]

I compared the behavior of different RabbitMQ images from the Docker community again, specifically rabbitmq:4.1.3-management and rabbitmq:4.2.0-beta.1-management. I noticed that starting from version 4.2, you explicitly need to enable the rabbitmq_auth_backend_oauth2 plugin for OAuth 2.0. With earlier versions, OAuth 2.0 worked even without explicitly enabling this plugin (at least with the image from the Docker community).

I’m aware that this was an incorrect use of RabbitMQ and that the OAuth 2.0 plugin must be enabled. However, since we didn’t have to enable the plugin before, we assumed during the tests with 4.2.0-beta.1 that the issue was caused by the switch to Khepri. To be honest, I still do not get, why OAuth 2.0 works with version 4.1.3 if the plugin is disabled. Anyway, thanks for your support and quick replies.

RabbitMQ 4.1.3 (with OAuth 2.0 config and default plugins)

• Default plugins are used, not customized: [rabbitmq_management,rabbitmq_prometheus].
• image: rabbitmq:4.1.3-management
• Result: Login with OAuth 2.0 is possible on the first boot of a clean install ✅

RabbitMQ 4.2.0-beta.1 (with OAuth 2.0 config and default plugins)

• Default plugins are used, not customized: [rabbitmq_management,rabbitmq_prometheus].
• image: rabbitmq:4.2.0-beta.1-management
• Result: Login with OAuth 2.0 is NOT possible on the first boot of a clean install but works after restarting the node ❌

RabbitMQ 4.2.0-beta.1 (with OAuth 2.0 config, with OAuth 2.0 plugin)

• Plugins are customized to: [rabbitmq_management,rabbitmq_prometheus,rabbitmq_auth_backend_oauth2].
• image: rabbitmq:4.2.0-beta.1-management
• Result: Login with OAuth 2.0 is possible on the first boot of a clean install ✅

[michaelklishin]

So enabling the plugin properly (before it is used, via preconfiguration) makes it work as expected.