RabbitMQ/oauth2 plugin doesn't pick up http_proxy and cannot fetch required keys #30

[avarf]

Describe the bug

We have a stack consist of:

• RabbitMQ (RMQ) with rabbitmq_auth_backend_oauth2 plugin enabled that fetches the keys from our Keycloak:

  {rabbitmq_auth_backend_oauth2, [
    {resource_server_id, <<"rabbitmq">>},
    {key_config, [
      {jwks_url, "${OIDC_BASE_URL}/protocol/openid-connect/certs"}
    ]}
  ]}

• Data publisher that publishes data via AMQP to RMQ
• Nginx that exposes an endpoint for STOMP
• Clients that consume data via STOMP (accessing the RMQ with their JWT tokens)

Everything works in our stack in normal conditions but when we run the stack behind a corporate proxy we face the issue.

RMQ receives the connections from STOMP clients but it cannot verify the tokens since it cannot get the key from our Keycloak (because of proxy it cannot reach out).

We solved this issue for other components of our stack by setting environment variables such as http_proxy, https_proxy and no_proxy.

I passed these variables to the RMQ as below but still, it cannot parse the JWT token:

  broker:
    image: docker.io/broker:latest
    environment:
      OIDC_BASE_URL: "https://my-keycloak.io/auth/realms/my-realm"
      http_proxy: "http://1.1.1.1:8080"
      https_proxy: "http://1.1.1.1:8080"
      no_proxy: "nginx,project_mgr,localhost,127.0.0.1"
      HTTP_PROXY: "http://1.1.1.1:8080"
      HTTPS_PROXY: "http://1.1.1.1:8080"
      NO_PROXY: "nginx,project_mgr,localhost,127.0.0.1"
    ports:
      - "5672:5672"
      - "15672:15672"
      - "15674:15674"
    networks:
      - local-network

RMQ logs:

2024-01-31 14:21:17.102107+00:00 [warn] <0.2042.0> Description: "Authenticity is not established by certificate path validation"
2024-01-31 14:21:17.102107+00:00 [warn] <0.2042.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-31 14:21:17.102107+00:00 [warn] <0.2042.0> 
2024-01-31 14:21:25.103685+00:00 [warn] <0.2036.0> STOMP login failed for user '': authentication failed
2024-01-31 14:21:25.103845+00:00 [erro] <0.2036.0> STOMP error frame sent:
2024-01-31 14:21:25.103845+00:00 [erro] <0.2036.0> Message: "Bad CONNECT"
2024-01-31 14:21:25.103845+00:00 [erro] <0.2036.0> Detail: "Access refused for user ''\n"
2024-01-31 14:21:25.103845+00:00 [erro] <0.2036.0> Server private detail: none

So what is the solution in this case? I couldn't find that much info on how to run RMQ behind corporate proxy set aside fetching key and working with JWT.

Reproduction steps

1. Run a RabbitMQ behind a proxy in a way that it can't access to the Keycloak
2. Set http_proxy for RMQ
3. Try to open a connection with the JWT token that you got from Keycloak

Expected behavior

RabbitMQ or Oauth2 plugin should pick up the http_proxy and no_proxy environment variables and use them to reach Keycloak and fetch the keys.

Additional context

No response

[michaelklishin]

RabbitMQ does not implement an HTTP API client, it relies on the one that comes with the runtime (that is, Erlang/OTP).

This is the first time that I recall someone asking for HTTP client proxy options support.

3.13.0 will include a separate (internally) OAuth 2 client #10012, so adding support for HTTP proxies will be more feasible. However, with just one request it's not going to be our team's priority.

You are welcome to contribute this feature once #10012 is merged. Here is the relevant portion of the Erlang standard library docs, it has an HTTP proxy example right on the first page.

[avarf]

I am a little confused. I am not asking for an "HTTP API client", there is already a client which is calling Keycloak. The oauth plugin (that exists in this repo) requires fetching the keys via http, and the only way to reach out to the internet is via using http_proxy while doing the call. I searched RMQ docs and I couldn't find a reference on how to set http_proxy to solve this issue so I passed them as env-vars (and I can confirm that from RMQ container I can successfully fetch the keys after adding the env-vars).

"it relies on the one that comes with the runtime (that is, Erlang/OTP)"
Exactly the problem is neither RMQ nor the oauth plugin is picking up these env-vars. It is not the language (Erlang) that reads the env-var but the library that you use. In Golang the http library automatically checks for env-vars such as "http_proxy" but in Python it is not like that and you have to explicitly read those env-vars and use them when doing the call.

I am not familiar with Erlang but based on your answer it seems that the http library in Erlang is not doing that and neither the RMQ nor the Oauth plugin doesn't check for http_proxy when doing the call.

[michaelklishin]

The OAuth 2 plugin uses an HTTP client to fetch the information it needs. It does not configure the client to use an HTTP proxy. It could. This is what I suggest that those who need this feature contribute, after #10012 is merged.

[michaelklishin]

In other news, please use Discussions for questions. This idea has next to no specifics attached to it, so this is discussion material.

For example, you try to use an environment variable and assume that RabbitMQ automatically will pick them up. But RabbitMQ uses environment variables only in a handful of places, all configuration happens via one or two configuration files, by design.

So such things must be clearly designed, explained in the proposal, and weighted against their alternatives. "setup http_proxy for RabbitMQ" is not a design that can be implemented.