RabbitMQ > 3.11.10 - AMQPv1 authentication stopped working (rabbitmq_auth_backend_http + rabbitmq_amqp1_0)

[motmot80]

Describe the bug

Since RabbitMQ 3.11.10 it isnt possible to authenticate amqp-v1.0 clients with the http-auth-plugin.

• RMQ 3.11.9 with Erlang 25.3.2.3: working
• RMQ 3.10.25​ with Erlang 25.3.2.3: working
• RMQ 3.11.10 with Erlang 25.3.2.3: not working => auth_failure,"Refused"

It seems that the lastest RMQ versions may have broken the plugin combination

• rabbitmq_auth_backend_http
• rabbitmq_amqp1_0

Reproduction steps

1. Install RMQ > 3.11.10
2. Install REST-Mock as http auth plugin counter part replying with success
3. Activate and configure rabbitmq_auth_backend_http
4. Activate rabbitmq_amqp1_0
5. Connect AMQPv1.0 client (QPID-JMS, amqpnetlite, ...) with username & password for http-auth

Minimal java client:

org.apache.qpid.jms.JmsConnectionFactory jmsConnectionFactory = new org.apache.qpid.jms.JmsConnectionFactory("amqps://host:5671");
jmsConnectionFactory.setUsername("BOB");
jmsConnectionFactory.setPassword("PASSWORD");
jmsConnectionFactory.createConnection().createSession();

Expected behavior

Same behavior as RMQ 3.11.9 or 3.10.25 (< 3.11.10):

Authentication is working

Additional context

Maybe the Oauth 2 refactorings have broken the interface between the amqpv1 and the http-auth plugin.

Thanks in advance and best regards
Thomas

[michaelklishin]

Can you put together an executable example? Without the HTTP service, we have examples that are easy to modify to return any value necessary.

#6931 and #6909 seem highly relevant but we would rather not guess how do you set up the applications.

[michaelklishin]

Also, you haven't provided any logs.

[motmot80]

@michaelklishin

This is the log entry related to the error. I'll try to get a http trace between RMQ and the REST service.

2023-08-03 12:14:17.144122+02:00 [info] <0.1553.0> accepting AMQP connection <0.1553.0> (xxx:55379 -> xxx:5671)
2023-08-03 12:14:17.250740+02:00 [info] <0.1553.0> AMQP 1.0 connection <0.1553.0>: user 'BOB' authenticated
2023-08-03 12:14:17.307781+02:00 [warning] <0.1565.0> Closing session for connection <0.1553.0>:
2023-08-03 12:14:17.307781+02:00 [warning] <0.1565.0> {auth_failure,"Refused"}
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>   crasher:
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     initial call: rabbit_amqp1_0_session_process:init/1
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     pid: <0.1565.0>
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     registered_name: []
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     exception exit: {auth_failure,"Refused"}
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>       in function  gen_server2:init_it/6 (gen_server2.erl, line 600)
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     ancestors: [<0.1563.0>,<0.1559.0>,<0.1552.0>,<0.1551.0>,<0.677.0>,
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>                   <0.676.0>,<0.675.0>,<0.673.0>,<0.672.0>,rabbit_sup,
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>                   <0.254.0>]
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     message_queue_len: 0
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     messages: []
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     links: [<0.1563.0>]
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     dictionary: [{gen_server_call_timeout,130000}]
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     trap_exit: true
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     status: running
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     heap_size: 1598
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     stack_size: 28
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>     reductions: 2050
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>   neighbours:
2023-08-03 12:14:17.307917+02:00 [error] <0.1565.0>
2023-08-03 12:14:17.308356+02:00 [error] <0.1553.0> Error on AMQP 1.0 connection <0.1553.0> (running), channel 0:
2023-08-03 12:14:17.308356+02:00 [error] <0.1553.0> <<"Reader error: {error,\n               {{auth_failure,\"Refused\"},\n                {child,undefined,channel,\n                 {rabbit_amqp1_0_session_process,start_link,\n                  [{0,<0.1553.0>,<0.1564.0>,\n                    {user,<<\"TS-AUTH-TEST-AP\">>,[],\n                     [{rabbit_auth_backend_cache,\n                       #Fun<rabbit_auth_backend_http.0.56694127>}]},\n                    <<\"/\">>,1048568,\n                    {amqp_adapter_info,\n                     {0,0,0,0,0,65535,2752,18580},\n                     5671,\n                     {0,0,0,0,0,65535,2750,25225},\n                     55379,<<\"xxx:55379 -> xxx:5671\">>,\n                     {'AMQP',\"1.0\"},\n                     [{ssl,true},\n                      {ssl_protocol,'tlsv1.3'},\n
                         #Fun<rabbit_auth_backend_http.0.56694127>}]}]},\n                    <0.1560.0>}]},\n                 intrinsic,300000,worker,\n                 [rabbit_amqp1_0_session_process]}}}\n[{rabbit_amql\"},{line,314}]},\n {rabbit_amqp1_0_reader,recvloop,2,\n                        [{file,\"rabbit_amqp1_0_reader.erl\"},{line,117}]},\n {rabbit_reader,run,1,[{file,\"rabbit_reader.erl\"},{line,454}]},\n {rabbit_reader,start_connection,4,[{file,\"rabbit_reader.erl\"},{line,353}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]">>
2023-08-03 12:14:17.420419+02:00 [info] <0.1553.0> closing AMQP connection <0.1553.0> (xxx:55379 -> xxx:5671)

[motmot80]

When downgrading the RMQ Version to 3.11.09 the problem disappears.

[motmot80]

The config:

auth_backends.1   = internal
auth_backends.2   = cache

auth_cache.cached_backend = http
auth_cache.cache_ttl      = 600000
auth_cache.cache_refusals = false

auth_http.http_method   = get
auth_http.user_path     = https://xxx/user
auth_http.vhost_path    = https://xxx/vhost
auth_http.resource_path = https://xxx/resource
auth_http.topic_path    = https://xxx/topic

[michaelklishin]

OK, thanks. Can you please put together an executable example of the app we can run? Using the client you use to compare.

Once we have the details, we will file an issue with the specifics on your behalf.

[lukebakken]

@motmot80 I've begun setting up an environment to reproduce this issue. I have created a basic Java application here and will also use that repo for anything else I need to commit or share. Stay tuned.

[lukebakken]

I can reproduce this issue and have opened the following: #9043

Time for some git bisect action... Thank you @motmot80 for providing the version that works, and the one that doesn't.