Occassional timeouts of cache auth backend

[maciejharczuk]

I am seeing intermittent timeouts of (apparently) the cache auth backend (that wraps a http backend):

2023-11-06 04:20:15.506466+00:00 [error] <0.17902.7987> write access to exchange 'REDACTED' in vhost 'REDACTED' refused for user 'REDACTED', backend rabbit_auth_backend_cache returned an error: timeout
2023-11-06 04:20:15.506807+00:00 [error] <0.17902.7987> Channel error on connection <0.26836.7977> (REDACTED:34440 -> REDACTED:5671, vhost: 'REDACTED', user: 'REDACTED'), channel 17:
2023-11-06 04:20:15.506807+00:00 [error] <0.17902.7987> operation basic.publish caused a channel exception access_refused: write access to exchange 'REDACTED' in vhost 'REDACTED' refused for user 'REDACTED', backend rabbit_auth_backend_cache returned an error: timeout

Env details:
3 node cluster running on Azure, single region, but spread over 3 Availability Zones. The problem doesn't seem to be local to that region, same thing happens in a second cluster in a different region.
RabbitMQ version: 3.12.1
Erlang version: 25.3.2.3
OS: Ubuntu 22.04
Relevant rabbitmq.conf part:

auth_backends.1 = internal
auth_backends.2 = cache
auth_cache.cached_backend = http
auth_cache.cache_ttl = 60000
auth_http.user_path = https://REDACTED/v1/auth/user
auth_http.vhost_path = https://REDACTED/v1/auth/vhost
auth_http.resource_path = https://REDACTED/v1/auth/resource
auth_http.topic_path = https://REDACTED/v1/auth/topic
auth_http.http_method = post

The timeouts happen relatively rarely (maybe once every 1-2 days on each node), which makes them even more difficult to debug. I'm not actually convinced it's a bug in RabbitMQ, an intermittent network issue is also a possibility. I've a few questions to better understand what's happening:

• Is the timeout really from the cache, or does it propagate the error from the http backend that it caches?
• Is the ETS cache backend distributed? I.e. does retrieving data from it require a network call and has the potential to time out?
• Does Erlang set TCP Keepalives on sockets it creates? I thought it might be a firewall silently closing idle sockets, but the problem persists even with net.ipv4.tcp_keepalive_time = 120

[lukebakken]

Is the timeout really from the cache, or does it propagate the error from the http backend that it caches?

It is propagated from the HTTP backend. The HTTP call is timing out.

Is the ETS cache backend distributed? I.e. does retrieving data from it require a network call and has the potential to time out?

No

The HTTP auth backend uses the httpc Erlang library (https://www.erlang.org/doc/man/httpc.html), which does use HTTP keep alive by default.