Ranch listener exit with reason badarg leaving k8s pod in not ready state

[jaroslaw-bochniak]

Context

After the upgrade to 3.11.0 (running in k8s) we occasionally observe that restarting the stateful set via k rollout restart sts/mq-server result in one of the pods ending up with not ready state. After deleting the pod, it is able to start up normally. It may be related to observing the cluster state via management UI during the restart.

Full log with error in the attachment.
mq-ai-server-1.txt

Server configuration:

  rabbitmq:
    additionalPlugins:
      - rabbitmq_federation
      - rabbitmq_federation_management
      - rabbitmq_shovel
      - rabbitmq_shovel_management
      - rabbitmq_prometheus
    additionalConfig: |
      channel_max = 500
      classic_queue.default_version = 2
      cluster_partition_handling = autoheal
      consumer_timeout = 900000
      disk_free_limit.relative = 1.0
      raft.segment_max_entries = 10000
      raft.wal_max_batch_size = 4096
      raft.wal_max_size_bytes = 16000000
      vm_memory_high_watermark.relative = 0.7
    envConfig: |
      RABBITMQ_FEATURE_FLAGS=quorum_queue,implicit_default_bindings,virtual_host_metadata,maintenance_mode_status,user_limits

[michaelklishin]

How was this cluster provisioned?

[jaroslaw-bochniak]

Cluster was provisioned with the rabbitmqoperator/cluster-operator in version 1.14.0

[michaelklishin]

That means the HTTP API call is something specific to your environment. The Operator uses a TCP check for port 5672 by default.

[jaroslaw-bochniak]

I think that this call could come from management UI from the browser on 15672 port as it performs calls to /api/overview when setting "Refresh every x seconds" is selected.

[michaelklishin]

We haven't upgraded Ranch or changed anything around TCP listeners in 3.11. Here is the relevant trace:

mq-ai-server-1 rabbitmq �[38;5;160m2022-10-03 13:31:49.197963+00:00 [error] <0.829.0> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15672}, connection process <0.829.0>, stream 19 had its request process <0.916.0> exit with reason badarg and stacktrace [{erlang,'++',[{badrpc,{'EXIT',{badarg,[{ets,match_object,[rabbit_listener_ets,'_'],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{ets,tab2list,1,[{file,"ets.erl"},{line,771}]}]}}},[{listener,'[email protected]',http,"mq-ai-server-2.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},15672,[{cowboy_opts,[{sendfile,false}]},{port,15672}]},{listener,'[email protected]','http/prometheus',"mq-ai-server-2.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},15692,[{cowboy_opts,[{sendfile,false}]},{port,15692},{protocol,'http/prometheus'}]},{listener,'[email protected]',clustering,"mq-ai-server-2.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},25672,[]},{listener,'[email protected]',amqp,"mq-ai-server-2.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},5672,[{backlog,128},{nodelay,true},{linger,{true,0}},{exit_on_close,false}]},{listener,'[email protected]',http,"mq-ai-server-1.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},15672,[{cowboy_opts,[{sendfile,false}]},{port,15672}]},{listener,'[email protected]','http/prometheus',"mq-ai-server-1.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},15692,[{cowboy_opts,[{sendfile,false}]},{port,15692},{protocol,'http/prometheus'}]},{listener,'[email protected]',clustering,"mq-ai-server-1.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},25672,[]},{listener,'[email protected]',amqp,"mq-ai-server-1.mq-ai-nodes.rabbitmq-system.svc.cluster.local",{0,0,0,0,0,0,0,0},5672,[{backlog,128},{nodelay,true},{linger,{true,0}},{exit_on_close,false}]}]],[{error_info,#{module => erl_erts_errors}}]},{lists,append,1,[{file,"lists.erl"},{line,155}]},{rabbit_mgmt_wm_overview,listeners,0,[{file,"rabbit_mgmt_wm_overview.erl"},{line,111}]},{rabbit_mgmt_wm_overview,to_json,2,[{file,"rabbit_mgmt_wm_overview.erl"},{line,66}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1575}]},{cowboy_rest,set_resp_body,2,[{file,"src/cowboy_rest.erl"},{line,1464}]},{cowboy_rest,upgrade,4,[{file,"src/cowboy_rest.erl"},{line,284}]},{cowboy_stream_h,execute,3,[{file,"src/cowboy_stream_h.erl"},{line,300}]}]�[0m

So something calls GET /api/overview and one of the objects in the ETS table is not yet ready early on boot. I don't think we can address that, so at best we can make rabbit_mgmt_wm_overview:listeners/0 more defensive and this cluster can be changed to use a different readiness probe.

@lhoguin have you seen something like this from Cowboy users?