Help debugging strange node crashes with quorum queues

[tkoft]

I've got a cluster deployed with three nodes that went into a weird state this morning when there shouldn't have been any traffic on the cluster at all.

Having a hard time figuring out what's going on at all, but it seems to be related to quorum queues? Using RabbitMQ 3.11.18 management image deployed with kubernetes cluster operator.

Here's a dump of the logs from the first crash, but it appears the node kept retrying and crashing (with successful client connections getting logged in between). The quorum queue ended up in a bad state (minority in cluster) even though two other nodes were supposedly up.

2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** Generic server <0.19440.118> terminating
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** Last message in was {'$gen_cast',
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            {method,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                                {'basic.consume',0,<<"accounts_events">>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                                    <<"None1">>,false,false,false,false,[]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                                none,noflow}}
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** When Server state == {ch,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {conf,running,rabbit_framing_amqp_0_9_1,1,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           <0.19433.118>,<0.19436.118>,<0.19433.118>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           <<"10.227.1.39:54170 -> 10.227.0.199:5672">>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           undefined,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           {user,<<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            [administrator],
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            [{rabbit_auth_backend_internal,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                              #Fun<rabbit_auth_backend_internal.3.131409679>}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           <<"venuesync">>,<<"accounts_events">>,<0.19434.118>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           [{<<"consumer_cancel_notify">>,bool,true},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            {<<"connection.blocked">>,bool,true},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            {<<"authentication_failure_close">>,bool,true}],
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           none,0,134217728,1800000,#{},1000000000},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {lstate,<0.19435.118>,false},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          none,1,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {0,[],[]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {state,#{},erlang},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          #{},#{},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {state,fine,5000,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           #Ref<0.1905515858.2512912385.164784>},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          false,1,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {rabbit_confirms,undefined,#{}},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          [],[],none,flow,[],
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {rabbit_queue_type,#{},#{}},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          #Ref<0.1905515858.2512912385.164780>,false}
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** Reason for termination ==
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** {{badmatch,{error,noproc}},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>     [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                      [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                      [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                      [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> 
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>   crasher:
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     initial call: rabbit_channel:init/1
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     pid: <0.19440.118>
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     registered_name: []
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     exception exit: {{badmatch,{error,noproc}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                      [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_queue_type,consume,3,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_misc,with_exit_handler,2,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,handle_cast,2,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {gen_server2,handle_msg,2,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     ancestors: [<0.19439.118>,<0.19437.118>,<0.19432.118>,<0.19431.118>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   <0.841.0>,<0.840.0>,<0.839.0>,<0.837.0>,<0.836.0>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   rabbit_sup,<0.230.0>]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     message_queue_len: 4
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     messages: [{'$gen_cast',{command,{'basic.consume_ok',<<"None1">>}}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   emit_stats,emit_gen_server2_stats,tick]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     links: [<0.19439.118>]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     dictionary: [{permission_cache,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       [{{resource,<<"venuesync">>,queue,<<"accounts_events">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},read},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,exchange,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                             <<"venuesync_events_consumer">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},read},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,queue,<<"accounts_events">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},write},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,queue,<<"accounts_events">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},configure},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,exchange,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                             <<"venuesync_events_consumer">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},configure}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {rand_seed,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {#{jump => #Fun<rand.3.34006561>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                          max => 288230376151711743,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                          next => #Fun<rand.5.34006561>,type => exsplus},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        [117114323147441561|99075661864211887]}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {process_name,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           {<<"10.227.1.39:54170 -> 10.227.0.199:5672">>,1}}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {permission_cache_can_expire,false},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {{xtype_to_module,topic},rabbit_exchange_type_topic},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {topic_permission_cache,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       [{{resource,<<"venuesync">>,topic,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                             <<"venuesync_events_consumer">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{routing_key => <<"*.users.*">>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           variable_map =>
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                               #{<<"username">> =>
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                                     <<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                                 <<"vhost">> => <<"venuesync">>}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         read}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {channel_operation_timeout,15000}]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     trap_exit: true
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     status: running
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     heap_size: 17731
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     stack_size: 28
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     reductions: 37673
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>   neighbours:
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118> 
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118> Error on AMQP connection <0.19433.118> (10.227.1.39:54170 -> 10.227.0.199:5672, vhost: 'venuesync', user: 'default_user_RXWP6jnJNs8x0hDGacN', state: running), channel 1:
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>  {{badmatch,{error,noproc}},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>  [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>                        [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>                   [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,basic_consume,8,[{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,handle_method,3,[{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     supervisor: {<0.19439.118>,rabbit_channel_sup}
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     errorContext: child_terminated
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     reason: {{badmatch,{error,noproc}},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>              [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                    [{file,"rabbit_quorum_queue.erl"},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                     {line,791}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_queue_type,consume,3,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                  [{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_misc,with_exit_handler,2,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                            [{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,handle_cast,2,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {gen_server2,handle_msg,2,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                            [{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     offender: [{pid,<0.19440.118>},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {id,channel},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {mfargs,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                    {rabbit_channel,start_link,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                        [1,<0.19433.118>,<0.19436.118>,<0.19433.118>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         <<"10.227.1.39:54170 -> 10.227.0.199:5672">>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         rabbit_framing_amqp_0_9_1,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         {user,<<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                             [administrator],
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                             [{rabbit_auth_backend_internal,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                  #Fun<rabbit_auth_backend_internal.3.131409679>}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         <<"venuesync">>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         [{<<"consumer_cancel_notify">>,bool,true},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                          {<<"connection.blocked">>,bool,true},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                          {<<"authentication_failure_close">>,bool,true}],
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         <0.19434.118>,<0.19435.118>]}},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {restart_type,intrinsic},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {shutdown,70000},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {child_type,worker}]
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118> Non-AMQP exit reason '{{badmatch,{error,noproc}},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                        [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_queue_type,consume,3,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_misc,with_exit_handler,2,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,handle_cast,2,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {gen_server2,handle_msg,2,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"gen_server2.erl"},{line,1067}]}]}'
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     supervisor: {<0.19439.118>,rabbit_channel_sup}
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     errorContext: shutdown
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     reason: reached_max_restart_intensity
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     offender: [{pid,<0.19440.118>},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {id,channel},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {mfargs,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                    {rabbit_channel,start_link,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                        [1,<0.19433.118>,<0.19436.118>,<0.19433.118>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         <<"10.227.1.39:54170 -> 10.227.0.199:5672">>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         rabbit_framing_amqp_0_9_1,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         {user,<<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                             [administrator],
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                             [{rabbit_auth_backend_internal,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                                  #Fun<rabbit_auth_backend_internal.3.131409679>}]},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         <<"venuesync">>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         [{<<"consumer_cancel_notify">>,bool,true},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                          {<<"connection.blocked">>,bool,true},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                          {<<"authentication_failure_close">>,bool,true}],
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         <0.19434.118>,<0.19435.118>]}},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {restart_type,intrinsic},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {shutdown,70000},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {child_type,worker}]

Any help with this would be appreciated!

[tkoft]

We do have classic queues on that node as well. Is this possibly related to this issue fixed in 3.11.19?

Nodes in clusters that had quorum queues and non-mirrored classic queues on stopped (or failed) nodes
could run into an exception.
The same exception could affect rabbitmqctl list_queues.
GitHub issue: #8596

[michaelklishin]

The line in question. It is one of the steps during consumer registration.

I do not see any obvious connection with #8564. There is no queue filtering going on when a consumer is added, the name of the queue is already known (client-provided).

In any case, the response is still: 3.11 is out of community support, and you are ten patches behind in that series alone.

[michaelklishin]

RabbitMQ 3.11 is out of community support, and 3.11.18 specifically is ten patch releases behind.

[tkoft]

Got it, will upgrade and report if it happens again. Thanks.

[laurentthiebaudezm]

FYI I created another discussion here. It seems to be exact same issue, I am on 3.12.10 (which is also no more supported by community).