Getting error in RabbitMQ and 2 out of 3 nodes goes in timout raft state

[gsingla294]

Can anyone help in to identify what has happened here? Suddenly getting an error.

Stacktrace

2024-04-09 15:04:18.714	{"log":"2024-04-09 09:34:18.513356+00:00 [info] <0.20164.3> queue 'gms.request.ingest.job.31eea51e-d87d-44af-b6dd-481a74876f54.q' in vhost '/': granting vote for {'%2F_gms.request.ingest.job.31eea51e-d87d-44af-b6dd-481a74876f54.q','[email protected]'} with last indexterm {290,7320} for term 8145 previous term was 8144\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193> ** Generic server <0.31648.193> terminating\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193> ** Last message in was {'$gen_cast',\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            {method,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                                {'basic.consume',0,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                                    <<\"gms.eventlog.q.request\">>,<<>>,false,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                                    false,false,false,[]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                                none,noflow}}\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193> ** When Server state == {ch,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {conf,running,rabbit_framing_amqp_0_9_1,32,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           <0.21813.193>,<0.31644.193>,<0.21813.193>,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           <<\"10.206.160.100:6884 -> 10.206.160.183:5671\">>,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           undefined,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           {user,<<\"rabbitmq\">>,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            [management],\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            [{rabbit_auth_backend_internal,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                              #Fun<rabbit_auth_backend_internal.3.73424183>}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           <<\"/\">>,<<\"gms.eventlog.q.request\">>,<0.21876.193>,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           [{<<\"exchange_exchange_bindings\">>,bool,true},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            {<<\"connection.blocked\">>,bool,true},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            {<<\"authentication_failure_close\">>,bool,true},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            {<<\"basic.nack\">>,bool,true},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            {<<\"publisher_confirms\">>,bool,true},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                            {<<\"consumer_cancel_notify\">>,bool,true}],\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           none,1,134217728,undefined,#{},1000000000,false},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {lstate,<0.31646.193>,false},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          none,1,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {0,[],[]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {state,#{},erlang},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          #{},#{},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {state,fine,5000,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           #Ref<0.3136501562.1039138822.122317>},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          false,1,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {rabbit_confirms,undefined,#{}},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          [],[],none,flow,[],\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {rabbit_queue_type,#{},#{}},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          #Ref<0.3136501562.1038614539.77155>,false}\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193> ** Reason for termination ==\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193> ** {{badmatch,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>         {timeout,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>             {'%2F_gms.eventlog.q.request',\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                 '[email protected]'}}},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>     [{rabbit_quorum_queue,consume,3,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>          [{file,\"rabbit_quorum_queue.erl\"},{line,795}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {rabbit_queue_type,consume,3,[{file,\"rabbit_queue_type.erl\"},{line,408}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {rabbit_channel,'-basic_consume/8-fun-0-',10,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>          [{file,\"rabbit_channel.erl\"},{line,1817}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {rabbit_misc,with_exit_handler,2,[{file,\"rabbit_misc.erl\"},{line,531}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {rabbit_channel,basic_consume,8,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>          [{file,\"rabbit_channel.erl\"},{line,1814}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {rabbit_channel,handle_method,3,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>          [{file,\"rabbit_channel.erl\"},{line,1493}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {rabbit_channel,handle_cast,2,[{file,\"rabbit_channel.erl\"},{line,652}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>      {gen_server2,handle_msg,2,[{file,\"gen_server2.erl\"},{line,1067}]}]}\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193> \n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>   crasher:\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>     initial call: rabbit_channel:init/1\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>     pid: <0.31648.193>\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>     registered_name: []\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>     exception exit: {{badmatch,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                          {timeout,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                              {'%2F_gms.eventlog.q.request',\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                                  '[email protected]'}}},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                      [{rabbit_quorum_queue,consume,3,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           [{file,\"rabbit_quorum_queue.erl\"},{line,795}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                       {rabbit_queue_type,consume,3,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           [{file,\"rabbit_queue_type.erl\"},{line,408}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                       {rabbit_channel,'-basic_consume/8-fun-0-',10,\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                           [{file,\"rabbit_channel.erl\"},{line,1817}]},\n"}
2024-04-09 15:04:18.714	{ [error] <0.31648.193>                       {rabbit_misc,with_exit_handler,2,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           [{file,\"rabbit_misc.erl\"},{line,531}]},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {rabbit_channel,basic_consume,8,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           [{file,\"rabbit_channel.erl\"},{line,1814}]},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {rabbit_channel,handle_method,3,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           [{file,\"rabbit_channel.erl\"},{line,1493}]},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {rabbit_channel,handle_cast,2,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           [{file,\"rabbit_channel.erl\"},{line,652}]},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {gen_server2,handle_msg,2,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           [{file,\"gen_server2.erl\"},{line,1067}]}]}\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     ancestors: [<0.31645.193>,<0.21897.193>,<0.21812.193>,<0.21811.193>,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   <0.31883.14>,<0.31878.14>,<0.31877.14>,<0.31875.14>,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   <0.31874.14>,rabbit_sup,<0.695.0>]\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     message_queue_len: 4\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     messages: [{'$gen_cast',\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {command,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           {'basic.consume_ok',\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                               <<\"amq.ctag-Z5aRKHgbz5_QeFqtvYMsZQ\">>}}},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   emit_stats,emit_gen_server2_stats,tick]\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     links: [<0.31645.193>]\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     dictionary: [{process_name,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {rabbit_channel,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                           {<<\"10.206.160.100:6884 -> 10.206.160.183:5671\">>,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                            32}}},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   {permission_cache_can_expire,false},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   {permission_cache,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       [{{resource,<<\"/\">>,queue,<<\"gms.eventlog.q.request\">>},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                         #{},read}]},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   {guid_secure,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {{16,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                         '[email protected]',\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                         #Ref<0.3136501562.1038352399.249144>},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                        0}},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   {channel_operation_timeout,15000},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                   {rand_seed,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                       {#{jump => #Fun<rand.3.34006561>,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                          max => 288230376151711743,\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                          next => #Fun<rand.5.34006561>,type => exsplus},\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>                        [214257254619724812|1981871495883992]}}]\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     trap_exit: true\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     status: running\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     heap_size: 17731\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     stack_size: 28\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>     reductions: 42216\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193>   neighbours:\n"}
2024-04-09 15:04:18.723	{ [error] <0.31648.193> \n"}

[lukebakken]

Hello, thanks for using RabbitMQ.

One stack trace with no other information is hardly anything to work with. We don't know what version of RabbitMQ and Erlang you're using. I'd like to point out this document as a guide for reporting RabbitMQ issues (it applies to software in general).

In addition, your node name ([email protected]) suggests that this is a production issue (produs-rmq). If RabbitMQ is a critical component of your business, you should contribute back by purchasing support - https://www.rabbitmq.com/contact#paid-support

You said "suddenly getting an error". Software does not work normally, then stop working normally, at random - something must have changed in your environment. So, your first task is to identify what may have changed.

[lukebakken]

You have already received responses in these places as well -

• 3.11: two Raft replicas are in the timeout state, one is a candidate #10936
• 3.11: two Raft replicas are in the timeout state, one is a candidate #10934

[michaelklishin]

@gsingla294 you were blocked from starting any more discussions for 30 days. "Out of support" means out of support. "Clues are in the logs" mean you must inspect and share all logs from all nodes, not just a single stack trace.

Follow the rules of the community or figure things out on your own, it's as simple as that. You get the software for free, updates for free, and a certain degree of support for free as long as you are on the most recent series and you cooperate.

[michaelklishin]

@gsingla294 you have already been told that 3.11 is out of community support. Let me clarify what "out of support" means: you are not entitled to any help from the core team until you upgrade to 3.13.x or pay.

All the message says that a basic.consume operation on a quorum queue timed out. This almost always means that the queue does not have an online majority. There can be any number of reasons for that, including certain upgrade strategies that require extra care (using certain health checks).

FTR, we are investigating one (seemingly AKS-specific) behavior where filesystem writes return errors that we do not see elsewhere. They, in turn, greatly complicate QQ replica operations. That is for a paying customer and required a lot of additional information that you are not providing, and is on a different version. This may or may not related, we do not have enough information here and we do not guess in this community.