Journal.jif prevents RabbitMQ from starting

[markusgust]

Hi team,

On a few occasions during the past couple weeks we’ve observed clusters that recently upgraded to 3.10.x sees RabbitMQ crash, restart and attempts to rebuild indices from scratch. We haven't seen a consistent crash reasons, but have attached two examples below. During this rebuild an error occurs which points to a journal.jif file inside mnesia and rabbitMQ refuses to start. When removing either this file or the corrupt .idx file in the queue, rabbitMQ starts again.

We haven’t been able to reproduce this internally. Interestingly, it will mention the last message to rabbit_classic_queue_index_v2.11, however neither of the clusters are using CQv2.

Below are two examples of where this occurred. Any ideas on what could be causing this?

Example 1:

2022-07-05 15:10:38.190355+00:00 [error] <0.524.0> ** Reason for termination ==
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0> ** {{case_clause,-1},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>     [{rabbit_variable_queue,maybe_deltas_to_betas,2,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,3213}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,fetch_from_q3,1,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,3140}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,queue_out,1,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,1901}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,collect_by_predicate,3,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,2140}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,fetch_by_predicate,4,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,2101}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,fetchwhile,4,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,738}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_priority_queue,'-fetchwhile/4-fun-0-',6,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_priority_queue.erl"},{line,286}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_priority_queue,findfold3,5,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_priority_queue.erl"},{line,578}]}]}
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>
…..
2022-07-05 15:10:57.303146+00:00 [warning] <0.504.0> Message store "VHOSTHASH/msg_store_persistent": rebuilding indices from scratch
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0> ** Generic server <0.374.0> terminating
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0> ** Last message in was {'$gen_cast',
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>                            {submit_async,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>                                #Fun<rabbit_classic_queue_index_v2.11.133535598>,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>                                <0.372.0>}}
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0> ** When Server state == undefined
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0> ** Reason for termination ==
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0> ** {function_clause,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>        [{rabbit_queue_index,journal_minus_segment1,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>             [{no_pub,no_del,ack},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>              {{true,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>                   <<REDACTED1>>},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>               no_del,no_ack}],
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>             [{file,"rabbit_queue_index.erl"},{line,1390}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>             [{file,"rabbit_queue_index.erl"},{line,1367}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {rabbit_queue_index,'-recover_journal/1-fun-0-',1,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>             [{file,"rabbit_queue_index.erl"},{line,1054}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {maps,map_1,2,[{file,"maps.erl"},{line,438}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {maps,map,2,[{file,"maps.erl"},{line,428}]},
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>         {rabbit_queue_index,segment_map,2,
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0>             [{file,"rabbit_queue_index.erl"},{line,1177}]}]}
2022-07-05 15:10:57.344844+00:00 [error] <0.374.0> 
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>   crasher:
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     initial call: worker_pool_worker:init/1
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     pid: <0.374.0>
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     registered_name: []
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     exception exit: {function_clause,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                         [{rabbit_queue_index,journal_minus_segment1,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{no_pub,no_del,ack},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                               {{true,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                                    <<REDACTED2>>},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                                no_del,no_ack}],
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{file,"rabbit_queue_index.erl"},{line,1390}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {rabbit_queue_index,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              '-journal_minus_segment/3-fun-0-',4,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{file,"rabbit_queue_index.erl"},{line,1367}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {array,sparse_foldl_3,7,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{file,"array.erl"},{line,1684}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {array,sparse_foldl_2,9,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{file,"array.erl"},{line,1678}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {rabbit_queue_index,'-recover_journal/1-fun-0-',1,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{file,"rabbit_queue_index.erl"},{line,1054}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {maps,map_1,2,[{file,"maps.erl"},{line,438}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {maps,map,2,[{file,"maps.erl"},{line,428}]},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          {rabbit_queue_index,segment_map,2,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              [{file,"rabbit_queue_index.erl"},{line,1177}]}]}
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     ancestors: [worker_pool_sup,rabbit_sup,<0.224.0>]
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     message_queue_len: 0
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     messages: []
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     links: [<0.371.0>]
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     dictionary: [{{#Ref<0.1298939975.721158145.15545>,fhc_handle},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                    {handle,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        {file_descriptor,prim_file,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                            #{handle => #Ref<0.1298939975.721289223.11473>,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              owner => <0.374.0>,r_ahead_size => 0,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                              r_buffer => #Ref<0.1298939975.721289218.13932>}},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        #Ref<0.1298939975.721158145.15545>,221477,false,0,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        infinity,[],<<>>,0,0,0,0,0,false,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        "/var/lib/rabbitmq/mnesia/rabbit@HOSTNAME2-01/msg_stores/vhosts/VHOSTHASH/queues/QUEUEHASH/journal.jif",
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        [write,binary,raw,read],
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        [{write_buffer,infinity}],
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        true,true,-576460745462573977}},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                   {{"/var/lib/rabbitmq/mnesia/rabbit@HOSTNAME2-01/msg_stores/vhosts/VHOSTHASH/queues/QUEUEHASH/journal.jif",
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                     fhc_file},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                    {file,1,true}},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                   {worker_pool_name,worker_pool},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                   {segment_entry_count,2048},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                   {fhc_age_tree,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                       {1,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        {{-576460745462573977,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          #Ref<0.1298939975.721158145.15545>},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                         true,nil,nil}}},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                   {rand_seed,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                       {#{jump => #Fun<rand.3.92093067>,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          max => 288230376151711743,
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                          next => #Fun<rand.5.92093067>,type => exsplus},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                        [81582003007021117|62511998823895942]}},
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>                   {worker_pool_worker,true}]
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     trap_exit: false
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     status: running
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     heap_size: 75113
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     stack_size: 28
2022-07-05 15:10:57.346756+00:00 [error] <0.374.0>     reductions: 1176686

Example 2:

2022-07-05 15:10:38.190355+00:00 [error] <0.524.0> ** Reason for termination ==
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0> ** {{case_clause,-1},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>     [{rabbit_variable_queue,maybe_deltas_to_betas,2,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,3213}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,fetch_from_q3,1,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,3140}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,queue_out,1,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,1901}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,collect_by_predicate,3,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,2140}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,fetch_by_predicate,4,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,2101}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_variable_queue,fetchwhile,4,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_variable_queue.erl"},{line,738}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_priority_queue,'-fetchwhile/4-fun-0-',6,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_priority_queue.erl"},{line,286}]},
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>      {rabbit_priority_queue,findfold3,5,
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>                             [{file,"rabbit_priority_queue.erl"},{line,578}]}]}
2022-07-05 15:10:38.190355+00:00 [error] <0.524.0>
……
2022-07-05 01:44:36.483470+00:00 [info] <0.16746.1164> Message store "VHOSTHASH/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-07-05 01:44:36.544032+00:00 [warn] <0.16746.1164> Message store "VHOSTHASH/msg_store_persistent" : recovery terms differ from present
2022-07-05 01:44:36.544032+00:00 [warn] <0.16746.1164> rebuilding indices from scratch
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0> ** Generic server <0.371.0> terminating
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0> ** Last message in was {'$gen_cast',
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>                            {submit_async,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>                                #Fun<rabbit_classic_queue_index_v2.11.92001267>,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>                                <0.370.0>}}
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0> ** When Server state == undefined
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0> ** Reason for termination ==
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0> ** {function_clause,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>        [{rabbit_queue_index,journal_minus_segment1,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>             [{no_pub,no_del,ack},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>              {{true,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>                   <<DATASEGMENT,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>                     DATASEGMENT>>,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>                   <<>>},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>               no_del,no_ack}],
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>             [{file,"rabbit_queue_index.erl"},{line,1390}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>             [{file,"rabbit_queue_index.erl"},{line,1367}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {rabbit_queue_index,'-recover_journal/1-fun-0-',1,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>             [{file,"rabbit_queue_index.erl"},{line,1054}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {lists,map,2,[{file,"lists.erl"},{line,1243}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {rabbit_queue_index,segment_map,2,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>             [{file,"rabbit_queue_index.erl"},{line,1178}]},
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>         {rabbit_queue_index,recover_journal,1,
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0>             [{file,"rabbit_queue_index.erl"},{line,1045}]}]}
2022-07-05 01:44:36.610020+00:00 [erro] <0.371.0> 
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>   crasher:
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     initial call: worker_pool_worker:init/1
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     pid: <0.371.0>
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     registered_name: []
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     exception exit: {function_clause,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                         [{rabbit_queue_index,journal_minus_segment1,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{no_pub,no_del,ack},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                               {{true,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                                    <<DATASEGMENT,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                                      DATASEGMENT,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                                      DATASEGMENT>>,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                                    <<>>},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                                no_del,no_ack}],
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"rabbit_queue_index.erl"},{line,1390}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {rabbit_queue_index,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              '-journal_minus_segment/3-fun-0-',4,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"rabbit_queue_index.erl"},{line,1367}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {array,sparse_foldl_3,7,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"array.erl"},{line,1684}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {array,sparse_foldl_2,9,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"array.erl"},{line,1678}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {rabbit_queue_index,'-recover_journal/1-fun-0-',1,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"rabbit_queue_index.erl"},{line,1054}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {lists,map,2,[{file,"lists.erl"},{line,1243}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {rabbit_queue_index,segment_map,2,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"rabbit_queue_index.erl"},{line,1178}]},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          {rabbit_queue_index,recover_journal,1,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              [{file,"rabbit_queue_index.erl"},{line,1045}]}]}
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     ancestors: [worker_pool_sup,rabbit_sup,<0.224.0>]
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     message_queue_len: 0
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     messages: []
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     links: [<0.369.0>]
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     dictionary: [{{"/var/lib/rabbitmq/mnesia/rabbit@HOSTNAME-01/msg_stores/vhosts/VHOSTHASH/queues/QUEUEHASH/journal.jif",
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                     fhc_file},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                    {file,1,true}},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {{#Ref<0.782707678.2675179521.67422>,fhc_handle},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                    {handle,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        {file_descriptor,prim_file,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                            #{handle => #Ref<0.782707677.59637780.40153>,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              owner => <0.371.0>,r_ahead_size => 0,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                              r_buffer => #Ref<0.782707678.2675310593.67424>}},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        #Ref<0.782707678.2675179521.67422>,8,false,0,infinity,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        [],<<>>,0,0,0,0,0,false,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        "/var/lib/rabbitmq/mnesia/rabbit@HOSTNAME-01/msg_stores/vhosts/VHOSTHASH/queues/QUEUEHASH/journal.jif",
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        [write,binary,raw,read],
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        [{write_buffer,infinity}],
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        true,true,-573134915964975382}},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {worker_pool_name,worker_pool},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {fhc_age_tree,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                       {1,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        {{-573134915964975382,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          #Ref<0.782707678.2675179521.67422>},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                         true,nil,nil}}},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {{xtype_to_module,direct},rabbit_exchange_type_direct},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {guid,{{125081840,1723787507,879285044,1413127255},0}},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {{xtype_to_module,topic},rabbit_exchange_type_topic},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {{xtype_to_module,fanout},rabbit_exchange_type_fanout},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {segment_entry_count,16384},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {rand_seed,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                       {#{jump => #Fun<rand.3.92093067>,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          max => 288230376151711743,
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                          next => #Fun<rand.5.92093067>,type => exsplus},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                        [37796445080923696|106264654589298098]}},
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>                   {worker_pool_worker,true}]
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     trap_exit: false
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     status: running
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     heap_size: 6772
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     stack_size: 29
2022-07-05 01:44:36.610667+00:00 [erro] <0.371.0>     reductions: 643116985

Happy to provide more info if required.

[michaelklishin]

We see this error a few times a year, less and less frequently. Please switch to at least CQv2, or a combination of CQv2, quorum queues and streams.

There won't be much time invested into CQv1. In 3.11, all improvements will go into CQv2 and its unification with lazy mode. It likely will become the new default storage option for classic queues.

[johanrhodin]

Yes it used to be that we saw this error a few times per year (as seen in the number of cases on the mailing list https://groups.google.com/g/rabbitmq-users/search?q=rabbit_queue_index%2Cjournal_minus_segment1), but recently we (CloudAMQP) are seeing this weekly in 3.10.x so that is why we have increased focus on this now.

Also it used to be submit_async,#Fun<rabbit_queue_index and now it is submit_async, #Fun<rabbit_classic_queue_index_v2 could that hint at something with v2 involved?

I hope the core team would still entertain a PR increasing the stability/robustness of the recovery mechanism? (It is vacation times for a large part of the team so no guarantee on timeline)

[mehdigmira]

We have also been experiencing this very frequently with 3.10.x. We're going back to 3.9 because this is a major issue for us.

[michaelklishin]

We would consider a PR but I'm not sure what the proposed changes "increasing the stability/robustness of the recovery mechanism" are.

[johanrhodin]

For anyone seeing this thread, this comment can be helpful: #7203 (comment)

Update Apr 21 2023: also see this bug: #7904