An exception in ra_log_segment:fold0/8 invoked by khepri_machine

[ssm6498]

While restarting the RabbitMQ server I am getting the following errors:

Log

 ** When server state  = [{id,{rabbitmq_metadata,'node'}},
                          {opt,terminate},
                          {raft_state,recover},
                          {leader_last_seen,undefined},
                          {num_pending_commands,0},
                          {num_low_priority_commands,0},
                          {num_pending_applied_notifications,0},
                          {election_timeout_set,false},
                          {ra_server_state,
                           #{id =>
                              {rabbitmq_metadata,'nbmqbroker@bptbs1t-w12-002'},
                             machine =>
                              #{tree =>
                                 #{child_nodes => #{},payload_version => 1,
                                   child_list_length => 0,
                                   child_list_version => 1},
                                store_id => rabbitmq_metadata,
                                keep_while_conds => #{},triggers => #{}},
                             log =>
                              #{type => ra_log,cache_size => 0,
                                open_segments => 1,snapshot_index => undefined,
                                last_index => 101,
                                last_written_index_term => {101,9},
                                first_index => 0,
                                last_wal_write => 1715640237491,
                                latest_checkpoint_index => undefined,
                                num_segments => 1},
                             aux => {khepri_machine_aux,rabbitmq_metadata},
                             membership => voter,
                             counter =>
                              {write_concurrency,
                               #Ref<0.436841897.3388342273.168564>},
                             last_applied => 0,commit_index => 71,
                             effective_machine_version => 0,
                             machine_version => 0,
                             uid => <<"RABBITSLF380RA86SP">>,
                             metrics_key => rabbitmq_metadata,
                             current_term => 6,
                             system_config =>
                              #{message_queue_data => off_heap,
                                name => coordination,
                                names =>
                                 #{directory => ra_coordination_directory,
                                   closed_mem_tbls =>
                                    ra_coordination_log_closed_mem_tables,
                                   directory_rev =>
                                    ra_coordination_directory_reverse,
                                   log_ets => ra_coordination_log_ets,
                                   log_meta => ra_coordination_log_meta,
                                   log_sup => ra_coordination_log_sup,
                                   open_mem_tbls =>
                                    ra_coordination_log_open_mem_tables,
                                   segment_writer =>
                                    ra_coordination_segment_writer,
                                   server_sup => ra_coordination_server_sup_sup,
                                   wal => ra_coordination_log_wal,
                                   wal_sup => ra_coordination_log_wal_sup},
                                data_dir => <xyz actual path>
                                                                wal_max_size_bytes => 64000000,
                                wal_max_batch_size => 4096,
                                compress_mem_tables => false,
                                default_max_append_entries_rpc_batch_size =>
                                 128,
                                segment_compute_checksums => true,
                                wal_compute_checksums => true,
                                wal_data_dir =>
                                 "<path> 
                                segment_max_entries => 4096,
                                segment_max_pending => 1024,
                                wal_max_entries => undefined,
                                wal_write_strategy => default,
                                wal_sync_method => datasync,
                                wal_garbage_collect => false,
                                wal_pre_allocate => false,
                                wal_min_bin_vheap_size => 46422,
                                server_min_bin_vheap_size => 46422,
                                default_max_pipeline_count => 4096,
                                snapshot_chunk_size => 1000000,
                                receive_snapshot_timeout => 30000,
                                low_priority_commands_flush_size => 16,
                                low_priority_commands_in_memory_size => 16},
                             voted_for =>
                              {rabbitmq_metadata,'<nodename>'},
                             max_pipeline_count => 4096,
                             log_id => "RabbitMQ metadata store",
                             cluster =>
                              #{{rabbitmq_metadata,
                                 '<nodename>'} =>
                                 #{status => normal,next_index => 1,
                                   query_index => 0,match_index => 0,
                                   commit_index_sent => 0}},
                             cluster_index_term => {0,0},
                             max_append_entries_rpc_batch_size => 128,
                             cluster_change_permitted => false,
                             query_index => 0,num_waiting_queries => 0,
                             machine_versions => [{0,0}],
                             effective_machine_module => khepri_machine,
                             effective_handle_aux_fun => {handle_aux,6}}}]
 ** Reason for termination = exit:{missing_key,72,
                                      "<path> 
 ** Callback modules = [ra_server_proc]
 ** Callback mode = [state_functions,state_enter]
 ** Stacktrace =
 **  [{ra_log_segment,fold0,8,[{file,"src/ra_log_segment.erl"},{line,366}]},
      {ra_log_reader,'-segment_fold/5-fun-1-',4,
                     [{file,"src/ra_log_reader.erl"},{line,378}]},
      {ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',3,
                     [{file,"src/ra_log_reader.erl"},{line,375}]},
      {ra_log_reader,segment_fold,5,
                     [{file,"src/ra_log_reader.erl"},{line,375}]},
      {ra_log_reader,'-fold/5-lists^foldl/2-1-',3,
                     [{file,"src/ra_log_reader.erl"},{line,173}]},
      {ra_log,fold,5,[{file,"src/ra_log.erl"},{line,358}]},
      {ra_server,recover,1,[{file,"src/ra_server.erl"},{line,404}]},
      {ra_server_proc,recover,3,[{file,"src/ra_server_proc.erl"},{line,365}]}]
 
   crasher:
     initial call: ra_server_proc:init/1
     pid: <0.292.0>
     registered_name: rabbitmq_metadata
     exception exit: {missing_key,72,
                                  "<path> 
       in function  ra_log_segment:fold0/8 (src/ra_log_segment.erl, line 366)
       in call from ra_log_reader:'-segment_fold/5-fun-1-'/4 (src/ra_log_reader.erl, line 378)
       in call from ra_log_reader:'-segment_fold/5-lists^foldl/2-1-'/3 (src/ra_log_reader.erl, 
       in call from ra_log_reader:segment_fold/5 (src/ra_log_reader.erl, line 375)
       in call from ra_log_reader:'-fold/5-lists^foldl/2-1-'/3 (src/ra_log_reader.erl, line 173)
       in call from ra_log:fold/5 (src/ra_log.erl, line 358)
       in call from ra_server:recover/1 (src/ra_server.erl, line 404)
       in call from ra_server_proc:recover/3 (src/ra_server_proc.erl, line 365)
     ancestors: [<0.291.0>,ra_coordination_server_sup_sup,<0.275.0>,
                   ra_systems_sup,ra_sup,<0.205.0>]
     message_queue_len: 1
     messages: [{'$gen_call',{<0.251.0>,
                                 [alias|
                                  #Ref<0.0.32131.436841897.3388276737.168641>]},
                                {local_query,#Fun<khepri_machine.0.16117032>}}]
     links: [<0.291.0>]
     dictionary: []
     trap_exit: true
     status: running
     heap_size: 46422
     stack_size: 28
     reductions: 137534
   neighbours:
 
     supervisor: {<0.291.0>,ra_server_sup}
     errorContext: child_terminated
     reason: {missing_key,72,
                          "<path> 
     offender: [{pid,<0.292.0>},
                {id,rabbitmq_metadata},
                {mfargs,
                 {ra_server_proc,start_link,
                  [#{id => {rabbitmq_metadata,'<nodename>'},
                     machine =>
                      {module,khepri_machine,
                       #{member =>
                          {rabbitmq_metadata,'<nodename>'},
                         store_id => rabbitmq_metadata}},
                     parent => <0.291.0>,
                     friendly_name => "RabbitMQ metadata store",
                     cluster_name => rabbitmq_metadata,
                     reply_to =>
                      {#Ref<0.436841897.3388211201.168552>,<0.251.0>},
                     uid => <<"RABBITSLF380RA86SP">>,initial_members => [],
                     log_init_args => #{uid => <<"RABBITSLF380RA86SP">>},
                     tick_timeout => 1000,
                     system_config =>
                      #{message_queue_data => off_heap,name => coordination,
                        names =>
                         #{directory => ra_coordination_directory,
                           closed_mem_tbls =>
                            ra_coordination_log_closed_mem_tables,
                           directory_rev => ra_coordination_directory_reverse,
                           log_ets => ra_coordination_log_ets,
                           log_meta => ra_coordination_log_meta,
                           log_sup => ra_coordination_log_sup,
                           open_mem_tbls => ra_coordination_log_open_mem_tables,
                           segment_writer => ra_coordination_segment_writer,
                           server_sup => ra_coordination_server_sup_sup,
                           wal => ra_coordination_log_wal,
                           wal_sup => ra_coordination_log_wal_sup},
                        data_dir =>
                         "<path> 
                        wal_max_size_bytes => 64000000,
                        wal_max_batch_size => 4096,
                        compress_mem_tables => false,
                        default_max_append_entries_rpc_batch_size => 128,
                        segment_compute_checksums => true,
                        wal_compute_checksums => true,
                        wal_data_dir =>
                         "<path> 
                        segment_max_entries => 4096,
                        segment_max_pending => 1024,
                        wal_max_entries => undefined,
                        wal_write_strategy => default,
                        wal_sync_method => datasync,
                        wal_garbage_collect => false,wal_pre_allocate => false,
                        wal_min_bin_vheap_size => 46422,
                        server_min_bin_vheap_size => 46422,
                        default_max_pipeline_count => 4096,
                        snapshot_chunk_size => 1000000,
                        receive_snapshot_timeout => 30000,
                        low_priority_commands_flush_size => 16,
                        low_priority_commands_in_memory_size => 16},
                     has_changed => false,broadcast_time => 100,
                     install_snap_rpc_timeout => 120000,
                     await_condition_timeout => 30000}]}},
                {restart_type,transient},
                {significant,false},
                {shutdown,5000},
                {child_type,worker}]
 
 ** State machine rabbitmq_metadata terminating
 ** Last event = {internal,go}
 ** When server state  = [{id,{rabbitmq_metadata,'<nodename>'}},
                          {opt,terminate},
                          {raft_state,recover},
                          {leader_last_seen,undefined},
                          {num_pending_commands,0},
                          {num_low_priority_commands,0},
                          {num_pending_applied_notifications,0},
                          {election_timeout_set,false},
                          {ra_server_state,
                           #{id =>
                              {rabbitmq_metadata,'<nodename>'},
                             machine =>
                              #{tree =>
                                 #{child_nodes => #{},payload_version => 1,
                                   child_list_length => 0,
                                   child_list_version => 1},
                                store_id => rabbitmq_metadata,
                                keep_while_conds => #{},triggers => #{}},
                             log =>
                              #{type => ra_log,cache_size => 0,
                                open_segments => 1,snapshot_index => undefined,
                                last_index => 101,
                                last_written_index_term => {101,9},
                                first_index => 0,
                                last_wal_write => 1715640237522,
                                latest_checkpoint_index => undefined,
                                num_segments => 1},
                             aux => {khepri_machine_aux,rabbitmq_metadata},
                             membership => voter,
                             counter =>
                              {write_concurrency,
                               #Ref<0.436841897.3388342273.168662>},
                             last_applied => 0,commit_index => 71,
                             effective_machine_version => 0,
                             machine_version => 0,
                             uid => <<"RABBITSLF380RA86SP">>,
                             metrics_key => rabbitmq_metadata,
                             current_term => 6,
                             system_config =>
                              #{message_queue_data => off_heap,
                                name => coordination,
                                names =>
                                 #{directory => ra_coordination_directory,
                                   closed_mem_tbls =>
                                    ra_coordination_log_closed_mem_tables,
                                   directory_rev =>
                                    ra_coordination_directory_reverse,
                                   log_ets => ra_coordination_log_ets,
                                   log_meta => ra_coordination_log_meta,
                                   log_sup => ra_coordination_log_sup,
                                   open_mem_tbls =>
                                    ra_coordination_log_open_mem_tables,
                                   segment_writer =>
                                    ra_coordination_segment_writer,
                                   server_sup => ra_coordination_server_sup_sup,
                                   wal => ra_coordination_log_wal,
                                   wal_sup => ra_coordination_log_wal_sup},
                                data_dir =>
                                 "<path> 
                                wal_max_size_bytes => 64000000,
                                wal_max_batch_size => 4096,
                                compress_mem_tables => false,
                                default_max_append_entries_rpc_batch_size =>
                                 128,
                                segment_compute_checksums => true,
                                wal_compute_checksums => true,
                                wal_data_dir =>
                                 "<path> 
                                segment_max_entries => 4096,
                                segment_max_pending => 1024,
                                wal_max_entries => undefined,
                                wal_write_strategy => default,
                                wal_sync_method => datasync,
                                wal_garbage_collect => false,
                                wal_pre_allocate => false,
                                wal_min_bin_vheap_size => 46422,
                                server_min_bin_vheap_size => 46422,
                                default_max_pipeline_count => 4096,
                                snapshot_chunk_size => 1000000,
                                receive_snapshot_timeout => 30000,
                                low_priority_commands_flush_size => 16,
                                low_priority_commands_in_memory_size => 16},
                             voted_for =>
                              {rabbitmq_metadata,'<nodename>'},
                             max_pipeline_count => 4096,
                             log_id => "RabbitMQ metadata store",
                             cluster =>
                              #{{rabbitmq_metadata,
                                 '<nodename>'} =>
                                 #{status => normal,next_index => 1,
                                   query_index => 0,match_index => 0,
                                   commit_index_sent => 0}},
                             cluster_index_term => {0,0},
                             max_append_entries_rpc_batch_size => 128,
                             cluster_change_permitted => false,
                             query_index => 0,num_waiting_queries => 0,
                             machine_versions => [{0,0}],
                             effective_machine_module => khepri_machine,
                             effective_handle_aux_fun => {handle_aux,6}}}]
 ** Reason for termination = exit:{missing_key,72,
                                      "<path> 
 ** Callback modules = [ra_server_proc]
 ** Callback mode = [state_functions,state_enter]
 ** Stacktrace =
 **  [{ra_log_segment,fold0,8,[{file,"src/ra_log_segment.erl"},{line,366}]},
      {ra_log_reader,'-segment_fold/5-fun-1-',4,
                     [{file,"src/ra_log_reader.erl"},{line,378}]},
      {ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',3,
                     [{file,"src/ra_log_reader.erl"},{line,375}]},
      {ra_log_reader,segment_fold,5,
                     [{file,"src/ra_log_reader.erl"},{line,375}]},
      {ra_log_reader,'-fold/5-lists^foldl/2-1-',3,
                     [{file,"src/ra_log_reader.erl"},{line,173}]},
      {ra_log,fold,5,[{file,"src/ra_log.erl"},{line,358}]},
      {ra_server,recover,1,[{file,"src/ra_server.erl"},{line,404}]},
      {ra_server_proc,recover,3,[{file,"src/ra_server_proc.erl"},{line,365}]}]
 
   crasher:
     initial call: ra_server_proc:init/1
     pid: <0.293.0>
     registered_name: rabbitmq_metadata
     exception exit: {missing_key,72,
                                  "<path> 
       in function  ra_log_segment:fold0/8 (src/ra_log_segment.erl, line 366)
       in call from ra_log_reader:'-segment_fold/5-fun-1-'/4 (src/ra_log_reader.erl, line 378)
       in call from ra_log_reader:'-segment_fold/5-lists^foldl/2-1-'/3 (src/ra_log_reader.erl, 
       in call from ra_log_reader:segment_fold/5 (src/ra_log_reader.erl, line 375)
       in call from ra_log_reader:'-fold/5-lists^foldl/2-1-'/3 (src/ra_log_reader.erl, line 173)
       in call from ra_log:fold/5 (src/ra_log.erl, line 358)
       in call from ra_server:recover/1 (src/ra_server.erl, line 404)
       in call from ra_server_proc:recover/3 (src/ra_server_proc.erl, line 365)
     ancestors: [<0.291.0>,ra_coordination_server_sup_sup,<0.275.0>,
                   ra_systems_sup,ra_sup,<0.205.0>]
     message_queue_len: 0
     messages: []
     links: [<0.291.0>]
     dictionary: []
     trap_exit: true
     status: running
     heap_size: 46422
     stack_size: 28
     reductions: 146330
   neighbours:
 
     supervisor: {<0.291.0>,ra_server_sup}
     errorContext: child_terminated
     reason: {missing_key,72,
                          "<path> 
     offender: [{pid,<0.293.0>},
                {id,rabbitmq_metadata},
                {mfargs,
                 {ra_server_proc,start_link,
                  [#{id => {rabbitmq_metadata,'<nodename>'},
                     machine =>
                      {module,khepri_machine,
                       #{member =>
                          {rabbitmq_metadata,'<nodename>'},
                         store_id => rabbitmq_metadata}},
                     parent => <0.291.0>,
                     friendly_name => "RabbitMQ metadata store",
                     cluster_name => rabbitmq_metadata,
                     reply_to =>
                      {#Ref<0.436841897.3388211201.168552>,<0.251.0>},
                     uid => <<"RABBITSLF380RA86SP">>,initial_members => [],
                     log_init_args => #{uid => <<"RABBITSLF380RA86SP">>},
                     tick_timeout => 1000,
                     system_config =>
                      #{message_queue_data => off_heap,name => coordination,
                        names =>
                         #{directory => ra_coordination_directory,
                           closed_mem_tbls =>
                            ra_coordination_log_closed_mem_tables,
                           directory_rev => ra_coordination_directory_reverse,
                           log_ets => ra_coordination_log_ets,
                           log_meta => ra_coordination_log_meta,
                           log_sup => ra_coordination_log_sup,
                           open_mem_tbls => ra_coordination_log_open_mem_tables,
                           segment_writer => ra_coordination_segment_writer,
                           server_sup => ra_coordination_server_sup_sup,
                           wal => ra_coordination_log_wal,
                           wal_sup => ra_coordination_log_wal_sup},
                        data_dir =>
                         "<path> 
                        wal_max_size_bytes => 64000000,
                        wal_max_batch_size => 4096,
                        compress_mem_tables => false,
                        default_max_append_entries_rpc_batch_size => 128,
                        segment_compute_checksums => true,
                        wal_compute_checksums => true,
                        wal_data_dir =>
                         "<path> 
                        segment_max_entries => 4096,
                        segment_max_pending => 1024,
                        wal_max_entries => undefined,
                        wal_write_strategy => default,
                        wal_sync_method => datasync,
                        wal_garbage_collect => false,wal_pre_allocate => false,
                        wal_min_bin_vheap_size => 46422,
                        server_min_bin_vheap_size => 46422,
                        default_max_pipeline_count => 4096,
                        snapshot_chunk_size => 1000000,
                        receive_snapshot_timeout => 30000,
                        low_priority_commands_flush_size => 16,
                        low_priority_commands_in_memory_size => 16},
                     has_changed => false,broadcast_time => 100,
                     install_snap_rpc_timeout => 120000,
                     await_condition_timeout => 30000}]}},
                {restart_type,transient},
                {significant,false},
                {shutdown,5000},
                {child_type,worker}]
 
 ** State machine rabbitmq_metadata terminating
 ** Last event = {internal,go}
 ** When server state  = [{id,{rabbitmq_metadata,'<nodename>'}},
                          {opt,terminate},
                          {raft_state,recover},
                          {leader_last_seen,undefined},
                          {num_pending_commands,0},
                          {num_low_priority_commands,0},
                          {num_pending_applied_notifications,0},
                          {election_timeout_set,false},
                          {ra_server_state,
                           #{id =>
                              {rabbitmq_metadata,'<nodename>'},
                             machine =>
                              #{tree =>
                                 #{child_nodes => #{},payload_version => 1,
                                   child_list_length => 0,
                                   child_list_version => 1},
                                store_id => rabbitmq_metadata,
                                keep_while_conds => #{},triggers => #{}},
                             log =>
                              #{type => ra_log,cache_size => 0,
                                open_segments => 1,snapshot_index => undefined,
                                last_index => 101,
                                last_written_index_term => {101,9},
                                first_index => 0,
                                last_wal_write => 1715640237537,
                                latest_checkpoint_index => undefined,
                                num_segments => 1},
                             aux => {khepri_machine_aux,rabbitmq_metadata},
                             membership => voter,
                             counter =>
                              {write_concurrency,
                               #Ref<0.436841897.3388342273.168677>},
                             last_applied => 0,commit_index => 71,
                             effective_machine_version => 0,
                             machine_version => 0,
                             uid => <<"RABBITSLF380RA86SP">>,
                             metrics_key => rabbitmq_metadata,
                             current_term => 6,
                             system_config =>
                              #{message_queue_data => off_heap,
                                name => coordination,
                                names =>
                                 #{directory => ra_coordination_directory,
                                   closed_mem_tbls =>
                                    ra_coordination_log_closed_mem_tables,
                                   directory_rev =>
                                    ra_coordination_directory_reverse,
                                   log_ets => ra_coordination_log_ets,
                                   log_meta => ra_coordination_log_meta,
                                   log_sup => ra_coordination_log_sup,
                                   open_mem_tbls =>
                                    ra_coordination_log_open_mem_tables,
                                   segment_writer =>
                                    ra_coordination_segment_writer,
                                   server_sup => ra_coordination_server_sup_sup,
                                   wal => ra_coordination_log_wal,
                                   wal_sup => ra_coordination_log_wal_sup},
                                data_dir =>
                                 "<path> 
                                wal_max_size_bytes => 64000000,
                                wal_max_batch_size => 4096,
                                compress_mem_tables => false,
                                default_max_append_entries_rpc_batch_size =>
                                 128,
                                segment_compute_checksums => true,
                                wal_compute_checksums => true,
                                wal_data_dir =>
                                 "<path> 
                                segment_max_entries => 4096,
                                segment_max_pending => 1024,
                                wal_max_entries => undefined,
                                wal_write_strategy => default,
                                wal_sync_method => datasync,
                                wal_garbage_collect => false,
                                wal_pre_allocate => false,
                                wal_min_bin_vheap_size => 46422,
                                server_min_bin_vheap_size => 46422,
                                default_max_pipeline_count => 4096,
                                snapshot_chunk_size => 1000000,
                                receive_snapshot_timeout => 30000,
                                low_priority_commands_flush_size => 16,
                                low_priority_commands_in_memory_size => 16},
                             voted_for =>
                              {rabbitmq_metadata,'<nodename>'},
                             max_pipeline_count => 4096,
                             log_id => "RabbitMQ metadata store",
                             cluster =>
                              #{{rabbitmq_metadata,
                                 '<nodename>'} =>
                                 #{status => normal,next_index => 1,
                                   query_index => 0,match_index => 0,
                                   commit_index_sent => 0}},
                             cluster_index_term => {0,0},
                             max_append_entries_rpc_batch_size => 128,
                             cluster_change_permitted => false,
                             query_index => 0,num_waiting_queries => 0,
                             machine_versions => [{0,0}],
                             effective_machine_module => khepri_machine,
                             effective_handle_aux_fun => {handle_aux,6}}}]
 ** Reason for termination = exit:{missing_key,72,
                                      "<path> 
 ** Callback modules = [ra_server_proc]
 ** Callback mode = [state_functions,state_enter]
 ** Stacktrace =
 **  [{ra_log_segment,fold0,8,[{file,"src/ra_log_segment.erl"},{line,366}]},
      {ra_log_reader,'-segment_fold/5-fun-1-',4,
                     [{file,"src/ra_log_reader.erl"},{line,378}]},
      {ra_log_reader,'-segment_fold/5-lists^foldl/2-1-',3,
                     [{file,"src/ra_log_reader.erl"},{line,375}]},
      {ra_log_reader,segment_fold,5,
                     [{file,"src/ra_log_reader.erl"},{line,375}]},
      {ra_log_reader,'-fold/5-lists^foldl/2-1-',3,
                     [{file,"src/ra_log_reader.erl"},{line,173}]},
      {ra_log,fold,5,[{file,"src/ra_log.erl"},{line,358}]},
      {ra_server,recover,1,[{file,"src/ra_server.erl"},{line,404}]},
      {ra_server_proc,recover,3,[{file,"src/ra_server_proc.erl"},{line,365}]}]
 
   crasher:
     initial call: ra_server_proc:init/1
     pid: <0.294.0>
     registered_name: rabbitmq_metadata
     exception exit: {missing_key,72,
                                  "<path> 
       in function  ra_log_segment:fold0/8 (src/ra_log_segment.erl, line 366)
       in call from ra_log_reader:'-segment_fold/5-fun-1-'/4 (src/ra_log_reader.erl, line 378)
       in call from ra_log_reader:'-segment_fold/5-lists^foldl/2-1-'/3 (src/ra_log_reader.erl, 
       in call from ra_log_reader:segment_fold/5 (src/ra_log_reader.erl, line 375)
       in call from ra_log_reader:'-fold/5-lists^foldl/2-1-'/3 (src/ra_log_reader.erl, line 173)
       in call from ra_log:fold/5 (src/ra_log.erl, line 358)
       in call from ra_server:recover/1 (src/ra_server.erl, line 404)
       in call from ra_server_proc:recover/3 (src/ra_server_proc.erl, line 365)
     ancestors: [<0.291.0>,ra_coordination_server_sup_sup,<0.275.0>,
                   ra_systems_sup,ra_sup,<0.205.0>]
     message_queue_len: 0
     messages: []
     links: [<0.291.0>]
     dictionary: []
     trap_exit: true
     status: running
     heap_size: 46422
     stack_size: 28
     reductions: 146330
   neighbours:
 
     supervisor: {<0.291.0>,ra_server_sup}
     errorContext: child_terminated
     reason: {missing_key,72,
                          "<path> 
     offender: [{pid,<0.294.0>},
                {id,rabbitmq_metadata},
                {mfargs,
                 {ra_server_proc,start_link,
                  [#{id => {rabbitmq_metadata,'<nodename>'},
                     machine =>
                      {module,khepri_machine,
                       #{member =>
                          {rabbitmq_metadata,'<nodename>'},
                         store_id => rabbitmq_metadata}},
                     parent => <0.291.0>,
                     friendly_name => "RabbitMQ metadata store",
                     cluster_name => rabbitmq_metadata,
                     reply_to =>
                      {#Ref<0.436841897.3388211201.168552>,<0.251.0>},
                     uid => <<"RABBITSLF380RA86SP">>,initial_members => [],
                     log_init_args => #{uid => <<"RABBITSLF380RA86SP">>},
                     tick_timeout => 1000,
                     system_config =>
                      #{message_queue_data => off_heap,name => coordination,
                        names =>
                         #{directory => ra_coordination_directory,
                           closed_mem_tbls =>
                            ra_coordination_log_closed_mem_tables,
                           directory_rev => ra_coordination_directory_reverse,
                           log_ets => ra_coordination_log_ets,
                           log_meta => ra_coordination_log_meta,
                           log_sup => ra_coordination_log_sup,
                           open_mem_tbls => ra_coordination_log_open_mem_tables,
                           segment_writer => ra_coordination_segment_writer,
                           server_sup => ra_coordination_server_sup_sup,
                           wal => ra_coordination_log_wal,
                           wal_sup => ra_coordination_log_wal_sup},
                        data_dir =>
                         "<path> 
                        wal_max_size_bytes => 64000000,
                        wal_max_batch_size => 4096,
                        compress_mem_tables => false,
                        default_max_append_entries_rpc_batch_size => 128,
                        segment_compute_checksums => true,
                        wal_compute_checksums => true,
                        wal_data_dir =>
                         "<path> 
                        segment_max_entries => 4096,
                        segment_max_pending => 1024,
                        wal_max_entries => undefined,
                        wal_write_strategy => default,
                        wal_sync_method => datasync,
                        wal_garbage_collect => false,wal_pre_allocate => false,
                        wal_min_bin_vheap_size => 46422,
                        server_min_bin_vheap_size => 46422,
                        default_max_pipeline_count => 4096,
                        snapshot_chunk_size => 1000000,
                        receive_snapshot_timeout => 30000,
                        low_priority_commands_flush_size => 16,
                        low_priority_commands_in_memory_size => 16},
                     has_changed => false,broadcast_time => 100,
                     install_snap_rpc_timeout => 120000,
                     await_condition_timeout => 30000}]}},
                {restart_type,transient},
                {significant,false},
                {shutdown,5000},
                {child_type,worker}]
 
     supervisor: {<0.291.0>,ra_server_sup}
     errorContext: shutdown
     reason: reached_max_restart_intensity
     offender: [{pid,<0.294.0>},
                {id,rabbitmq_metadata},
                {mfargs,
                 {ra_server_proc,start_link,
                  [#{id => {rabbitmq_metadata,'<nodename>'},
                     machine =>
                      {module,khepri_machine,
                       #{member =>
                          {rabbitmq_metadata,'<nodename>'},
                         store_id => rabbitmq_metadata}},
                     parent => <0.291.0>,
                     friendly_name => "RabbitMQ metadata store",
                     cluster_name => rabbitmq_metadata,
                     reply_to =>
                      {#Ref<0.436841897.3388211201.168552>,<0.251.0>},
                     uid => <<"RABBITSLF380RA86SP">>,initial_members => [],
                     log_init_args => #{uid => <<"RABBITSLF380RA86SP">>},
                     tick_timeout => 1000,
                     system_config =>
                      #{message_queue_data => off_heap,name => coordination,
                        names =>
                         #{directory => ra_coordination_directory,
                           closed_mem_tbls =>
                            ra_coordination_log_closed_mem_tables,
                           directory_rev => ra_coordination_directory_reverse,
                           log_ets => ra_coordination_log_ets,
                           log_meta => ra_coordination_log_meta,
                           log_sup => ra_coordination_log_sup,
                           open_mem_tbls => ra_coordination_log_open_mem_tables,
                           segment_writer => ra_coordination_segment_writer,
                           server_sup => ra_coordination_server_sup_sup,
                           wal => ra_coordination_log_wal,
                           wal_sup => ra_coordination_log_wal_sup},
                        data_dir =>
                         "<path> 
                        wal_max_size_bytes => 64000000,
                        wal_max_batch_size => 4096,
                        compress_mem_tables => false,
                        default_max_append_entries_rpc_batch_size => 128,
                        segment_compute_checksums => true,
                        wal_compute_checksums => true,
                        wal_data_dir =>
                         "<path> 
                        segment_max_entries => 4096,
                        segment_max_pending => 1024,
                        wal_max_entries => undefined,
                        wal_write_strategy => default,
                        wal_sync_method => datasync,
                        wal_garbage_collect => false,wal_pre_allocate => false,
                        wal_min_bin_vheap_size => 46422,
                        server_min_bin_vheap_size => 46422,
                        default_max_pipeline_count => 4096,
                        snapshot_chunk_size => 1000000,
                        receive_snapshot_timeout => 30000,
                        low_priority_commands_flush_size => 16,
                        low_priority_commands_in_memory_size => 16},
                     has_changed => false,broadcast_time => 100,
                     install_snap_rpc_timeout => 120000,
                     await_condition_timeout => 30000}]}},
                {restart_type,transient},
                {significant,false},
                {shutdown,5000},
                {child_type,worker}]
 
 
 BOOT FAILED
 ===========
 Exception during startup:
 
 exit:{{missing_key,72,"<path> 
 
     gen:do_call/4, line 269
     gen_statem:call/3, line 923
     ra_server_proc:gen_statem_safe_call/3, line 1573
     ra_server_proc:statem_call/3, line 234
     khepri_machine:process_local_query/3, line 846
     khepri_machine:fold/5, line 212
     khepri:exists/3, line 1166
     rabbit_khepri:wait_for_leader/2, line 286
 
   crasher:
     initial call: application_master:init/4
     pid: <0.250.0>
     registered_name: []
     exception exit: {{{missing_key,72,
                           "<path> 
                       {gen_statem,call,
                           [{rabbitmq_metadata,'<nodename>'},
                            {local_query,#Fun<khepri_machine.0.16117032>},
                            30000]}},
                      {rabbit,start,[normal,[]]}}
       in function  application_master:init/4 (application_master.erl, line 142)
     ancestors: [<0.249.0>]
     message_queue_len: 1
     messages: [{'EXIT',<0.251.0>,normal}]
     links: [<0.249.0>,<0.44.0>]
     dictionary: []
     trap_exit: true
     status: running
     heap_size: 1598
     stack_size: 28
     reductions: 223
   neighbours:
 
 Application rabbit exited with reason: {{{missing_key,72,"<path> 
> Logging: configured log handlers are now ACTIVE
ra: starting system quorum_queues
starting Ra system: quorum_queues in directory: <path> 
ra system 'quorum_queues' running pre init for 0 registered servers

can anyone please check.
Can any one please suggest the solution

Why ra:coordination is needed? is it configurable?

[kjnilsson]

Was this system upgraded from an earlier version?

[lukebakken]

@ssm6498 rather than pasting a large amount of text into a GitHub issue, either save the text to a file and attach it, or use <details> as I have done.

[ssm6498]

Was this system upgraded from an earlier version?

Rabbitmq and erlang both are upgraded. No OS upgrade
Rabbitmq upgrade from v3.11.10 to v3.13.2 Erlang upgrade from v25.3.2 to v 26.2.3

[michaelklishin]

@ssm6498 all we can say from the stack trace is the function involved, the fact that it could not find a key ("72") and that you have Khepri enabled because the Ra machine in question is a khepri_machine.

Please share the data directory. We do not guess in this community.

[ssm6498]

Unfortunately currently I am not having the access to the host, I am reproducing it.
But in the logs i can see only below logs
Starting server logs:
Server startup complete; 7 plugins started.

• rabbitmq_auth_backend_http
• rabbitmq_stomp
• rabbitmq_auth_backend_cache
• rabbitmq_management
• rabbitmq_management_agent
• rabbitmq_web_dispatch
• oauth2_client

while stopping:
Stopping RabbitMQ applications and their dependencies in the following order:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_web_dispatch
cowboy
cowlib
rabbitmq_auth_backend_cache
oauth2_client
rabbitmq_stomp
rabbitmq_auth_backend_http
amqp_client
rabbit
rabbitmq_prelaunch
rabbit_common
khepri
ra
os_mon
sysmon_handler
osiris
mnesia

[michaelklishin]

Why ra:coordination is needed? is it configurable?

The exception comes from ra_log_segment:fold0/8 used by khepri_machine. Khepri is an experimental alternative data store for all metadata and a certain component of Khepri is started even when the feature is not enabled. It does next to nothing, however, so it's unusual to see it run into an exception.

In order to suggest anything else we need the node's data directory.

[ssm6498]

Okay, Initially only few plugins are enabled its trying to stop Khepri even if it is not enable.
I understand your ask, will be sharing data directory soon.

[ssm6498]

The contents of feature flag file

[classic_mirrored_queue_version,
classic_queue_type_delivery_support,
detailed_queues_endpoint,
direct_exchange_routing_v2,
drop_unroutable_metric,
empty_basic_get_metric,
feature_flags_v2,
implicit_default_bindings,
listener_records_in_ets,
maintenance_mode_status,
message_containers,
quorum_queue,
quorum_queue_non_voters,
restart_streams,
stream_filtering,
stream_queue,
stream_sac_coordinator_unblock_group,
stream_single_active_consumer,
stream_update_config_command,
tracking_records_in_ets,
user_limits,
virtual_host_metadata].

Also 'coordination' is got cretaed inside mnesia directory.
which has segment file, config file, checkpoints directory and snapshot directory.

contents of config file :

#{id => {rabbitmq_metadata,'nbmqbroker@mrp-win-7fb-061'},
machine =>
{module,khepri_machine,
#{member => {rabbitmq_metadata,'node@host'},
store_id => rabbitmq_metadata}},
friendly_name => "RabbitMQ metadata store",
cluster_name => rabbitmq_metadata,uid => <<"RABBIT7DVZN7U8ABLD">>,
initial_members => [],
log_init_args => #{uid => <<"RABBIT7DVZN7U8ABLD">>},
tick_timeout => 1000,broadcast_time => 100,
install_snap_rpc_timeout => 120000,await_condition_timeout => 30000}.

do you want me to share any specific file or directory content?

How one can avoid starting of certain components of Khepri if the feature is disabled and how we can mark it as disabled.

It will be great help if one can provide the way to handle this upgrade scenario gracefully

[ssm6498]

I would like to add one more point, this issue is observed on WIndows platform only, till now on Linux RHEL platform we didnt find any issue. Though it is starting rs system coordination, its not logging any error.

ra: starting system quorum_queues starting Ra system: quorum_queues in directory: ra system 'quorum_queues' running pre init for 0 registered servers ra: meta data store initialised for system quorum_queues. 0 record(s) recovered WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables ra: starting system coordination starting Ra system: coordination in directory: ra system 'coordination' running pre init for 1 registered servers ra: meta data store initialised for system coordination. 1 record(s) recovered WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables ra: starting system coordination starting Ra system: coordination in directory: Waiting for Khepri leader for 30000 ms, 9 retries left Khepri leader elected RabbitMQ metadata store: candidate -> leader in term: 5 machine version: 0

Starting RabbitMQ 3.13.2 on Erlang 26.2.3 [emu]

The rabbitmq upgrade is blocked due to this issue.

[kjnilsson]

Please share the full coordination directory with us. Then try removing it and restarting the server.