RabbitMQ high memory consumption with {badmatch,{not_found, ra_log}}

[vb0om]

I reguraly meet the issue when one of my rabbitmq nodes in the k8s suddenly consumes all my memory on the node, and then will be killed by OOM. After restart everything works fine, but ... this situation can repeat once or twice a week, so it's not expected behaviour. And I didn't find any reason fo it. Maybe somebody can help me with this?

My environment:

• Kubernetes cluster 1.25
• Rabbitmq was installed through helm package with version of docker image rabbitmq - docker.io/bitnami/rabbitmq:3.10.7-debian-11-r4
• Rabbitmq Cluster with 3 nodes with quorum queues(about 180)
• Consumers are .Net apps
• Erlang version is 24.3.4
  RabbitMQ was deployed via this conf:
  
  After RabbitMQ node was down I got next logs:

2024-03-14T15:45:30.587934978Z stdout F 2024-03-14 15:45:30.587524+00:00 [info] <0.1277.0> vm_memory_high_watermark set. Memory used:3418648576 allowed:3221225472
2024-03-14T15:45:30.588245587Z stdout F 2024-03-14 15:45:30.587780+00:00 [warning] <0.1275.0> memory resource limit alarm set on node '[email protected]'.
2024-03-14T15:45:30.588266609Z stdout F 2024-03-14 15:45:30.587780+00:00 [warning] <0.1275.0> **********************************************************
2024-03-14T15:45:30.588274577Z stdout F 2024-03-14 15:45:30.587780+00:00 [warning] <0.1275.0> *** Publishers will be blocked until this alarm clears ***
2024-03-14T15:45:30.588279298Z stdout F 2024-03-14 15:45:30.587780+00:00 [warning] <0.1275.0> **********************************************************
2024-03-14T15:45:36.245444636Z stdout F 2024-03-14 15:45:08.706447+00:00 [error] <0.2064.0> ** State machine '%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5' terminating
2024-03-14T15:45:36.245493345Z stdout F 2024-03-14 15:45:08.706447+00:00 [error] <0.2064.0> ** Last event = {cast,
                                                                                                                {append_entries_rpc,80,
                                                                                                                    {'%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                        '[email protected]'},
                                                                                                                    2600142,2600142,79,[]}}
2024-03-14T15:45:36.245527246Z stdout F 2024-03-14 15:45:08.706447+00:00 [error] <0.2064.0> ** When server state  = [{id,
                                                                                                                      {'%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                       '[email protected]'}},
                                                                                                                     {opt,terminate},
                                                                                                                     {raft_state,follower},
                                                                                                                     {leader_last_seen,undefined},
                                                                                                                     {num_pending_commands,0},
                                                                                                                     {num_delayed_commands,0},
                                                                                                                     {num_pending_applied_notifications,0},
                                                                                                                     {election_timeout_set,true},
                                                                                                                     {ra_server_state,
                                                                                                                      #{aux =>
                                                                                                                        {aux_v2,
                                                                                                                          '%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                          {0,true},
                                                                                                                          {inactive,-576460717736502,1,1.0},
                                                                                                                          {aux_gc,0},
                                                                                                                          undefined,undefined},
                                                                                                                        cluster =>
                                                                                                                         #{{'%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                            '[email protected]'} =>
                                                                                                                            #{commit_index_sent => 2600142,
                                                                                                                              match_index => 2600142,
                                                                                                                              next_index => 2600144,query_index => 0,
                                                                                                                              status => normal},
                                                                                                                          {'%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                            '[email protected]'} =>
                                                                                                                            #{commit_index_sent => 0,match_index => 0,
                                                                                                                              next_index => 1,query_index => 0,
                                                                                                                              status => normal},
                                                                                                                          {'%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                            '[email protected]'} =>
                                                                                                                            #{commit_index_sent => 2600142,
                                                                                                                              match_index => 2600142,
                                                                                                                              next_index => 2600144,query_index => 0,
                                                                                                                              status => normal}},
                                                                                                                        commit_index => 2600142,
                                                                                                                        counter =>
                                                                                                                          {write_concurrency,
                                                                                                                           #Ref<0.839822614.4015652865.199022>},
                                                                                                                      current_term => 80,
                                                                                                                        effective_machine_module => rabbit_fifo,
                                                                                                                        effective_machine_version => 2,
                                                                                                                        id =>
                                                                                                                          {'%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                           '[email protected]'},
                                                                                                                        last_applied => 2600142,leader_id => undefined,
                                                                                                                        log =>
                                                                                                                         #{cache_size => 1,first_index => 2600143,
                                                                                                                           last_index => 2600143,
                                                                                                                          last_written_index_term => {2600142,79},
                                                                                                                            num_segments => 0,open_segments => 0,
                                                                                                                            snapshot_index => 2600142,type => ra_log},
                                                                                                                        log_id =>
                                                                                                                         "queue '092f56e3-1277-4bcc-b5de-479d59e8a0a5' in vhost '/'",
                                                                                                                        machine =>
                                                                                                                         #{checkout_message_bytes => 0,
                                                                                                                          config =>
                                                                                                                            #{consumer_strategy => competing,
                                                                                                                              dead_lettering_enabled => false,
                                                                                                                              delivery_limit => undefined,
                                                                                                                              expires => undefined,max_bytes => undefined,
                                                                                                                              max_length => undefined,
                                                                                                                              msg_ttl => undefined,
                                                                                                                              name =>
                                                                                                                              '%2F_092f56e3-1277-4bcc-b5de-479d59e8a0a5',
                                                                                                                              release_cursor_interval => {2048,2048},
                                                                                                                              resource =>
                                                                                                                               {resource,<<"/">>,queue,
                                                                                                                                <<"092f56e3-1277-4bcc-b5de-479d59e8a0a5">>}},
                                                                                                                           discard_checkout_message_bytes => 0,
                                                                                                                           discard_message_bytes => 0,
                                                                                                                           enqueue_message_bytes => 0,
                                                                                                                           in_memory_message_bytes => 0,
                                                                                                                          num_checked_out => 0,num_consumers => 10,
                                                                                                                           num_discard_checked_out => 0,
                                                                                                                           num_discarded => 0,num_enqueuers => 4,
                                                                                                                           num_in_memory_ready_messages => 0,
                                                                                                                           num_messages => 0,num_ready_messages => 0,},
                                                                                                                        machine_version => 2,
                                                                                                                        uid => <<"2F_092IRAXC8BNGV2Y">>,
                                                                                                                        voted_for => undefined}}]
2024-03-14T15:45:36.246097117Z stdout F 2024-03-14 15:45:08.706447+00:00 [error] <0.2064.0> ** Reason for termination = error:{badmatch,
                                                                                                                               {not_found,
                                                                                                                                {ra_log,
                                                                                                                                 {cfg,<<"2F_092IRAXC8BNGV2Y">>,
                                                                                                                                  "queue '092f56e3-1277-4bcc-b5de-479d59e8a0a5' in vhost '/'",
                                                                                                                                  "/bitnami/rabbitmq/mnesia/[email protected]/quorum/[email protected]/2F_092IRAXC8BNGV2Y",
                                                                                                                                  8192,ra_log_snapshot,20,ra_log_wal,
                                                                                                                                  ra_log_segment_writer,
                                                                                                                                  {write_concurrency,
                                                                                                                                   #Ref<0.839822614.4015652865.199022>},
                                                                                                                                  #{closed_mem_tbls =>
                                                                                                                                     ra_log_closed_mem_tables,
                                                                                                                                    directory => ra_directory,
                                                                                                                                    directory_rev => ra_directory_reverse,
                                                                                                                                    log_ets => ra_log_ets,
                                                                                                                                    wal => ra_log_wal,
                                                                                                                                2600143,2600143,79,
                                                                                                                                 {2600142,79},
                                                                                                                                 {ra_snapshot,<<"2F_092IRAXC8BNGV2Y">>,
                                                                                                                                 ra_log_snapshot,
                                                                                                                                  "/bitnami/rabbitmq/mnesia/[email protected]/quorum/[email protected]/2F_092IRAXC8BNGV2Y/snapshots",
                                                                                                                                  undefined,undefined,
                                                                                                                                  {2600142,79}},
                                                                                                                                 #{2600143 =>
                                                                                                                                    {2600143,79,
                                                                                                                                     {'$usr',
                                                                                                                                      #{ts => 1710431108471},
                                                                                                                                      {enqueue,<0.8668.937>,9014,
                                                                                                                                       {basic_message,
                                                                                                                                        {resource,<<"/">>,exchange,<<>>},
                                                                                                                                        [<<"092f56e3-1277-4bcc-b5de-479d59e8a0a5">>],
                                                                                                                                        {content,60,none,
                                                                                                                                         <<152,0,16,97,112,112,108,105,99,
                                                                                                                                           97,116,105,111,110,47,106,115,
                                                                                                                                           111,110,2,0>>,
                                                                                                                                         rabbit_framing_amqp_0_9_1,
                                                                                                                                         [<<191,209,140,209,143,32,208,190,
                                                                                                                                            208,178,209,129,209,143,208,
                                                                                                                                            178,208,176,208,189,208,184,
                                                                                                                                            49,48,46,49,49,51,46,50,52,46,
                                                                                                                                            .................
2024-03-14T15:46:04.335079169Z stderr F Killed

[mkuratczyk]

RabbitMQ 3.10 is no longer supported. Step one would be to upgrade to 3.12/3.13.
https://www.rabbitmq.com/release-information