Issue creating Stream in RabbitMQ clustered mode

[hanyEthan]

Describe the bug

hi,

• we're having a clustered setup of 3 RabbitMQ instances, hosted in Kubernetes.
• our app is using Masstransit, and tries to create RabbitMQ Stream in the app initialization.
• the stream is being created using this queue argument : endpoint.SetQueueArgument("x-queue-type", "stream");
• in most cases, the stream is getting created successfully, and working seamlessly.
• but in other cases, especially if we restarted the RabbitMQ node instances, we start witnessing errors in one of the slave nodes, failing to initialize/replicate the stream queue.
• other quorum queues are created and replicated correctly among RMQ cluster nodes.
• the error we are getting is this the following ...

�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** Generic server <0.2000.0> terminating�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** Last message in was {'EXIT',<26154.2598.0>,noproc}�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** When Server state == #{committed_offset => -1,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                           external_reference =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                               {resource,<<"/">>,queue,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                   <<"vp.vc.Transactions.Cache.Invalidation.queue">>},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                           has_parse_state => false,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                           log =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                               #{directory =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                     "/var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/stream/__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150",�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                 file => "00000000000000000000.segment",�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                 max_segment_size_bytes => 500000000,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                 max_segment_size_chunks => 256000,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                 mode =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                     #{epoch => 1,mode => write,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                       tail => {0,empty},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                       type => acceptor},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                 retention => [],tracking_config => #{}},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                           name =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                               <<"__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150">>,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                           num_offset_listeners => 0}�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** Reason for termination ==�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** {function_clause,[{inet,tcp_close,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                            [undefined],�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                            [{file,"inet.erl"},{line,2179}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {osiris_replica,terminate,2,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                      [{file,"src/osiris_replica.erl"},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                       {line,580}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {gen_server,try_terminate,3,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                  [{file,"gen_server.erl"},{line,1161}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {gen_server,terminate,10,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                  [{file,"gen_server.erl"},{line,1351}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {proc_lib,init_p_do_apply,3,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                [{file,"proc_lib.erl"},{line,240}]}]}�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> �[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>   crasher:�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     initial call: osiris_replica:init/1�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     pid: <0.2000.0>�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     registered_name: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     exception error: no function clause matching inet:tcp_close(undefined) (inet.erl, line 2179)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       in function  osiris_replica:terminate/2 (src/osiris_replica.erl, line 580)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       in call from gen_server:try_terminate/3 (gen_server.erl, line 1161)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       in call from gen_server:terminate/10 (gen_server.erl, line 1351)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     ancestors: [osiris_server_sup,osiris_sup,<0.215.0>]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     message_queue_len: 0�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     messages: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     links: [<0.2002.0>,#Port<0.423>,<0.219.0>]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                                 next => #Fun<rand.0.34006561>,type => exsss,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                                 uniform => #Fun<rand.1.34006561>,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                                 uniform_n => #Fun<rand.2.34006561>},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                               [255910585991013159|136415235798472597]}}]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     trap_exit: true�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     status: running�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     heap_size: 10958�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     stack_size: 28�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     reductions: 47069�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>   neighbours:�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>     neighbour:�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       pid: <0.2002.0>�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       registered_name: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       initial_call: {erlang,apply,2}�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       current_function: {prim_inet,accept0,3}�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       ancestors: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       message_queue_len: 0�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       links: [<0.2000.0>]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       trap_exit: false�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       status: waiting�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       heap_size: 233�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       stack_size: 10�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       reductions: 72�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>       current_stacktrace: [{prim_inet,accept0,3,[]},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                   {inet_tcp,accept,1,[{file,"inet_tcp.erl"},{line,219}]},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                   {osiris_replica,accept,4,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                                   [{file,"src/osiris_replica.erl"},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0>                                    {line,312}]}]�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>     supervisor: {local,osiris_server_sup}�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>     errorContext: child_terminated�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>     reason: {function_clause,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                 [{inet,tcp_close,[undefined],[{file,"inet.erl"},{line,2179}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                  {osiris_replica,terminate,2,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                      [{file,"src/osiris_replica.erl"},{line,580}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                  {gen_server,try_terminate,3,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                      [{file,"gen_server.erl"},{line,1161}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                  {gen_server,terminate,10,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                      [{file,"gen_server.erl"},{line,1351}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                  {proc_lib,init_p_do_apply,3,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                      [{file,"proc_lib.erl"},{line,240}]}]}�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>     offender: [{pid,<0.2000.0>},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                {id,"__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150"},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                {mfargs,{osiris_replica,start_link,undefined}},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                {restart_type,temporary},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                {significant,false},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                {shutdown,5000},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0>                {child_type,worker}]�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> �[0m

• can you please help or suggest how can we solve this issue ?

Reproduction steps

1. start RabbitMQ cluster nodes.
2. start our app, creating queues & streams.
3. restart containers after maintenance events.
4. sometimes it works fine.
5. other times one of the RabbitMQ nodes starts to show error.
6. also master RabbitMQ node starts showing same errors.
7. the issue is not resolved anymore after we restart pods multiple times.
8. delete all queues of RabbitMQ
9. remove the streams from the list of queues maintained through Masstransit.
10. deploy and start the app again.
11. other quorum queues are created successfully, and replicated among all RMQ instances.
12. change the app to include the streams again,
13. after app deployment, the issue appears again.

Expected behavior

RabbitMQ streams should be maintained the same like other quorum queues, in RabbitMQ clustered mode.

Additional context

version : rabbitmq:3.11.18

[michaelklishin]

According to

�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** {function_clause,[{inet,tcp_close,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                            [undefined],�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                            [{file,"inet.erl"},{line,2179}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {osiris_replica,terminate,2,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                      [{file,"src/osiris_replica.erl"},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                       {line,580}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {gen_server,try_terminate,3,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                  [{file,"gen_server.erl"},{line,1161}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {gen_server,terminate,10,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                  [{file,"gen_server.erl"},{line,1351}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                      {proc_lib,init_p_do_apply,3,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0>                                [{file,"proc_lib.erl"},{line,240}]}]}�[0m

a stream replica tries to perform an operation on an unintialized socket. This was addressed in rabbitmq/osiris#127. Upgrading to 3.12.4 should make this exception go away.

[michaelklishin]

RabbitMQ 3.11.22 also includes the but please upgrade to 3.12 as your earliest convenience.

[hanyEthan]

thank you, let me try and return back to you.

[kjnilsson]

Is this actually causing an issue with any of the external functionality of RabbitMQ or are you just observing this as an error in the logs?

[hanyEthan]

it's causing our apps failing to listen or create queues in RabbitMQ clustered instances as long as this error persists.

[kjnilsson]

Can we see a bit more of the log around these events. Ideally at debug level.

[hanyEthan]

hi, sure ...

2023-08-31 12:49:26.300986+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2023-08-31 12:49:26.312502+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
�[38;5;87m2023-08-31 12:49:26.334705+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE�[0m
�[38;5;87m2023-08-31 12:49:31.686450+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped�[0m
�[38;5;87m2023-08-31 12:49:31.686789+00:00 [notice] <0.230.0> Feature flags: checking nodes rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01 and rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01 compatibility...�[0m
�[38;5;87m2023-08-31 12:49:31.701738+00:00 [notice] <0.230.0> Feature flags: nodes rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01 and rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01 are compatible�[0m
�[38;5;160m2023-08-31 12:49:31.722241+00:00 [error] <0.293.0> Mnesia('rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'}�[0m
�[38;5;160m2023-08-31 12:49:31.722241+00:00 [error] <0.293.0> �[0m
�[38;5;160m2023-08-31 12:49:31.722415+00:00 [error] <0.293.0> Mnesia('rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'rabbit@rc-rabbitmq-instance-server-1.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'}�[0m
�[38;5;160m2023-08-31 12:49:31.722415+00:00 [error] <0.293.0> �[0m
2023-08-31 12:49:31.740586+00:00 [info] <0.230.0> ra: starting system quorum_queues�[0m
2023-08-31 12:49:31.740694+00:00 [info] <0.230.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/quorum/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01�[0m
2023-08-31 12:49:31.743503+00:00 [info] <0.333.0> ra system 'quorum_queues' running pre init for 0 registered servers�[0m
2023-08-31 12:49:31.746169+00:00 [info] <0.334.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered�[0m
�[38;5;87m2023-08-31 12:49:31.746438+00:00 [notice] <0.339.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables�[0m
2023-08-31 12:49:31.756546+00:00 [info] <0.230.0> ra: starting system coordination�[0m
2023-08-31 12:49:31.756687+00:00 [info] <0.230.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/coordination/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01�[0m
2023-08-31 12:49:31.758704+00:00 [info] <0.351.0> ra system 'coordination' running pre init for 1 registered servers�[0m
2023-08-31 12:49:31.766619+00:00 [info] <0.352.0> ra: meta data store initialised for system coordination. 1 record(s) recovered�[0m
�[38;5;87m2023-08-31 12:49:31.766939+00:00 [notice] <0.357.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables�[0m
2023-08-31 12:49:31.812360+00:00 [info] <0.230.0> �[0m
2023-08-31 12:49:31.812360+00:00 [info] <0.230.0> Starting RabbitMQ 3.11.18 on Erlang 25.3.2.2 [jit]�[0m
2023-08-31 12:49:31.812360+00:00 [info] <0.230.0> Copyright (c) 2007-2023 VMware, Inc. or its affiliates.�[0m
2023-08-31 12:49:31.812360+00:00 [info] <0.230.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com�[0m

## RabbitMQ 3.11.18

########## Copyright (c) 2007-2023 VMware, Inc. or its affiliates.

########## Licensed under the MPL 2.0. Website: https://rabbitmq.com

Erlang: 25.3.2.2 [jit]
TLS Library: OpenSSL - OpenSSL 3.1.1 30 May 2023
Release series support status: supported

Doc guides: https://rabbitmq.com/documentation.html
Support: https://rabbitmq.com/contact.html
Tutorials: https://rabbitmq.com/getstarted.html
Monitoring: https://rabbitmq.com/monitoring.html

Logs: /var/log/rabbitmq/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01_upgrade.log

Config file(s): /etc/rabbitmq/conf.d/10-defaults.conf
/etc/rabbitmq/conf.d/10-operatorDefaults.conf
/etc/rabbitmq/conf.d/11-default_user.conf
/etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf

Starting broker...2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> �[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> node : rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> home dir : /var/lib/rabbitmq�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> config file(s) : /etc/rabbitmq/conf.d/10-defaults.conf�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> : /etc/rabbitmq/conf.d/10-operatorDefaults.conf�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> : /etc/rabbitmq/conf.d/11-default_user.conf�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> : /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> cookie hash : 0J5Dsbmg4b80ktYUNH5cng==�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> log(s) : /var/log/rabbitmq/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01_upgrade.log�[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> : �[0m
2023-08-31 12:49:31.830797+00:00 [info] <0.230.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01�[0m
2023-08-31 12:49:33.796814+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit�[0m
2023-08-31 12:49:33.796925+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit�[0m
2023-08-31 12:49:33.797236+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit�[0m
2023-08-31 12:49:33.797369+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit�[0m
2023-08-31 12:49:33.798104+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit�[0m
2023-08-31 12:49:33.805196+00:00 [info] <0.458.0> Memory high watermark set to 327 MiB (343597384 bytes) of 819 MiB (858993460 bytes) total�[0m
2023-08-31 12:49:33.811120+00:00 [info] <0.460.0> Enabling free disk space monitoring (disk free space: 10446766080, total memory: 858993460)�[0m
2023-08-31 12:49:33.811262+00:00 [info] <0.460.0> Disk free limit set to 2000MB�[0m
2023-08-31 12:49:33.814331+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit�[0m
2023-08-31 12:49:33.814483+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit�[0m
2023-08-31 12:49:33.814730+00:00 [info] <0.463.0> Limiting to approx 1048479 file handles (943629 sockets)�[0m
2023-08-31 12:49:33.814852+00:00 [info] <0.464.0> FHC read buffering: OFF�[0m
2023-08-31 12:49:33.814897+00:00 [info] <0.464.0> FHC write buffering: ON�[0m
2023-08-31 12:49:33.815406+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit�[0m
2023-08-31 12:49:33.815513+00:00 [info] <0.370.0> Will use 8 processes for default worker pool�[0m
2023-08-31 12:49:33.815574+00:00 [info] <0.370.0> Starting worker pool 'worker_pool' with 8 processes in it�[0m
2023-08-31 12:49:33.815981+00:00 [info] <0.230.0> Running boot step database defined by app rabbit�[0m
2023-08-31 12:49:33.843502+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left�[0m
2023-08-31 12:49:33.843747+00:00 [info] <0.230.0> Successfully synced tables from a peer�[0m
2023-08-31 12:49:33.843843+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left�[0m
2023-08-31 12:49:33.843961+00:00 [info] <0.230.0> Successfully synced tables from a peer�[0m
2023-08-31 12:49:33.858660+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left�[0m
2023-08-31 12:49:33.858890+00:00 [info] <0.230.0> Successfully synced tables from a peer�[0m
2023-08-31 12:49:33.859032+00:00 [info] <0.230.0> Will register with peer discovery backend rabbit_peer_discovery_k8s�[0m
�[38;5;214m2023-08-31 12:49:33.861448+00:00 [warning] <0.486.0> Description: "Server authenticity is not verified since certificate path validation is not enabled"�[0m
�[38;5;214m2023-08-31 12:49:33.861448+00:00 [warning] <0.486.0> Reason: "The option {verify, verify_peer} and one of the options 'cacertfile' or 'cacerts' are required to enable this."�[0m
�[38;5;214m2023-08-31 12:49:33.861448+00:00 [warning] <0.486.0> �[0m
2023-08-31 12:49:33.883801+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit�[0m
2023-08-31 12:49:33.884107+00:00 [info] <0.493.0> Setting up a table for connection tracking on this node: tracked_connection�[0m
2023-08-31 12:49:33.884185+00:00 [info] <0.493.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost�[0m
2023-08-31 12:49:33.884238+00:00 [info] <0.493.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user�[0m
2023-08-31 12:49:33.884297+00:00 [info] <0.493.0> Setting up a table for channel tracking on this node: tracked_channel�[0m
2023-08-31 12:49:33.884352+00:00 [info] <0.493.0> Setting up a table for channel tracking on this node: tracked_channel_per_user�[0m
2023-08-31 12:49:33.884517+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit�[0m
2023-08-31 12:49:33.884643+00:00 [info] <0.230.0> Running boot step database_sync defined by app rabbit�[0m
2023-08-31 12:49:33.884717+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit�[0m
2023-08-31 12:49:33.884922+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit�[0m
2023-08-31 12:49:33.884969+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit�[0m
2023-08-31 12:49:33.884995+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit�[0m
2023-08-31 12:49:33.885126+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit�[0m
2023-08-31 12:49:33.885216+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit�[0m
2023-08-31 12:49:33.885350+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit�[0m
2023-08-31 12:49:33.885438+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit�[0m
2023-08-31 12:49:33.885518+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit�[0m
2023-08-31 12:49:33.885633+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit�[0m
2023-08-31 12:49:33.885719+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit�[0m
2023-08-31 12:49:33.885799+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit�[0m
2023-08-31 12:49:33.885886+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit�[0m
2023-08-31 12:49:33.885965+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit�[0m
2023-08-31 12:49:33.886030+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit�[0m
2023-08-31 12:49:33.886098+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit�[0m
2023-08-31 12:49:33.886137+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue�[0m
2023-08-31 12:49:33.886254+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit�[0m
2023-08-31 12:49:33.886328+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit�[0m
2023-08-31 12:49:33.886416+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit�[0m
2023-08-31 12:49:33.886504+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit�[0m
2023-08-31 12:49:33.886548+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit�[0m
2023-08-31 12:49:33.886721+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit�[0m
2023-08-31 12:49:33.888910+00:00 [info] <0.502.0> epmd monitor knows us, inter-node communication (distribution) port: 25672�[0m
2023-08-31 12:49:33.889055+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit�[0m
2023-08-31 12:49:33.890990+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit�[0m
2023-08-31 12:49:33.893455+00:00 [info] <0.506.0> Starting rabbit_node_monitor�[0m
2023-08-31 12:49:33.893618+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit�[0m
2023-08-31 12:49:33.894070+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit�[0m
2023-08-31 12:49:33.894247+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit�[0m
2023-08-31 12:49:33.894338+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit�[0m
2023-08-31 12:49:33.894385+00:00 [info] <0.230.0> Running boot step upgrade_queues defined by app rabbit�[0m
2023-08-31 12:49:33.904739+00:00 [info] <0.230.0> Running boot step channel_tracking defined by app rabbit�[0m
2023-08-31 12:49:33.904851+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit�[0m
2023-08-31 12:49:33.904931+00:00 [info] <0.230.0> Running boot step connection_tracking defined by app rabbit�[0m
2023-08-31 12:49:33.904959+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit�[0m
2023-08-31 12:49:33.905015+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit�[0m
2023-08-31 12:49:33.905115+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit�[0m
2023-08-31 12:49:33.905191+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit�[0m
2023-08-31 12:49:33.905528+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit�[0m
2023-08-31 12:49:33.905767+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit�[0m
2023-08-31 12:49:33.905858+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit�[0m
2023-08-31 12:49:33.905923+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit�[0m
2023-08-31 12:49:33.905976+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit�[0m
2023-08-31 12:49:33.910661+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit�[0m
2023-08-31 12:49:33.910887+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management�[0m
2023-08-31 12:49:33.911379+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent�[0m
2023-08-31 12:49:33.911439+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'�[0m
2023-08-31 12:49:33.919031+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit�[0m
2023-08-31 12:49:33.921382+00:00 [info] <0.542.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists�[0m
2023-08-31 12:49:33.925708+00:00 [info] <0.542.0> Starting message stores for vhost '/'�[0m
2023-08-31 12:49:33.926034+00:00 [info] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index�[0m
2023-08-31 12:49:33.929064+00:00 [info] <0.542.0> Started message store of type transient for vhost '/'�[0m
2023-08-31 12:49:33.929358+00:00 [info] <0.551.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index�[0m
2023-08-31 12:49:33.931929+00:00 [info] <0.542.0> Started message store of type persistent for vhost '/'�[0m
2023-08-31 12:49:33.932110+00:00 [info] <0.542.0> Recovering 0 queues of type rabbit_classic_queue took 9ms�[0m
2023-08-31 12:49:33.932169+00:00 [info] <0.542.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms�[0m
2023-08-31 12:49:33.932203+00:00 [info] <0.542.0> Recovering 0 queues of type rabbit_stream_queue took 0ms�[0m
2023-08-31 12:49:33.934046+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit�[0m
2023-08-31 12:49:33.934150+00:00 [info] <0.230.0> Will not seed default virtual host and user: have definitions to load...�[0m
2023-08-31 12:49:33.934184+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit�[0m
2023-08-31 12:49:33.934283+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit�[0m
2023-08-31 12:49:33.934331+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit�[0m
2023-08-31 12:49:33.934500+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit�[0m
2023-08-31 12:49:33.934647+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit�[0m
2023-08-31 12:49:33.934679+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit�[0m
2023-08-31 12:49:33.934710+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit�[0m
2023-08-31 12:49:33.934735+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit�[0m
2023-08-31 12:49:33.934860+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit�[0m
2023-08-31 12:49:33.934946+00:00 [info] <0.370.0> Starting worker pool 'definition_import_pool' with 8 processes in it�[0m
2023-08-31 12:49:33.935751+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit�[0m
2023-08-31 12:49:33.935814+00:00 [info] <0.230.0> Setting cluster name to 'rc-rabbitmq-instance' as configured�[0m
2023-08-31 12:49:33.936097+00:00 [info] <0.506.0> rabbit on node 'rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01' up�[0m
2023-08-31 12:49:33.940928+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit�[0m
2023-08-31 12:49:33.941097+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit�[0m
2023-08-31 12:49:33.941166+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status�[0m
2023-08-31 12:49:33.942197+00:00 [info] <0.230.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management�[0m
2023-08-31 12:49:33.942371+00:00 [info] <0.591.0> Resetting node maintenance status�[0m
2023-08-31 12:49:33.942772+00:00 [info] <0.506.0> rabbit on node 'rabbit@rc-rabbitmq-instance-server-1.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01' up�[0m
2023-08-31 12:49:33.981357+00:00 [info] <0.651.0> Management plugin: HTTP (non-TLS) listener started on port 15672�[0m
2023-08-31 12:49:33.981625+00:00 [info] <0.679.0> Statistics database started.�[0m
2023-08-31 12:49:33.981729+00:00 [info] <0.678.0> Starting worker pool 'management_worker_pool' with 3 processes in it�[0m
2023-08-31 12:49:33.991263+00:00 [info] <0.694.0> Peer discovery: node cleanup is disabled�[0m
2023-08-31 12:49:33.994704+00:00 [info] <0.703.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692�[0m
2023-08-31 12:49:33.994936+00:00 [info] <0.591.0> Ready to start client connection listeners�[0m
2023-08-31 12:49:33.998046+00:00 [info] <0.747.0> started TCP listener on [::]:5672�[0m
completed with 6 plugins.
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> Server startup complete; 6 plugins started.�[0m
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> * rabbitmq_prometheus�[0m
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> * rabbitmq_peer_discovery_k8s�[0m
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> * rabbitmq_peer_discovery_common�[0m
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> * rabbitmq_management�[0m
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> * rabbitmq_web_dispatch�[0m
2023-08-31 12:49:34.096257+00:00 [info] <0.591.0> * rabbitmq_management_agent�[0m
2023-08-31 12:49:34.336299+00:00 [info] <0.528.0> rabbit_stream_coordinator: follower did not have entry at 8705 in 2. Requesting {rabbit_stream_coordinator,'rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'} from 8704�[0m
2023-08-31 12:49:34.336731+00:00 [info] <0.528.0> rabbit_stream_coordinator: detected a new leader {rabbit_stream_coordinator,'rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'} in term 3�[0m
2023-08-31 12:58:56.470779+00:00 [info] <0.1128.0> queue 'vp.vc.staticdata.transactions.queue' in vhost '/': granting vote for {'%2F_vp.vc.staticdata.transactions.queue','rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'} with last indexterm {0,0} for term 1 previous term was 0�[0m
2023-08-31 12:58:56.474778+00:00 [info] <0.1128.0> queue 'vp.vc.staticdata.transactions.queue' in vhost '/': detected a new leader {'%2F_vp.vc.staticdata.transactions.queue','rabbit@rc-rabbitmq-instance-server-0.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01'} in term 1�[0m
2023-08-31 13:12:14.616697+00:00 [info] <0.2000.0> Stream: __vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150 will use /var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/stream/__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150 for osiris log data directory�[0m
2023-08-31 13:12:14.618888+00:00 [info] <0.2000.0> __vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150 [osiris_replica:handle_continue/2] osiris replica starting in epoch 1, next offset 0, tail info {0,empty}�[0m
�[38;5;160m2023-08-31 13:12:14.646936+00:00 [error] <0.2000.0> __vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150 [osiris_replica:handle_info/2] replica reader <26154.2598.0> exited with noproc�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** Generic server <0.2000.0> terminating�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** Last message in was {'EXIT',<26154.2598.0>,noproc}�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** When Server state == #{committed_offset => -1,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> external_reference =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> {resource,<<"/">>,queue,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> <<"vp.vc.Transactions.Cache.Invalidation.queue">>},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> has_parse_state => false,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> log =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> #{directory =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> "/var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/stream/__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150",�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> file => "00000000000000000000.segment",�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> max_segment_size_bytes => 500000000,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> max_segment_size_chunks => 256000,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> mode =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> #{epoch => 1,mode => write,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> tail => {0,empty},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> type => acceptor},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> retention => [],tracking_config => #{}},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> name =>�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> <<"__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150">>,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> num_offset_listeners => 0}�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** Reason for termination ==�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> ** {function_clause,[{inet,tcp_close,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> [undefined],�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> [{file,"inet.erl"},{line,2179}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> {osiris_replica,terminate,2,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> [{file,"src/osiris_replica.erl"},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> {line,580}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> {gen_server,try_terminate,3,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> [{file,"gen_server.erl"},{line,1161}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> {gen_server,terminate,10,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> [{file,"gen_server.erl"},{line,1351}]},�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> {proc_lib,init_p_do_apply,3,�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> [{file,"proc_lib.erl"},{line,240}]}]}�[0m
�[38;5;160m2023-08-31 13:12:14.647454+00:00 [error] <0.2000.0> �[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> crasher:�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> initial call: osiris_replica:init/1�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> pid: <0.2000.0>�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> registered_name: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> exception error: no function clause matching inet:tcp_close(undefined) (inet.erl, line 2179)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> in function osiris_replica:terminate/2 (src/osiris_replica.erl, line 580)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> in call from gen_server:try_terminate/3 (gen_server.erl, line 1161)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> in call from gen_server:terminate/10 (gen_server.erl, line 1351)�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> ancestors: [osiris_server_sup,osiris_sup,<0.215.0>]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> message_queue_len: 0�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> messages: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> links: [<0.2002.0>,#Port<0.423>,<0.219.0>]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> next => #Fun<rand.0.34006561>,type => exsss,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> uniform => #Fun<rand.1.34006561>,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> uniform_n => #Fun<rand.2.34006561>},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> [255910585991013159|136415235798472597]}}]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> trap_exit: true�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> status: running�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> heap_size: 10958�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> stack_size: 28�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> reductions: 47069�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> neighbours:�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> neighbour:�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> pid: <0.2002.0>�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> registered_name: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> initial_call: {erlang,apply,2}�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> current_function: {prim_inet,accept0,3}�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> ancestors: []�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> message_queue_len: 0�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> links: [<0.2000.0>]�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> trap_exit: false�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> status: waiting�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> heap_size: 233�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> stack_size: 10�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> reductions: 72�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> current_stacktrace: [{prim_inet,accept0,3,[]},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> {inet_tcp,accept,1,[{file,"inet_tcp.erl"},{line,219}]},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> {osiris_replica,accept,4,�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> [{file,"src/osiris_replica.erl"},�[0m
�[38;5;160m2023-08-31 13:12:14.648414+00:00 [error] <0.2000.0> {line,312}]}]�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> supervisor: {local,osiris_server_sup}�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> errorContext: child_terminated�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> reason: {function_clause,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> [{inet,tcp_close,[undefined],[{file,"inet.erl"},{line,2179}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {osiris_replica,terminate,2,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> [{file,"src/osiris_replica.erl"},{line,580}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {gen_server,try_terminate,3,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> [{file,"gen_server.erl"},{line,1161}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {gen_server,terminate,10,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> [{file,"gen_server.erl"},{line,1351}]},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {proc_lib,init_p_do_apply,3,�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> [{file,"proc_lib.erl"},{line,240}]}]}�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> offender: [{pid,<0.2000.0>},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {id,"__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150"},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {mfargs,{osiris_replica,start_link,undefined}},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {restart_type,temporary},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {significant,false},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {shutdown,5000},�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> {child_type,worker}]�[0m
�[38;5;160m2023-08-31 13:12:14.649348+00:00 [error] <0.219.0> �[0m
2023-08-31 13:12:14.657584+00:00 [info] <0.2003.0> Stream: __vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150 will use /var/lib/rabbitmq/mnesia/rabbit@rc-rabbitmq-instance-server-2.rc-rabbitmq-instance-nodes.vrp-ns-rnd-vc-01/stream/__vp_vc_Transactions_Cache_Invalidation_queue_1693487534576630150 for osiris log data directory�[0m