Exception in queue index recovery

[haloxinyu]

Dear RabbitMQ team,
I'm currently upgrading RabbitMQ version to 3.10.7 and testing it on single node, it works fine for the few days. But after I tried to restart the node, I found that rabbimq crashed at "rebuilding indices from scratch" and could not be started.

System info
RabbitMQ: v3.10.7
Erlang: v24.3.4.1

Error logs
`2023-02-06 18:50:12.863006-08:00 [info] <0.426.0> Starting message stores for vhost '/'

2023-02-06 18:50:12.863400-08:00 [info] <0.431.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2023-02-06 18:50:12.865259-08:00 [info] <0.426.0> Started message store of type transient for vhost '/'
2023-02-06 18:50:12.865583-08:00 [info] <0.435.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2023-02-06 18:50:12.866183-08:00 [warning] <0.435.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> ** Generic server <0.364.0> terminating
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> ** Last message in was {'$gen_cast',
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {submit_async,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> #Fun<rabbit_classic_queue_index_v2.11.65393203>,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> <0.357.0>}}
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> ** When Server state == undefined
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> ** Reason for termination ==
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> ** {{case_clause,undefined},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {rabbit_queue_index,segment_entries_foldr,3,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1234}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {rabbit_queue_index,scan_queue_segments,4,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,864}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {rabbit_queue_index,queue_index_walker_reader,2,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,844}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {rabbit_classic_queue_index_v2,'-queue_index_walker/1-fun-2-',2,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"rabbit_classic_queue_index_v2.erl"},{line,1104}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {worker_pool_worker,handle_cast,2,
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> [{file,"worker_pool_worker.erl"},{line,113}]},
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0> {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]}]}
2023-02-06 18:50:12.871456-08:00 [error] <0.364.0>
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> crasher:
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> initial call: worker_pool_worker:init/1
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> pid: <0.364.0>
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> registered_name: []
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> exception exit: {{case_clause,undefined},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {rabbit_queue_index,segment_entries_foldr,3,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1234}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {rabbit_queue_index,scan_queue_segments,4,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,864}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {rabbit_queue_index,queue_index_walker_reader,2,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,844}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {rabbit_classic_queue_index_v2,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> '-queue_index_walker/1-fun-2-',2,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"rabbit_classic_queue_index_v2.erl"},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {line,1104}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {worker_pool_worker,handle_cast,2,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"worker_pool_worker.erl"},{line,113}]},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {gen_server2,handle_msg,2,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{file,"gen_server2.erl"},{line,1067}]}]}
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183)
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> ancestors: [worker_pool_sup,rabbit_sup,<0.220.0>]
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> message_queue_len: 0
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> messages: []
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> links: [<0.356.0>]
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> dictionary: [{segment_entry_count,2048},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {{"/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> fhc_file},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {file,1,true}},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {worker_pool_worker,true},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {fhc_age_tree,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {1,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {{-576460746799512580,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> #Ref<0.3178237416.3131834373.122149>},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> true,nil,nil}}},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {{#Ref<0.3178237416.3131834373.122149>,fhc_handle},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {handle,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {file_descriptor,prim_file,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> #{handle => #Ref<0.3178237416.3131965464.122432>,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> owner => <0.364.0>,r_ahead_size => 0,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> r_buffer =>
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> #Ref<0.3178237416.3131965441.124499>}},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> #Ref<0.3178237416.3131834373.122149>,0,false,0,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> infinity,[],<<>>,0,0,0,0,0,false,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> "/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [write,binary,raw,read],
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [{write_buffer,infinity}],
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> true,true,-576460746799512580}},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {rand_seed,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {#{jump => #Fun<rand.3.92093067>,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> max => 288230376151711743,
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> next => #Fun<rand.5.92093067>,type => exsplus},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> [201068679141252428|15188688121591671]}},
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> {worker_pool_name,worker_pool}]
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> trap_exit: false
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> status: running
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> heap_size: 6772
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> stack_size: 28
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> reductions: 96947
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0> neighbours:
2023-02-06 18:50:12.872835-08:00 [error] <0.364.0>
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> supervisor: {local,worker_pool_sup}
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> errorContext: child_terminated
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> reason: {{case_clause,undefined},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {rabbit_queue_index,segment_entries_foldr,3,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"rabbit_queue_index.erl"},{line,1234}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {rabbit_queue_index,scan_queue_segments,4,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"rabbit_queue_index.erl"},{line,864}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {rabbit_queue_index,queue_index_walker_reader,2,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"rabbit_queue_index.erl"},{line,844}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {rabbit_classic_queue_index_v2,'-queue_index_walker/1-fun-2-',
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> 2,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"rabbit_classic_queue_index_v2.erl"},{line,1104}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {worker_pool_worker,handle_cast,2,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"worker_pool_worker.erl"},{line,113}]},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {gen_server2,handle_msg,2,
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> [{file,"gen_server2.erl"},{line,1067}]}]}
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> offender: [{pid,<0.364.0>},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {id,7},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {mfargs,{worker_pool_worker,start_link,[worker_pool]}},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {restart_type,transient},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {significant,false},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {shutdown,4294967295},
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0> {child_type,worker}]
2023-02-06 18:50:12.874515-08:00 [error] <0.356.0>
`
Question:
I have encounted the issue servel times after upgrading rabbimq to 3.10.7, and not seen it in previous versions. Can someone help to analyze the rabbitmq crashed issue?

[michaelklishin]

There is no evidence of a node termination in this log. There was an exception in rabbit_queue_index :add_segment_relseq_entry/3.

Nodes have to rebuild indices when they were not given a chance to persist all data during shutdown.

Quorum queues, streams, classic queues v2

classic_queue.default_version = 2

won't run into this because they do not use the function in question.

[haloxinyu]

Below logs contains rabbitmq termination and crashed after restarting

`2023-02-05 18:57:11.266832-08:00 [info] <0.11209.4> RabbitMQ is asked to stop...

2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> Stopping RabbitMQ applications and their dependencies in the following order:
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbitmq_management
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> amqp_client
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbitmq_web_dispatch
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> cowboy
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> cowlib
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbitmq_management_agent
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbitmq_auth_mechanism_ssl
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbit
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbitmq_prelaunch
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> rabbit_common
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> sysmon_handler
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> os_mon
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> osiris
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> ra
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4> mnesia
2023-02-05 18:57:11.313985-08:00 [info] <0.11209.4>
2023-02-05 18:57:11.314141-08:00 [info] <0.11209.4> Stopping application 'rabbitmq_management'
2023-02-05 18:57:11.337588-08:00 [warning] <0.768.0> HTTP listener registry could not find context rabbitmq_management_tls
2023-02-05 18:57:11.351219-08:00 [notice] <0.44.0> Application rabbitmq_management exited with reason: stopped
2023-02-05 18:57:11.351367-08:00 [info] <0.11209.4> Stopping application 'amqp_client'
2023-02-05 18:57:11.354452-08:00 [notice] <0.44.0> Application amqp_client exited with reason: stopped
2023-02-05 18:57:11.354564-08:00 [info] <0.11209.4> Stopping application 'rabbitmq_web_dispatch'
2023-02-05 18:57:11.358438-08:00 [notice] <0.44.0> Application rabbitmq_web_dispatch exited with reason: stopped
2023-02-05 18:57:11.358551-08:00 [info] <0.11209.4> Stopping application 'cowboy'
2023-02-05 18:57:11.360966-08:00 [notice] <0.44.0> Application cowboy exited with reason: stopped
2023-02-05 18:57:11.361057-08:00 [info] <0.11209.4> Stopping application 'cowlib'
2023-02-05 18:57:11.361127-08:00 [notice] <0.44.0> Application cowlib exited with reason: stopped
2023-02-05 18:57:11.361205-08:00 [info] <0.11209.4> Stopping application 'rabbitmq_management_agent'
2023-02-05 18:57:11.371317-08:00 [notice] <0.44.0> Application rabbitmq_management_agent exited with reason: stopped
2023-02-05 18:57:11.371521-08:00 [info] <0.11209.4> Stopping application 'rabbitmq_auth_mechanism_ssl'
2023-02-05 18:57:11.373670-08:00 [notice] <0.44.0> Application rabbitmq_auth_mechanism_ssl exited with reason: stopped
2023-02-05 18:57:11.373762-08:00 [info] <0.11209.4> Stopping application 'rabbit'
2023-02-05 18:57:11.378302-08:00 [info] <0.220.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping unregistration.
2023-02-05 18:57:11.384795-08:00 [info] <0.852.0> stopped TLS (SSL) listener on [::]:5671
2023-02-05 18:57:11.402454-08:00 [error] <0.1908.0> Error on AMQP connection <0.1908.0> (127.0.0.1:47846 -> 127.0.0.1:5671 - MonitoringServer#70e8d481:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.402454-08:00 [error] <0.1908.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.404304-08:00 [error] <0.924.0> Error on AMQP connection <0.924.0> (127.0.0.1:47792 -> 127.0.0.1:5671 - LogManager#1496fc38:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.404304-08:00 [error] <0.924.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.405209-08:00 [error] <0.2500.0> Error on AMQP connection <0.2500.0> (127.0.0.1:47866 -> 127.0.0.1:5671 - ServerDisasterRecovery#48a875b3:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.405209-08:00 [error] <0.2500.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.405766-08:00 [error] <0.1679.0> Error on AMQP connection <0.1679.0> (127.0.0.1:47836 -> 127.0.0.1:5671 - ReplicationManager#62956310:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.405766-08:00 [error] <0.1679.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.405932-08:00 [error] <0.2276.0> Error on AMQP connection <0.2276.0> (127.0.0.1:47850 -> 127.0.0.1:5671 - VMwarePlugin#2ee5c08d:20, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.405932-08:00 [error] <0.2276.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406013-08:00 [error] <0.1417.0> Error on AMQP connection <0.1417.0> (127.0.0.1:47818 -> 127.0.0.1:5671 - ADM Component Service#13be1c30:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406013-08:00 [error] <0.1417.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406030-08:00 [error] <0.1109.0> Error on AMQP connection <0.1109.0> (127.0.0.1:47794 -> 127.0.0.1:5671 - VMwareService#1f4de96e:14, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406030-08:00 [error] <0.1109.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406118-08:00 [error] <0.867.0> Error on AMQP connection <0.867.0> (127.0.0.1:47776 -> 127.0.0.1:5671 - SysMgr#2009b81a:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406118-08:00 [error] <0.867.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406148-08:00 [error] <0.897.0> Error on AMQP connection <0.897.0> (127.0.0.1:47788 -> 127.0.0.1:5671 - AAA Component Service#5ee1a359:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406148-08:00 [error] <0.897.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406224-08:00 [error] <0.893.0> Error on AMQP connection <0.893.0> (127.0.0.1:47786 -> 127.0.0.1:5671 - telmgr#d22747f:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406224-08:00 [error] <0.893.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406234-08:00 [error] <0.2527.0> Error on AMQP connection <0.2527.0> (127.0.0.1:47868 -> 127.0.0.1:5671 - NASDM Component Service#e5ca0bd:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406234-08:00 [error] <0.2527.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406293-08:00 [error] <0.903.0> Error on AMQP connection <0.903.0> (172.24.0.253:56474 -> 172.24.0.193:5671 - complianceverification#74f8dc1b:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406293-08:00 [error] <0.903.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406265-08:00 [error] <0.1200.0> Error on AMQP connection <0.1200.0> (127.0.0.1:47810 -> 127.0.0.1:5671 - CNDM#4d0c609:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406265-08:00 [error] <0.1200.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406292-08:00 [error] <0.2484.0> Error on AMQP connection <0.2484.0> (127.0.0.1:47854 -> 127.0.0.1:5671 - CommonBusinessService#3343f1db:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406292-08:00 [error] <0.2484.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406517-08:00 [error] <0.2655.0> Error on AMQP connection <0.2655.0> (127.0.0.1:47876 -> 127.0.0.1:5671 - discoveryservice#222136e9:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406517-08:00 [error] <0.2655.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.404467-08:00 [error] <0.1147.0> Error on AMQP connection <0.1147.0> (172.24.0.251:52160 -> 172.24.0.193:5671 - Scheduler#3905b66:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.404467-08:00 [error] <0.1147.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406100-08:00 [error] <0.1433.0> Error on AMQP connection <0.1433.0> (127.0.0.1:47824 -> 127.0.0.1:5671 - VMDM#6cef2367:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406100-08:00 [error] <0.1433.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.405974-08:00 [error] <0.1140.0> Error on AMQP connection <0.1140.0> (172.24.0.252:42052 -> 172.24.0.193:5671 - PolicyService#78ceb0a4:38, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.405974-08:00 [error] <0.1140.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406226-08:00 [error] <0.1617.0> Error on AMQP connection <0.1617.0> (127.0.0.1:47828 -> 127.0.0.1:5671 - CloudManager#64a27d27:68, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406226-08:00 [error] <0.1617.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.406517-08:00 [error] <0.1471.0> Error on AMQP connection <0.1471.0> (172.24.0.250:57018 -> 172.24.0.193:5671 - workflow#276f5549:608, vhost: '/', user: 'dpswq063.drm.lab.emc.com', state: running), channel 0:
2023-02-05 18:57:11.406517-08:00 [error] <0.1471.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.511497-08:00 [info] <0.832.0> stopped TCP listener on [::]:5672
2023-02-05 18:57:11.515160-08:00 [error] <0.2869.0> Error on AMQP connection <0.2869.0> (172.24.0.251:42698 -> 172.24.0.193:5672 - scheduler-reactive-sender, vhost: '/', user: 'guest', state: running), channel 0:
2023-02-05 18:57:11.515160-08:00 [error] <0.2869.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-02-05 18:57:11.519148-08:00 [info] <0.11929.4> Closing all connections in vhost '/' on node 'rabbit@dpswq063' because the vhost is stopping
2023-02-05 18:57:11.588982-08:00 [info] <0.441.0> Stopping message store for directory '/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2023-02-05 18:57:11.611119-08:00 [info] <0.441.0> Message store for directory '/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2023-02-05 18:57:11.611345-08:00 [info] <0.437.0> Stopping message store for directory '/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2023-02-05 18:57:11.615218-08:00 [info] <0.437.0> Message store for directory '/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2023-02-05 18:57:11.628143-08:00 [info] <0.385.0> Management plugin: to stop collect_statistics.
2023-02-05 18:57:16.647716-08:00 [notice] <0.44.0> Application rabbit exited with reason: stopped
2023-02-05 18:57:16.647858-08:00 [info] <0.11209.4> Stopping application 'rabbitmq_prelaunch'
2023-02-05 18:57:16.650018-08:00 [notice] <0.44.0> Application rabbitmq_prelaunch exited with reason: stopped
2023-02-05 18:57:16.650136-08:00 [info] <0.11209.4> Stopping application 'rabbit_common'
2023-02-05 18:57:16.650202-08:00 [notice] <0.44.0> Application rabbit_common exited with reason: stopped
2023-02-05 18:57:16.650312-08:00 [info] <0.11209.4> Stopping application 'sysmon_handler'
2023-02-05 18:57:16.652419-08:00 [notice] <0.44.0> Application sysmon_handler exited with reason: stopped
2023-02-05 18:57:16.652510-08:00 [info] <0.11209.4> Stopping application 'os_mon'
2023-02-05 18:57:16.656037-08:00 [notice] <0.44.0> Application os_mon exited with reason: stopped
2023-02-05 18:57:16.656152-08:00 [info] <0.11209.4> Stopping application 'osiris'
2023-02-05 18:57:16.657964-08:00 [notice] <0.44.0> Application osiris exited with reason: stopped
2023-02-05 18:57:16.658904-08:00 [info] <0.11209.4> Stopping application 'ra'
2023-02-05 18:57:16.678705-08:00 [notice] <0.44.0> Application ra exited with reason: stopped
2023-02-05 18:57:16.678845-08:00 [info] <0.11209.4> Stopping application 'mnesia'
2023-02-05 18:57:16.691405-08:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-02-05 18:57:16.691625-08:00 [info] <0.11209.4> Successfully stopped RabbitMQ and its dependencies
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> Halting Erlang VM with the following applications:
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> amqp10_common
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> observer_cli
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> stdout_formatter
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> seshat
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> gen_batch_server
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> aten
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> ranch
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> inets
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> systemd
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> enough
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> cuttlefish
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> credentials_obfuscation
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> recon
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> jsx
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> xmerl
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> tools
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> syntax_tools
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> ssl
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> public_key
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> asn1
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> crypto
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> compiler
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> sasl
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> stdlib
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4> kernel
2023-02-05 18:57:16.691835-08:00 [info] <0.11209.4>
2023-02-05 18:57:21.286587-08:00 [notice] <0.220.0> Logging: configured log handlers are now ACTIVE
2023-02-05 18:57:23.737348-08:00 [info] <0.220.0> ra: starting system quorum_queues
2023-02-05 18:57:23.737490-08:00 [info] <0.220.0> starting Ra system: quorum_queues in directory: /data01/rabbitmq/mnesia/rabbit@dpswq063/quorum/rabbit@dpswq063
2023-02-05 18:57:23.739090-08:00 [info] <0.283.0> ra system 'quorum_queues' running pre init for 0 registered servers
2023-02-05 18:57:23.740395-08:00 [info] <0.284.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2023-02-05 18:57:23.740613-08:00 [notice] <0.289.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2023-02-05 18:57:23.742367-08:00 [info] <0.220.0> ra: starting system coordination
2023-02-05 18:57:23.742424-08:00 [info] <0.220.0> starting Ra system: coordination in directory: /data01/rabbitmq/mnesia/rabbit@dpswq063/coordination/rabbit@dpswq063
2023-02-05 18:57:23.748511-08:00 [info] <0.310.0> ra system 'coordination' running pre init for 0 registered servers
2023-02-05 18:57:23.750242-08:00 [info] <0.313.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2023-02-05 18:57:23.750505-08:00 [notice] <0.329.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2023-02-05 18:57:23.752909-08:00 [info] <0.220.0>
2023-02-05 18:57:23.752909-08:00 [info] <0.220.0> Starting RabbitMQ 3.10.7 on Erlang 24.3.4.1 [emu]
2023-02-05 18:57:23.752909-08:00 [info] <0.220.0> Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
2023-02-05 18:57:23.752909-08:00 [info] <0.220.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0>
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> node : rabbit@dpswq063
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> home dir : /var/lib/rabbitmq
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> config file(s) : /etc/rabbitmq/rabbitmq.config
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> cookie hash : NNS5blN/IMuSFWaO1vdCIw==
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> log(s) : /var/log/rabbitmq/[email protected]
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> : /var/log/rabbitmq/rabbit@dpswq063_upgrade.log
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> :
2023-02-05 18:57:23.754256-08:00 [info] <0.220.0> database dir : /data01/rabbitmq/mnesia/rabbit@dpswq063
2023-02-05 18:57:24.059683-08:00 [info] <0.220.0> Feature flags: list of feature flags found:
2023-02-05 18:57:24.059787-08:00 [info] <0.220.0> Feature flags: [x] classic_mirrored_queue_version
2023-02-05 18:57:24.059826-08:00 [info] <0.220.0> Feature flags: [ ] drop_unroutable_metric
2023-02-05 18:57:24.059859-08:00 [info] <0.220.0> Feature flags: [ ] empty_basic_get_metric
2023-02-05 18:57:24.059896-08:00 [info] <0.220.0> Feature flags: [x] implicit_default_bindings
2023-02-05 18:57:24.059935-08:00 [info] <0.220.0> Feature flags: [x] maintenance_mode_status
2023-02-05 18:57:24.059965-08:00 [info] <0.220.0> Feature flags: [x] quorum_queue
2023-02-05 18:57:24.060079-08:00 [info] <0.220.0> Feature flags: [x] stream_queue
2023-02-05 18:57:24.060139-08:00 [info] <0.220.0> Feature flags: [x] user_limits
2023-02-05 18:57:24.060173-08:00 [info] <0.220.0> Feature flags: [x] virtual_host_metadata
2023-02-05 18:57:24.060209-08:00 [info] <0.220.0> Feature flags: feature flag states written to disk: yes
2023-02-05 18:57:24.223036-08:00 [info] <0.220.0> Running boot step pre_boot defined by app rabbit
2023-02-05 18:57:24.223140-08:00 [info] <0.220.0> Running boot step rabbit_global_counters defined by app rabbit
2023-02-05 18:57:24.223446-08:00 [info] <0.220.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2023-02-05 18:57:24.223577-08:00 [info] <0.220.0> Running boot step rabbit_core_metrics defined by app rabbit
2023-02-05 18:57:24.224581-08:00 [info] <0.220.0> Running boot step rabbit_alarm defined by app rabbit
2023-02-05 18:57:24.229470-08:00 [info] <0.353.0> Memory high watermark set to 9585 MiB (10050720563 bytes) of 23962 MiB (25126801408 bytes) total
2023-02-05 18:57:24.234203-08:00 [info] <0.355.0> Enabling free disk space monitoring
2023-02-05 18:57:24.234272-08:00 [info] <0.355.0> Disk free limit set to 50MB
2023-02-05 18:57:24.238315-08:00 [info] <0.220.0> Running boot step code_server_cache defined by app rabbit
2023-02-05 18:57:24.238494-08:00 [info] <0.220.0> Running boot step file_handle_cache defined by app rabbit
2023-02-05 18:57:24.238903-08:00 [info] <0.360.0> Limiting to approx 32671 file handles (29401 sockets)
2023-02-05 18:57:24.239094-08:00 [info] <0.361.0> FHC read buffering: OFF
2023-02-05 18:57:24.239174-08:00 [info] <0.361.0> FHC write buffering: ON
2023-02-05 18:57:24.240210-08:00 [info] <0.220.0> Running boot step worker_pool defined by app rabbit
2023-02-05 18:57:24.240327-08:00 [info] <0.338.0> Will use 10 processes for default worker pool
2023-02-05 18:57:24.240459-08:00 [info] <0.338.0> Starting worker pool 'worker_pool' with 10 processes in it
2023-02-05 18:57:24.241224-08:00 [info] <0.220.0> Running boot step database defined by app rabbit
2023-02-05 18:57:24.243029-08:00 [info] <0.220.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-02-05 18:57:24.243425-08:00 [info] <0.220.0> Successfully synced tables from a peer
2023-02-05 18:57:24.243600-08:00 [info] <0.220.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-02-05 18:57:24.243800-08:00 [info] <0.220.0> Successfully synced tables from a peer
2023-02-05 18:57:24.253082-08:00 [info] <0.220.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-02-05 18:57:24.253212-08:00 [info] <0.220.0> Successfully synced tables from a peer
2023-02-05 18:57:24.253258-08:00 [info] <0.220.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2023-02-05 18:57:24.253431-08:00 [info] <0.220.0> Running boot step database_sync defined by app rabbit
2023-02-05 18:57:24.253633-08:00 [info] <0.220.0> Running boot step feature_flags defined by app rabbit
2023-02-05 18:57:24.253821-08:00 [info] <0.220.0> Running boot step codec_correctness_check defined by app rabbit
2023-02-05 18:57:24.253893-08:00 [info] <0.220.0> Running boot step external_infrastructure defined by app rabbit
2023-02-05 18:57:24.253996-08:00 [info] <0.220.0> Running boot step rabbit_registry defined by app rabbit
2023-02-05 18:57:24.254117-08:00 [info] <0.220.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2023-02-05 18:57:24.254247-08:00 [info] <0.220.0> Running boot step rabbit_queue_location_random defined by app rabbit
2023-02-05 18:57:24.254341-08:00 [info] <0.220.0> Running boot step rabbit_event defined by app rabbit
2023-02-05 18:57:24.254506-08:00 [info] <0.220.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2023-02-05 18:57:24.254579-08:00 [info] <0.220.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2023-02-05 18:57:24.254637-08:00 [info] <0.220.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2023-02-05 18:57:24.254870-08:00 [info] <0.220.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2023-02-05 18:57:24.254947-08:00 [info] <0.220.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2023-02-05 18:57:24.255013-08:00 [info] <0.220.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2023-02-05 18:57:24.255074-08:00 [info] <0.220.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2023-02-05 18:57:24.255133-08:00 [info] <0.220.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2023-02-05 18:57:24.255233-08:00 [info] <0.220.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2023-02-05 18:57:24.255293-08:00 [info] <0.220.0> Running boot step rabbit_priority_queue defined by app rabbit
2023-02-05 18:57:24.255328-08:00 [info] <0.220.0> Priority queues enabled, real BQ is rabbit_variable_queue
2023-02-05 18:57:24.255383-08:00 [info] <0.220.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2023-02-05 18:57:24.255438-08:00 [info] <0.220.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2023-02-05 18:57:24.255505-08:00 [info] <0.220.0> Running boot step rabbit_auth_mechanism_ssl defined by app rabbitmq_auth_mechanism_ssl
2023-02-05 18:57:24.255554-08:00 [info] <0.220.0> Running boot step kernel_ready defined by app rabbit
2023-02-05 18:57:24.255595-08:00 [info] <0.220.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2023-02-05 18:57:24.255737-08:00 [info] <0.220.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2023-02-05 18:57:24.256876-08:00 [info] <0.389.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2023-02-05 18:57:24.257145-08:00 [info] <0.220.0> Running boot step guid_generator defined by app rabbit
2023-02-05 18:57:24.258483-08:00 [info] <0.220.0> Running boot step rabbit_node_monitor defined by app rabbit
2023-02-05 18:57:24.258671-08:00 [info] <0.393.0> Starting rabbit_node_monitor
2023-02-05 18:57:24.258816-08:00 [info] <0.220.0> Running boot step delegate_sup defined by app rabbit
2023-02-05 18:57:24.259408-08:00 [info] <0.220.0> Running boot step rabbit_memory_monitor defined by app rabbit
2023-02-05 18:57:24.259710-08:00 [info] <0.220.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2023-02-05 18:57:24.259830-08:00 [info] <0.220.0> Running boot step core_initialized defined by app rabbit
2023-02-05 18:57:24.259882-08:00 [info] <0.220.0> Running boot step upgrade_queues defined by app rabbit
2023-02-05 18:57:24.266494-08:00 [info] <0.220.0> Running boot step channel_tracking defined by app rabbit
2023-02-05 18:57:24.266708-08:00 [info] <0.220.0> Setting up a table for channel tracking on this node: tracked_channel_on_node_rabbit@dpswq063
2023-02-05 18:57:24.266936-08:00 [info] <0.220.0> Setting up a table for channel tracking on this node: tracked_channel_table_per_user_on_node_rabbit@dpswq063
2023-02-05 18:57:24.267197-08:00 [info] <0.220.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2023-02-05 18:57:24.267294-08:00 [info] <0.220.0> Running boot step connection_tracking defined by app rabbit
2023-02-05 18:57:24.267485-08:00 [info] <0.220.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@dpswq063
2023-02-05 18:57:24.267654-08:00 [info] <0.220.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@dpswq063
2023-02-05 18:57:24.267875-08:00 [info] <0.220.0> Setting up a table for per-user connection counting on this node: tracked_connection_table_per_user_on_node_rabbit@dpswq063
2023-02-05 18:57:24.268246-08:00 [info] <0.220.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2023-02-05 18:57:24.268348-08:00 [info] <0.220.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2023-02-05 18:57:24.268510-08:00 [info] <0.220.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2023-02-05 18:57:24.268621-08:00 [info] <0.220.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2023-02-05 18:57:24.268792-08:00 [info] <0.220.0> Running boot step rabbit_policies defined by app rabbit
2023-02-05 18:57:24.269020-08:00 [info] <0.220.0> Running boot step rabbit_policy defined by app rabbit
2023-02-05 18:57:24.269080-08:00 [info] <0.220.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2023-02-05 18:57:24.269162-08:00 [info] <0.220.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2023-02-05 18:57:24.269239-08:00 [info] <0.220.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2023-02-05 18:57:24.269345-08:00 [info] <0.220.0> Running boot step rabbit_vhost_limit defined by app rabbit
2023-02-05 18:57:24.269404-08:00 [info] <0.220.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2023-02-05 18:57:24.269460-08:00 [info] <0.220.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2023-02-05 18:57:24.269503-08:00 [info] <0.220.0> Management plugin: using rates mode 'basic'
2023-02-05 18:57:24.270070-08:00 [info] <0.220.0> Running boot step recovery defined by app rabbit
2023-02-05 18:57:24.272281-08:00 [info] <0.432.0> Making sure data directory '/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2023-02-05 18:57:24.276723-08:00 [info] <0.432.0> Starting message stores for vhost '/'
2023-02-05 18:57:24.276968-08:00 [info] <0.437.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2023-02-05 18:57:24.278433-08:00 [info] <0.432.0> Started message store of type transient for vhost '/'
2023-02-05 18:57:24.278713-08:00 [info] <0.441.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2023-02-05 18:57:24.282116-08:00 [info] <0.432.0> Started message store of type persistent for vhost '/'
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> ** Generic server <0.567.0> terminating
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> ** Last message in was {init,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {<0.432.0>,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [[{segments,[{32,-2}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {next_seq_id,94},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {next_deliver_seq_id,94},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {persistent_ref,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> <<125,166,122,216,157,220,81,199,127,215,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> 121,70,243,233,169,167>>},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {persistent_count,0},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {persistent_bytes,0}]]}}
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> ** When Server state == {q,{amqqueue,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> true,false,none,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> <0.567.0>,[],[],[],undefined,undefined,[],
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> undefined,stopped,0,[],<<"/">>,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> none,false,undefined,undefined,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {state,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {queue,[],[],0},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {active,-576460746531024,1.0}},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {state,fine,5000,undefined},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> #{},undefined,undefined,undefined,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {state,#{},delegate},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> 'drop-head',0,0,running,false}
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> ** Reason for termination ==
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> ** {function_clause,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{rabbit_variable_queue,d,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{delta,65536,0,0,94}],
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{file,"rabbit_variable_queue.erl"},{line,1587}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {rabbit_variable_queue,init,11,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{file,"rabbit_variable_queue.erl"},{line,1802}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{file,"rabbit_priority_queue.erl"},{line,150}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {rabbit_amqqueue_process,init_it2,3,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{file,"rabbit_amqqueue_process.erl"},{line,209}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {rabbit_amqqueue_process,handle_call,3,
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> [{file,"rabbit_amqqueue_process.erl"},{line,1306}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1046}]},
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2023-02-05 18:57:24.297725-08:00 [error] <0.567.0>
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> crasher:
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> initial call: rabbit_prequeue:init/1
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> pid: <0.567.0>
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> registered_name: []
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> exception exit: {function_clause,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{rabbit_variable_queue,d,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{delta,65536,0,0,94}],
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"rabbit_variable_queue.erl"},{line,1587}]},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {rabbit_variable_queue,init,11,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"rabbit_variable_queue.erl"},{line,1802}]},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"rabbit_priority_queue.erl"},{line,150}]},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {rabbit_amqqueue_process,init_it2,3,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"rabbit_amqqueue_process.erl"},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {line,209}]},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {rabbit_amqqueue_process,handle_call,3,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"rabbit_amqqueue_process.erl"},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {line,1306}]},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {gen_server2,handle_msg,2,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"gen_server2.erl"},{line,1046}]},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {proc_lib,init_p_do_apply,3,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{file,"proc_lib.erl"},{line,226}]}]}
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183)
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> ancestors: [<0.566.0>,<0.447.0>,<0.431.0>,<0.430.0>,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> rabbit_vhost_sup_sup,rabbit_sup,<0.220.0>]
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> message_queue_len: 0
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> messages: []
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> links: [<0.566.0>]
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> dictionary: [{{#Ref<0.2214027736.2641362950.246611>,fhc_handle},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {handle,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {file_descriptor,prim_file,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> #{handle => #Ref<0.2214027736.2641494045.246463>,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> owner => <0.567.0>,r_ahead_size => 0,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> r_buffer =>
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> #Ref<0.2214027736.2641494022.246613>}},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> #Ref<0.2214027736.2641362950.246611>,0,false,0,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> infinity,[],<<>>,0,0,0,0,0,false,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> "/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [write,binary,raw,read],
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [{write_buffer,infinity}],
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> true,true,-576460746519042348}},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {guid,{{1185513795,2139000247,3314394821,2652233382},0}},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {{"/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> fhc_file},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {file,1,true}},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {process_name,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {rabbit_amqqueue_process,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> <<"PlatformDeadLetterQueue">>}}},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {segment_entry_count,2048},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {fhc_age_tree,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {1,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {{-576460746519042348,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> #Ref<0.2214027736.2641362950.246611>},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> true,nil,nil}}},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {rand_seed,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> {#{jump => #Fun<rand.3.92093067>,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> max => 288230376151711743,
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> next => #Fun<rand.5.92093067>,type => exsplus},
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> [235779577337460233|196591710873816466]}}]
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> trap_exit: true
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> status: running
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> heap_size: 17731
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> stack_size: 28
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> reductions: 39726
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0> neighbours:
2023-02-05 18:57:24.298563-08:00 [error] <0.567.0>
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> Queue <0.567.0> failed to initialise: {function_clause,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{rabbit_variable_queue,d,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{delta,65536,0,0,94}],
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"rabbit_variable_queue.erl"},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {line,1587}]},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {rabbit_variable_queue,init,11,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"rabbit_variable_queue.erl"},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {line,1802}]},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"rabbit_priority_queue.erl"},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {line,150}]},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {rabbit_amqqueue_process,init_it2,3,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"rabbit_amqqueue_process.erl"},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {line,209}]},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {rabbit_amqqueue_process,handle_call,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> 3,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"rabbit_amqqueue_process.erl"},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {line,1306}]},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {gen_server2,handle_msg,2,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"gen_server2.erl"},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {line,1046}]},
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> {proc_lib,init_p_do_apply,3,
2023-02-05 18:57:24.299885-08:00 [error] <0.432.0> [{file,"proc_lib.erl"},{line,226}]}]}
2023-02-05 18:57:24.300270-08:00 [info] <0.432.0> Recovering 41 queues of type rabbit_classic_queue took 27ms
2023-02-05 18:57:24.300322-08:00 [info] <0.432.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2023-02-05 18:57:24.300357-08:00 [info] <0.432.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> supervisor: {<0.566.0>,rabbit_amqqueue_sup}
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> errorContext: child_terminated
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> reason: {function_clause,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{rabbit_variable_queue,d,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{delta,65536,0,0,94}],
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"rabbit_variable_queue.erl"},{line,1587}]},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {rabbit_variable_queue,init,11,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"rabbit_variable_queue.erl"},{line,1802}]},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"rabbit_priority_queue.erl"},{line,150}]},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {rabbit_amqqueue_process,init_it2,3,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"rabbit_amqqueue_process.erl"},{line,209}]},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {rabbit_amqqueue_process,handle_call,3,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"rabbit_amqqueue_process.erl"},{line,1306}]},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {gen_server2,handle_msg,2,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"gen_server2.erl"},{line,1046}]},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {proc_lib,init_p_do_apply,3,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{file,"proc_lib.erl"},{line,226}]}]}
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> offender: [{pid,<0.567.0>},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {id,rabbit_amqqueue},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {mfargs,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {rabbit_prequeue,start_link,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{amqqueue,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> true,false,none,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> <0.567.0>,[],[],[],undefined,undefined,[],
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> undefined,stopped,0,[],<<"/">>,
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> recovery,<0.565.0>]}},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {restart_type,intrinsic},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {shutdown,600000},
2023-02-05 18:57:24.299812-08:00 [error] <0.566.0> {child_type,worker}]
2023-02-05 18:57:24.300618-08:00 [error] <0.655.0> Restarting crashed queue 'PlatformDeadLetterQueue' in vhost '/'.
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> ** Generic server <0.655.0> terminating
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> ** Last message in was {'$gen_cast',init}
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> ** When Server state == {q,{amqqueue,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> true,false,none,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> <0.655.0>,[],[],[],undefined,undefined,[],[],
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> live,0,[],<<"/">>,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> none,false,undefined,undefined,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {state,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {queue,[],[],0},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {active,-576460746516008,1.0}},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {state,fine,5000,undefined},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> #{},undefined,undefined,undefined,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {state,#{},delegate},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> 'drop-head',0,0,running,false}
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> ** Reason for termination ==
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> ** {{case_clause,undefined},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,785}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,674}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,670}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_variable_queue.erl"},{line,604}]},
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0> [{file,"rabbit_priority_queue.erl"},{line,150}]}]}
2023-02-05 18:57:24.303846-08:00 [error] <0.655.0>
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> crasher:
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> initial call: rabbit_prequeue:init/1
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> pid: <0.655.0>
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> registered_name: []
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> exception exit: {{case_clause,undefined},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,785}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,674}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_queue_index.erl"},{line,670}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_variable_queue.erl"},{line,604}]},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{file,"rabbit_priority_queue.erl"},{line,150}]}]}
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183)
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> ancestors: [<0.566.0>,<0.447.0>,<0.431.0>,<0.430.0>,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> rabbit_vhost_sup_sup,rabbit_sup,<0.220.0>]
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> message_queue_len: 0
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> messages: []
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> links: [<0.566.0>]
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> dictionary: [{segment_entry_count,2048},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {guid,{{3288342472,2601459470,3612238921,1540649971},1}},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {{"/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> fhc_file},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {file,1,true}},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {{#Ref<0.2214027736.2641362949.246688>,fhc_handle},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {handle,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {file_descriptor,prim_file,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> #{handle => #Ref<0.2214027736.2641494045.246469>,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> owner => <0.655.0>,r_ahead_size => 0,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> r_buffer =>
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> #Ref<0.2214027736.2641494021.246690>}},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> #Ref<0.2214027736.2641362949.246688>,0,false,0,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> infinity,[],<<>>,0,0,0,0,0,false,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> "/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [write,binary,raw,read],
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [{write_buffer,infinity}],
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> true,true,-576460746513282174}},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {process_name,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rabbit_amqqueue_process,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> <<"PlatformDeadLetterQueue">>}}},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {fhc_age_tree,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {1,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {{-576460746513282174,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> #Ref<0.2214027736.2641362949.246688>},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> true,nil,nil}}},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {rand_seed,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> {#{jump => #Fun<rand.3.92093067>,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> max => 288230376151711743,
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> next => #Fun<rand.5.92093067>,type => exsplus},
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> [215001474251303669|217318126384171665]}}]
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> trap_exit: true
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> status: running
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> heap_size: 10958
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> stack_size: 28
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> reductions: 52739
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0> neighbours:
2023-02-05 18:57:24.304514-08:00 [error] <0.655.0>
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> supervisor: {<0.566.0>,rabbit_amqqueue_sup}
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> errorContext: child_terminated
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> reason: {{case_clause,undefined},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,1297}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,1281}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,785}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,674}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,670}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_variable_queue.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,604}]},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{file,"rabbit_priority_queue.erl"},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {line,150}]}]}
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> offender: [{pid,<0.655.0>},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {id,rabbit_amqqueue},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {mfargs,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {rabbit_prequeue,start_link,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{amqqueue,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> true,false,none,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> <0.567.0>,[],[],[],undefined,undefined,[],
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> undefined,stopped,0,[],<<"/">>,
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> recovery,<0.565.0>]}},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {restart_type,intrinsic},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {shutdown,600000},
2023-02-05 18:57:24.305728-08:00 [error] <0.566.0> {child_type,worker}]
2023-02-05 18:57:24.306502-08:00 [error] <0.687.0> Restarting crashed queue 'PlatformDeadLetterQueue' in vhost '/'.
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> ** Generic server <0.687.0> terminating
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> ** Last message in was {'$gen_cast',init}
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> ** When Server state == {q,{amqqueue,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> true,false,none,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> <0.687.0>,[],[],[],undefined,undefined,[],[],
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> live,0,[],<<"/">>,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> none,false,undefined,undefined,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {state,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {queue,[],[],0},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {active,-576460746510125,1.0}},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {state,fine,5000,undefined},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> #{},undefined,undefined,undefined,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {state,#{},delegate},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> 'drop-head',0,0,running,false}
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> ** Reason for termination ==
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> ** {{case_clause,undefined},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,785}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,674}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,670}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_variable_queue.erl"},{line,604}]},
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0> [{file,"rabbit_priority_queue.erl"},{line,150}]}]}
2023-02-05 18:57:24.317237-08:00 [error] <0.687.0>
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> crasher:
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> initial call: rabbit_prequeue:init/1
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> pid: <0.687.0>
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> registered_name: []
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> exception exit: {{case_clause,undefined},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,785}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,674}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_queue_index.erl"},{line,670}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_variable_queue.erl"},{line,604}]},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{file,"rabbit_priority_queue.erl"},{line,150}]}]}
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183)
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> ancestors: [<0.566.0>,<0.447.0>,<0.431.0>,<0.430.0>,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> rabbit_vhost_sup_sup,rabbit_sup,<0.220.0>]
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> message_queue_len: 0
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> messages: []
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> links: [<0.566.0>]
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> dictionary: [{segment_entry_count,2048},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {guid,{{836653950,1640325821,33329800,3302796508},1}},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {{"/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> fhc_file},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {file,1,true}},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {{#Ref<0.2214027736.2641362947.248583>,fhc_handle},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {handle,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {file_descriptor,prim_file,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> #{handle => #Ref<0.2214027736.2641494042.246805>,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> owner => <0.687.0>,r_ahead_size => 0,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> r_buffer =>
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> #Ref<0.2214027736.2641494018.248902>}},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> #Ref<0.2214027736.2641362947.248583>,0,false,0,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> infinity,[],<<>>,0,0,0,0,0,false,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> "/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [write,binary,raw,read],
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [{write_buffer,infinity}],
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> true,true,-576460746500198765}},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {process_name,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rabbit_amqqueue_process,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> <<"PlatformDeadLetterQueue">>}}},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {fhc_age_tree,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {1,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {{-576460746500198765,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> #Ref<0.2214027736.2641362947.248583>},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> true,nil,nil}}},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {rand_seed,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> {#{jump => #Fun<rand.3.92093067>,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> max => 288230376151711743,
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> next => #Fun<rand.5.92093067>,type => exsplus},
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> [260609370484778985|171733626899171682]}}]
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> trap_exit: true
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> status: running
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> heap_size: 10958
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> stack_size: 28
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> reductions: 52726
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0> neighbours:
2023-02-05 18:57:24.317802-08:00 [error] <0.687.0>
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> supervisor: {<0.566.0>,rabbit_amqqueue_sup}
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> errorContext: child_terminated
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> reason: {{case_clause,undefined},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,1297}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,1281}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,785}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,674}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,670}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_variable_queue.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,604}]},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{file,"rabbit_priority_queue.erl"},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {line,150}]}]}
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> offender: [{pid,<0.687.0>},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {id,rabbit_amqqueue},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {mfargs,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {rabbit_prequeue,start_link,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{amqqueue,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> true,false,none,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> <0.567.0>,[],[],[],undefined,undefined,[],
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> undefined,stopped,0,[],<<"/">>,
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> recovery,<0.565.0>]}},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {restart_type,intrinsic},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {shutdown,600000},
2023-02-05 18:57:24.319050-08:00 [error] <0.566.0> {child_type,worker}]
2023-02-05 18:57:24.319755-08:00 [error] <0.704.0> Restarting crashed queue 'PlatformDeadLetterQueue' in vhost '/'.
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> ** Generic server <0.704.0> terminating
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> ** Last message in was {'$gen_cast',init}
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> ** When Server state == {q,{amqqueue,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> true,false,none,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> <0.704.0>,[],[],[],undefined,undefined,[],[],
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> live,0,[],<<"/">>,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> none,false,undefined,undefined,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {state,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {queue,[],[],0},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {active,-576460746496868,1.0}},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {state,fine,5000,undefined},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> #{},undefined,undefined,undefined,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {state,#{},delegate},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> undefined,undefined,undefined,undefined,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> 'drop-head',0,0,running,false}
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> ** Reason for termination ==
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> ** {{case_clause,undefined},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,785}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,674}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,670}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_variable_queue.erl"},{line,604}]},
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0> [{file,"rabbit_priority_queue.erl"},{line,150}]}]}
2023-02-05 18:57:24.395071-08:00 [error] <0.704.0>
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> crasher:
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> initial call: rabbit_prequeue:init/1
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> pid: <0.704.0>
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> registered_name: []
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> exception exit: {{case_clause,undefined},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,1297}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,1281}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,785}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,674}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_queue_index.erl"},{line,670}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_variable_queue.erl"},{line,604}]},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{file,"rabbit_priority_queue.erl"},{line,150}]}]}
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183)
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> ancestors: [<0.566.0>,<0.447.0>,<0.431.0>,<0.430.0>,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> rabbit_vhost_sup_sup,rabbit_sup,<0.220.0>]
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> message_queue_len: 0
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> messages: []
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> links: [<0.566.0>]
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> dictionary: [{segment_entry_count,2048},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {guid,{{2474073616,444785983,684604721,525222601},1}},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {{"/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> fhc_file},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {file,1,true}},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {{#Ref<0.2214027736.2641362946.251028>,fhc_handle},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {handle,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {file_descriptor,prim_file,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> #{handle => #Ref<0.2214027736.2641494042.246813>,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> owner => <0.704.0>,r_ahead_size => 0,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> r_buffer =>
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> #Ref<0.2214027736.2641494018.251030>}},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> #Ref<0.2214027736.2641362946.251028>,0,false,0,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> infinity,[],<<>>,0,0,0,0,0,false,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> "/data01/rabbitmq/mnesia/rabbit@dpswq063/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [write,binary,raw,read],
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [{write_buffer,infinity}],
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> true,true,-576460746421994522}},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {process_name,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rabbit_amqqueue_process,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> <<"PlatformDeadLetterQueue">>}}},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {fhc_age_tree,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {1,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {{-576460746421994522,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> #Ref<0.2214027736.2641362946.251028>},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> true,nil,nil}}},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {rand_seed,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> {#{jump => #Fun<rand.3.92093067>,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> max => 288230376151711743,
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> next => #Fun<rand.5.92093067>,type => exsplus},
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> [276024948249671502|84162600645768587]}}]
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> trap_exit: true
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> status: running
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> heap_size: 10958
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> stack_size: 28
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> reductions: 52736
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0> neighbours:
2023-02-05 18:57:24.395796-08:00 [error] <0.704.0>
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> supervisor: {<0.566.0>,rabbit_amqqueue_sup}
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> errorContext: child_terminated
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> reason: {{case_clause,undefined},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{rabbit_queue_index,add_segment_relseq_entry,3,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,1297}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_queue_index,parse_segment_entries,3,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,1281}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_queue_index,recover_segment,4,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,785}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_queue_index,'-init_dirty/4-fun-0-',6,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,674}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {lists,foldl,3,[{file,"lists.erl"},{line,1267}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_queue_index,init_dirty,4,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_queue_index.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,670}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_variable_queue,init,6,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_variable_queue.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,604}]},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_priority_queue,init,3,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{file,"rabbit_priority_queue.erl"},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {line,150}]}]}
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> offender: [{pid,<0.704.0>},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {id,rabbit_amqqueue},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {mfargs,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {rabbit_prequeue,start_link,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{amqqueue,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {resource,<<"/">>,queue,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> <<"PlatformDeadLetterQueue">>},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> true,false,none,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> <0.567.0>,[],[],[],undefined,undefined,[],
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> undefined,stopped,0,[],<<"/">>,
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> #{user => <<"dpswq063.drm.lab.emc.com">>},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> rabbit_classic_queue,#{}},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> recovery,<0.565.0>]}},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {restart_type,intrinsic},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {shutdown,600000},
2023-02-05 18:57:24.396989-08:00 [error] <0.566.0> {child_type,worker}]`

[michaelklishin]

Moving away from classic queues v1 and RabbitMQ 3.10.x is the only short term solution I can offer.

[haloxinyu]

Thanks for your prompt reply! How do you find the classic queue version from the log?Is there a simple way to check the version?
As far as I know, RabbitMQ 3.10.x supported classic queue v2, can we migrate a classic queue version 1 to version 2 at 3.10.x? Or as you suggested, move away from 3.10.x and use 3.11.x?

[mkuratczyk]

Classic queues v2 were added in 3.10 and haven't changed much in 3.11. You can switch by applying a policy that sets x-queue-version=2 (note, if you remove that policy, it will revert to v1 currently). You can also change the default through configuration classic_queue.default_version = 2. In RabbitMQ 3.12 (to be released in the coming weeks), v2 will be significantly improved compared to v2 from 3.10/3.11 (we will publish a blog post about it).

[haloxinyu]

Thanks @michaelklishin and @mkuratczyk ! Now I'm cleare about it.

[michaelklishin]

@lhoguin anything you'd like to add to the above?

[haloxinyu]

In addition, my rabbitmq server in another node suddenly crashed recently. The version is also 3.10.7. I'm not sure the issue is still related to classic queues v1, could you please give some sugesstions? Below is the logs:

`2023-02-06 10:42:39.933957+01:00 [info] <0.3504.0> connection <0.3504.0> (127.0.0.1:59772 -> 127.0.0.1:5671) has a client-provided name: ReplicationManager#25364645:0

2023-02-06 10:42:39.938287+01:00 [info] <0.3504.0> connection <0.3504.0> (127.0.0.1:59772 -> 127.0.0.1:5671 - ReplicationManager#25364645:0): user 'ppdm1pc01.colruyt.int' authenticated and granted access to vhost '/'
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> ** Generic server <0.570.0> terminating
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> ** Last message in was {drop_expired,1}
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> ** When Server state == {q,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {amqqueue,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {resource,<<"/">>,queue,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> <<"PlatformDeadLetterQueue">>},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> true,false,none,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{<<"x-message-ttl">>,long,86400000}],
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> <0.570.0>,[],[],[],undefined,undefined,[],undefined,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> stopped,0,[],<<"/">>,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> #{user => <<"ppdm1pc01.colruyt.int">>},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> rabbit_classic_queue,#{}},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> none,false,rabbit_priority_queue,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {passthrough,rabbit_variable_queue,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {vqstate,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {3,
...
...
...
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> ** Reason for termination ==
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> ** {function_clause,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{rabbit_queue_index,segment_plus_journal1,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{{true,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> <<82,164,71,16,208,35,204,173,38,29,5,223,120,19,206,24,0,5,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> 238,191,176,154,153,25,0,0,13,118>>,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> <<>>},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> no_del,no_ack},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {no_pub,no_del,ack}],
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1342}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {rabbit_queue_index,'-segment_plus_journal/2-fun-0-',3,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1329}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {rabbit_queue_index,segment_entries_foldr,3,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1235}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {rabbit_queue_index,read_bounded_segment,5,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1216}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {rabbit_queue_index,read,3,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,506}]},
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> {rabbit_variable_queue,maybe_deltas_to_betas,2,
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0> [{file,"rabbit_variable_queue.erl"},{line,3192}]}]}
2023-02-07 09:05:53.566151+01:00 [error] <0.570.0>
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> crasher:
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> initial call: rabbit_prequeue:init/1
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> pid: <0.570.0>
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> registered_name: []
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> exception exit: {function_clause,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{rabbit_queue_index,segment_plus_journal1,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{{true,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> <<82,164,71,16,208,35,204,173,38,29,5,223,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> 120,19,206,24,0,5,238,191,176,154,153,25,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> 0,0,13,118>>,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> <<>>},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> no_del,no_ack},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {no_pub,no_del,ack}],
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1342}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rabbit_queue_index,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> '-segment_plus_journal/2-fun-0-',3,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1329}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {array,sparse_foldl_3,7,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"array.erl"},{line,1684}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {array,sparse_foldl_2,9,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"array.erl"},{line,1678}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rabbit_queue_index,segment_entries_foldr,3,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1235}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rabbit_queue_index,read_bounded_segment,5,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,1216}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rabbit_queue_index,read,3,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"rabbit_queue_index.erl"},{line,506}]},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rabbit_variable_queue,maybe_deltas_to_betas,2,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{file,"rabbit_variable_queue.erl"},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {line,3192}]}]}
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183)
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> ancestors: [<0.569.0>,<0.447.0>,<0.431.0>,<0.430.0>,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> rabbit_vhost_sup_sup,rabbit_sup,<0.220.0>]
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> message_queue_len: 0
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> messages: []
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> links: [<0.569.0>]
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> dictionary: [{segment_entry_count,16384},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {guid,{{2982936330,3597972645,1420962668,442673272},0}},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {{#Ref<0.1173766887.1217134597.225756>,fhc_handle},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {handle,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {file_descriptor,prim_file,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> #{handle => #Ref<0.1173766887.1217265703.225330>,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> owner => <0.570.0>,r_ahead_size => 0,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> r_buffer =>
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> #Ref<0.1173766887.1217265669.225761>}},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> #Ref<0.1173766887.1217134597.225756>,16,false,0,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> infinity,[],<<>>,0,0,0,0,0,true,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> "/data01/rabbitmq/mnesia/rabbit@ppdm1pc01/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [write,binary,raw,read],
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [{write_buffer,infinity}],
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> true,true,-576379680131623430}},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {process_name,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rabbit_amqqueue_process,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {resource,<<"/">>,queue,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> <<"PlatformDeadLetterQueue">>}}},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {fhc_age_tree,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {1,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {{-576379680131623430,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> #Ref<0.1173766887.1217134597.225756>},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> true,nil,nil}}},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {{"/data01/rabbitmq/mnesia/rabbit@ppdm1pc01/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/4QKF1N2MOXGFLHDA7RR79UZRJ/journal.jif",
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> fhc_file},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {file,1,true}},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {explicit_gc_run_operation_threshold,1000},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {rand_seed,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> {#{jump => #Fun<rand.3.92093067>,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> max => 288230376151711743,
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> next => #Fun<rand.5.92093067>,type => exsplus},
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> [284119928262478977|148236199159123344]}}]
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> trap_exit: true
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> status: running
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> heap_size: 121536
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> stack_size: 28
2023-02-07 09:05:53.598209+01:00 [error] <0.570.0> reductions: 1449187
`

[lhoguin]

There were a few versions in 3.10 that had a crash there. You may want to upgrade to at least the most recent 3.10.

[michaelklishin]

Or better yet, to 3.11.9 :)

[haloxinyu]

Thanks for your warm help and suggestions! I'm trying to upgrade Rabbitmq Server