RabbitMQ pod stuck on termination with fedaration crash

[jaroslaw-bochniak]

On clusters where we have federation configured we have observed that sometimes one or more pods are stuck in Terminating state during the restart/upgrade.

Before the cluster restart, the commands rabbitmq-queues check_if_node_is_quorum_critical and rabbitmq-queues check_if_node_is_mirror_sync_critical executed for each of the nodes confirmed that all nodes are in sync with data, so there were no leftovers from some quorum queues.

Node [email protected] reported no quorum queues with minimum quorum
Node [email protected] reported no classic mirrored queues without online synchronised mirrors

We are using RabbitMQ in version 3.11.1 running in k8s, provisioned by the cluster-operator in version 2.0.0. We are using cluster configured with a 3 nodes with following config:

  terminationGracePeriodSeconds: 1800
  skipPostDeploySteps: false
  rabbitmq:
    additionalPlugins:
      - rabbitmq_federation
      - rabbitmq_federation_management
      - rabbitmq_shovel
      - rabbitmq_shovel_management
      - rabbitmq_prometheus
      - rabbitmq_stream
    additionalConfig: |
      cluster_partition_handling = autoheal
      vm_memory_high_watermark.relative = 0.7
      disk_free_limit.relative = 1.0
      channel_max = 500
      consumer_timeout = 900000
      raft.segment_max_entries = 10000
      raft.wal_max_size_bytes = 16000000
      raft.wal_max_batch_size = 4096
      classic_queue.default_version = 2

On clusters where there is no federation configured this problem does not exists.

Some pods are shutting down quickly and without errors in logs, on others we are observing logs with:

2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>   crasher:
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     initial call: rabbit_federation_exchange_link:init/1
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     pid: <0.12690.3>
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     registered_name: []
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     exception exit: {upstream_channel_down,shutdown}
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     ancestors: [<0.12637.3>,<0.2088.0>,
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                   rabbit_federation_exchange_link_sup_sup,
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                   rabbit_federation_sup,rabbit_sup,<0.229.0>]
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     message_queue_len: 0
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     messages: []
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     links: [<0.12637.3>]
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                                 next => #Fun<rand.0.34006561>,type => exsss,
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                                 uniform => #Fun<rand.1.34006561>,
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                                 uniform_n => #Fun<rand.2.34006561>},
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                               [132833495143445419|15429724879150317]}},
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>                   {gen_server_call_timeout,130000}]
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     trap_exit: true
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     status: running
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     heap_size: 28690
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     stack_size: 28
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>     reductions: 69670
2022-10-19 07:06:12.237372+00:00 [error] <0.12690.3>   neighbours:

Full log in the attachment.
rabbit-stuck-on-terminate.txt

[michaelklishin]

This exception simply says that an upstream channel terminated (can be due to a restart of a remote node). This is unlikely to be a root cause.

See what rabbitmq-diagnostics maybe_stuck might report.

[jaroslaw-bochniak]

@michaelklishin Thanks Michael for a tip. After upgrading 10 clusters (each in 3 node config) from 3.11.1 to 3.11.2 I have not encountered the errors with federation crash. Only one pod had troubles and stuck in Termination phase. I am attaching output from rabbitmq-diagnostics maybe_stuck. I will keep an eye on that in future upgrades.
rabbitmq-maybe-stuck-diag.txt