rabbitmq operator delete pods failure

[lanzhiwang]

On k8s, I use the rabbitmq operator to create and manage the rabbitmq cluster. When I delete this cluster, the corresponding pod cannot be deleted immediately, and the Termination Grace Period takes a long time, as shown below

$ kubectl -n rabbitmq describe pods test-grace-period-rabbitmq-server-0
Name:                      test-grace-period-rabbitmq-server-0
Namespace:                 rabbitmq
Priority:                  0
Node:                      10.0.128.64/10.0.128.64
Start Time:                Wed, 24 Mar 2021 14:06:01 +0800
Labels:                    app.kubernetes.io/component=rabbitmq
                           app.kubernetes.io/name=test-grace-period
                           app.kubernetes.io/part-of=rabbitmq
                           controller-revision-hash=test-grace-period-rabbitmq-server-5867dd895c
                           skipPreStopChecks=true
                           statefulset.kubernetes.io/pod-name=test-grace-period-rabbitmq-server-0
Annotations:               prometheus.io/port: 15692
                           prometheus.io/scrape: true
Status:                    Terminating (lasts <invalid>)
Termination Grace Period:  604800s
IP:                        10.199.0.162
IPs:
  IP:           10.199.0.162
Controlled By:  StatefulSet/test-grace-period-rabbitmq-server
Init Containers:
  copy-config:
    Container ID:  docker://9e144a30ff8cfad5c5b27a13a673a47c0e66ef50dab4197e1b0cee97ad55bcd0
    Image:         10.0.129.0:5000/3rdparty/rabbitmq:3.8.5
    Image ID:      docker-pullable://10.0.129.0:5000/3rdparty/rabbitmq@sha256:ffd879024b985f742de389b1d9572fb048d1f09da4bb050dc81069c57329baef
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf ; cp /tmp/rabbitmq/advanced.config /etc/rabbitmq/advanced.config ; cp /tmp/rabbitmq/rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf ; cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chown 999:999 /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /etc/rabbitmq/enabled_plugins && chown 999:999 /etc/rabbitmq/enabled_plugins
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 24 Mar 2021 14:06:02 +0800
      Finished:     Wed, 24 Mar 2021 14:06:02 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  500Mi
    Requests:
      cpu:        100m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /etc/rabbitmq/ from rabbitmq-etc (rw)
      /tmp/erlang-cookie-secret/ from erlang-cookie-secret (rw)
      /tmp/rabbitmq-plugins/ from plugins-conf (rw)
      /tmp/rabbitmq/ from server-conf (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from test-grace-period-rabbitmq-server-token-bngm8 (ro)
Containers:
  rabbitmq:
    Container ID:   docker://7760e643f85c03feb7a3c917565c54cffd7f4baaaea8ca1b169fa08b68ff9f09
    Image:          10.0.129.0:5000/3rdparty/rabbitmq:3.8.5
    Image ID:       docker-pullable://10.0.129.0:5000/3rdparty/rabbitmq@sha256:ffd879024b985f742de389b1d9572fb048d1f09da4bb050dc81069c57329baef
    Ports:          4369/TCP, 5672/TCP, 15672/TCP, 15692/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Wed, 24 Mar 2021 14:06:03 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:      200m
      memory:   205Mi
    Readiness:  exec [/bin/sh -c rabbitmq-diagnostics ping] delay=10s timeout=5s period=30s #success=1 #failure=3
    Environment:
      RABBITMQ_DEFAULT_PASS_FILE:  /opt/rabbitmq-secret/password
      RABBITMQ_DEFAULT_USER_FILE:  /opt/rabbitmq-secret/username
      MY_POD_NAME:                 test-grace-period-rabbitmq-server-0 (v1:metadata.name)
      MY_POD_NAMESPACE:            rabbitmq (v1:metadata.namespace)
      K8S_SERVICE_NAME:            test-grace-period-rabbitmq-headless
      RABBITMQ_USE_LONGNAME:       true
      RABBITMQ_NODENAME:           rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
      K8S_HOSTNAME_SUFFIX:         .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
    Mounts:
      /etc/pod-info/ from pod-info (rw)
      /etc/rabbitmq/ from rabbitmq-etc (rw)
      /opt/rabbitmq-secret/ from rabbitmq-admin (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from test-grace-period-rabbitmq-server-token-bngm8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  persistence:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  persistence-test-grace-period-rabbitmq-server-0
    ReadOnly:   false
  rabbitmq-admin:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-grace-period-rabbitmq-admin
    Optional:    false
  server-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      test-grace-period-rabbitmq-server-conf
    Optional:  false
  plugins-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      test-grace-period-rabbitmq-plugins-conf
    Optional:  false
  rabbitmq-etc:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  rabbitmq-erlang-cookie:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  erlang-cookie-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-grace-period-rabbitmq-erlang-cookie
    Optional:    false
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels['skipPreStopChecks'] -> skipPreStopChecks
  test-grace-period-rabbitmq-server-token-bngm8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-grace-period-rabbitmq-server-token-bngm8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 30s
                 node.kubernetes.io/unreachable:NoExecute for 30s
Events:
  Type     Reason     Age                From                  Message
  ----     ------     ----               ----                  -------
  Normal   Killing    73s                kubelet, 10.0.128.64  Stopping container rabbitmq
  Warning  Unhealthy  30s (x2 over 60s)  kubelet, 10.0.128.64  Readiness probe failed: Will ping rabbit@test-grace-period-rabbitmq-server-0.test-grace-period-rabbitmq-headless.rabbitmq. This only checks if the OS process is running and registered with epmd. Timeout: 60000 ms.
Error:
Failed to connect and authenticate to rabbit@test-grace-period-rabbitmq-server-0.test-grace-period-rabbitmq-headless.rabbitmq in 60000 ms

What is the reason？