[Bug]: Error deleting StatefulSet during Strimzi and Kafka upgrades. Conversion from StatefulSet to StrimziPodSet

[charris-ca]

Bug Description

Strimzi upgrade from 0.28 to 0.33.2 errors when trying to convert pods from StatefulSet to StrimziPodSet. I also received the error on the zookeeper pod set and I worked past it by manually deleting the statefulset. The StrimziPodSet took over after that, but all zookeper pods went down and then restarted.

I am worried that deleting the Kafka statefulset will do the same and cause an outage with our Kafka instances. How do we get around the operator not able to delete the statefulset? I have tried issuing a statefulset delete with --cascade=orphan, but that command hangs and never finishes.

status:
  conditions:
    - lastTransitionTime: '2023-04-05T03:01:36.365369407Z'
      message: >-
        "observe deletion of StatefulSet kafka/default-kafka-cluster-1-kafka"
        timed out after 300000ms
      reason: TimeoutException
      status: 'True'
      type: NotReady

Steps to reproduce

1. Upgrade Strimzi Operator
2. View Kafka Reconcile
3. Upgrade starts
4. Zookeper statefulset delete timeout
5. Kafka statefulset delete timeout

Expected behavior

No response

Strimzi version

0.33.2

Kubernetes version

Kubernetes 1.22

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

You did not provide any logs or anything. You would need to check the operator logs and/or the Kubernetes logs (assuming there won't be any error in the operator log) to see why the StatefulSet was not deleted.

The operator does non-cascading delete. So if that is failing even when doing it manually, you have probably some tooling in your system which is blocking this operation and you will need to fix it so that the operator can do it.