All Zookeeper pods are deleted when the operator could not pull the Kafka image

[AshutoshBhattacharya]

Describe the bug

We are using Strimzi version 0.31.1 with StrimziPodSets enabled and have been experiencing some issues with it recently. When we tried to roll out our latest release, we found that the Kafka cluster is unusable. With further investigation, we discovered that the operator could not pull the Kafka image (caused by improper singing of the Kafka image, therefore, our ClusterImagePolicy blocks it) so, it could not create the new Zookeeper pod, but it destroyed all the old Zookeeper pods. That causes the issue with the Kafka cluster. The same problem persists even with the upgradation to the Strimzi version 0.33.0.

We found this problem only with the StrimziPodSets. If we disable it and fallback to StateFulSet this problem does not occur.

To Reproduce

Steps to reproduce the behavior:

1. Use helm command to deploy the Strimzi operator v0.33.0
2. Tag the kafka image for example kafka:0.33.0-kafka-3.2.3-bad
3. Create a cluster image policy to block the above-tagged image. For example, we used portieris and used following ClusterImagePolicy:

    apiVersion: portieris.cloud.ibm.com/v1
[Strimzi_reports.zip](https://github.com/strimzi/strimzi-kafka-operator/files/10765082/Strimzi_reports.zip)

    kind: ClusterImagePolicy
    metadata:
        name: block-bad-kafka-image
    spec:
        repositories:
        - name: <artifactory path>/kafka:0.33.0-kafka-3.2.3-bad
        policy:
            mutateImage: false
            trust:
                enabled: true
                trustServer: <server address>

5. Use kubectl -n kafka set env command to change STRIMZI_KAFKA_IMAGES and forced to use the kafka:0.33.0-kafka-3.2.3-bad image
6. Check the rolling update of Zookeeper pods

Expected behavior

It should behave the same as with StateFulSet i.e., if it cannot create the new pods, it must not remove the existing ones.

Environment:

• Strimzi version: 0.31.1, 0.33.0
• Installation method: Helm chart
• Kubernetes cluster: Kubernetes 1.24.9
• Infrastructure: Kubernetes cluster on AWS

YAML files and logs

kubectl get StrimziPodSets

NAME                    PODS   READY PODS   CURRENT PODS   AGE
edp-cluster-kafka       9      9            9              10h
edp-cluster-zookeeper   0      0            0              10h

We also observed the following logs on Strimzi operator:

2023-02-15 11:00:17 ERROR StrimziPodSetController:323 - Reconciliation #23(watch) StrimziPodSet(tollans-live/edp-cluster-zookeeper): StrimziPodSet edp-cluster-zookeeper in namespace tollans-live reconciliation failed
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://api.edp-g-su1.edp.internal.live.k8s.ondemand.com:443/api/v1/namespaces/tollans-live/pods. Message: admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
trust: policy denied the request: Deny "eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka:0.33.0-kafka-3.2.3-bad", failed to get content trust information: signing.repositories.cloud.sap does not have trust data for eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
trust: policy denied the request: Deny "eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka:0.33.0-kafka-3.2.3-bad", failed to get content trust information: signing.repositories.cloud.sap does not have trust data for eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:536) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:347) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:704) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1107) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.maybeCreateOrPatchPod(StrimziPodSetController.java:384) ~[io.strimzi.cluster-operator-0.33.0.jar:0.33.0]
at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.reconcile(StrimziPodSetController.java:312) ~[io.strimzi.cluster-operator-0.33.0.jar:0.33.0]
at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.run(StrimziPodSetController.java:480) ~[io.strimzi.cluster-operator-0.33.0.jar:0.33.0]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://api.edp-g-su1.edp.internal.live.k8s.ondemand.com:443/api/v1/namespaces/tollans-live/pods. Message: admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
trust: policy denied the request: Deny "eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka:0.33.0-kafka-3.2.3-bad", failed to get content trust information: signing.repositories.cloud.sap does not have trust data for eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
trust: policy denied the request: Deny "eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka:0.33.0-kafka-3.2.3-bad", failed to get content trust information: signing.repositories.cloud.sap does not have trust data for eu.gcr.io/sap-se-cx-edp-stg/strimzi/kafka, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:742) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:722) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:673) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:601) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133) ~[io.fabric8.kubernetes-httpclient-okhttp-6.3.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more

Strimzi_reports.zip

[scholzj]

This should be fixed in #7421 ... so what versions did you tried it with? As that part of your description is a bit confusing.

[scholzj]

PS: There is also this as an open issue which relates to it: #7685

[AshutoshBhattacharya]

@scholzj Thanks for your response. We were using version 0.31.1. When we encounter this problem we upgraded to version 0.33.0 and were hoping that the problem will be solved. Unfortunately, The same problem persists even in version 0.33.0.

Did you mean the invalid configuration in the Pod spec and in our scenario, the operator could not fetch the image are similar issues?

[scholzj]

Ok, thanks for the clarification. The issue #7685 covers situations when the pods don't exist. That sounds to be the case in your situation. So I think it should apply. The #7421 addressed only situations when the pod exists but is not ready => for example when it would be in ImagePullBackoff.