upgrade to 0.27.0 caused the strimzi-cluster-opeator to go into endless CrashLoopBackOff

[yxiang92128]

I used helm to upgrade to 0.27.0 to fix the log4j vulnerability issues.

helm repo update
helm upgrade --install strimzi-kafka strimzi/strimzi-kafka-operator --version="0.27.0" -n kafka
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /root/.kube/tc-clusters/tc763d7d
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /root/.kube/tc-clusters/tc763d7d
Release "strimzi-kafka" has been upgraded. Happy Helming!
NAME: strimzi-kafka
LAST DEPLOYED: Wed Jan  5 15:09:58 2022
NAMESPACE: kafka
STATUS: deployed
REVISION: 7
TEST SUITE: None
NOTES:
Thank you for installing strimzi-kafka-operator-0.27.0

To create a Kafka cluster refer to the following documentation.

https://strimzi.io/docs/operators/latest/using.html#deploying-cluster-operator-helm-chart-str

It seemed have worked but strimzi-cluster-operator pod is now in CrashLoopBackOff.

kubectl get pods -n kafka
NAME                                                 READY   STATUS             RESTARTS   AGE
grafana-f5797fcc9-6jnch                              1/1     Running            0          35d
pod-kafka-cluster-entity-operator-5cd68c8866-tswpz   2/3     Running            0          65s
pod-kafka-cluster-kafka-0                            1/1     Running            0          6m17s
pod-kafka-cluster-kafka-1                            0/1     Running            1          3m17s
pod-kafka-cluster-kafka-2                            0/1     Running            1          2m33s
pod-kafka-cluster-kafka-exporter-869dc4988b-gxsms    0/1     Running            0          41s
pod-kafka-cluster-zookeeper-0                        1/1     Running            0          97s
pod-kafka-cluster-zookeeper-1                        1/1     Running            1          90s
pod-kafka-cluster-zookeeper-2                        1/1     Running            0          84s
prometheus-operator-844d67c8d6-bmx9r                 1/1     Running            0          35d
prometheus-prometheus-0                              2/2     Running            0          35d
strimzi-cluster-operator-5fb779f7b4-pk4tw            0/1     CrashLoopBackOff   1          24s

Our namespace is set to be "kafka" and the log shows the following error:

2022-01-05 23:06:40 INFO  ClusterOperator:79 - Creating ClusterOperator for namespace kafka
2022-01-05 23:06:40 INFO  ClusterOperator:94 - Starting ClusterOperator for namespace kafka
2022-01-05 23:06:40 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2022-01-05 23:06:41 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2022-01-05 23:06:41 ERROR Main:154 - Cluster Operator verticle in namespace kafka failed to start
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/apis/core.strimzi.io/v1beta2/namespaces/kafka/strimzipodsets. Message: 404 page not found
.
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:140) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:422) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:411) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:91) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.informers.cache.Reflector.getList(Reflector.java:53) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.informers.cache.Reflector.listSyncAndWatch(Reflector.java:77) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.informers.impl.DefaultSharedIndexInformer.run(DefaultSharedIndexInformer.java:146) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.inform(BaseOperation.java:954) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.Informable.inform(Informable.java:52) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
        at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.informer(AbstractResourceOperator.java:458) ~[io.strimzi.operator-common-0.27.0.jar:0.27.0]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.<init>(StrimziPodSetController.java:94) ~[io.strimzi.cluster-operator-0.27.0.jar:0.27.0]
        at io.strimzi.operator.cluster.ClusterOperator.start(ClusterOperator.java:99) ~[io.strimzi.cluster-operator-0.27.0.jar:0.27.0]
        at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:196) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
        at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
        at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
        at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]

I tried to "kubectl delete pod" of all the pods and it still ends up in CrashLoop.

Can somebody please help me so I can get out of this situation?

Thanks!

[scholzj]

The HTTP 404 error suggests that you didn't installed/updated the CRDs. You might need to do it separately -> I think that for whatever reason Helm doesn't do it anymore.

[yxiang92128]

I am unfamiliar with where I can download CRDs. You mean the following yaml files? Does it matter the order I apply these YAMLs?
https://github.com/strimzi/strimzi-kafka-operator/tree/0.27.0/helm-charts/helm3/strimzi-kafka-operator/crds

[scholzj]

That is certainly an option. You can also go to the release page as I suggested (https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.0) and download the strimzi-crds-0.27.0.yaml file which should have all of them in one file.

[yxiang92128]

after I downloaded and kubectl apply -f of "strimzi-crds-0.27.0.yaml", my cluster showed there are two "exporter" and the rest stuck in terminating state:

kubectl get pods -n kafka
NAME                                                 READY   STATUS             RESTARTS   AGE
grafana-f5797fcc9-6jnch                              1/1     Running            0          35d
pod-kafka-cluster-entity-operator-5cdffbdc77-zlzcc   3/3     Running            0          72m
pod-kafka-cluster-kafka-0                            1/1     Terminating        0          75m
pod-kafka-cluster-kafka-1                            1/1     Terminating        0          78m
pod-kafka-cluster-kafka-2                            1/1     Terminating        0          76m
pod-kafka-cluster-kafka-exporter-f8969f8fd-5r6g5     0/1     CrashLoopBackOff   2          46s
pod-kafka-cluster-kafka-exporter-f8969f8fd-86rgk     1/1     Terminating        0          71m
pod-kafka-cluster-zookeeper-0                        1/1     Terminating        0          84m
pod-kafka-cluster-zookeeper-1                        1/1     Running            0          130m
pod-kafka-cluster-zookeeper-2                        1/1     Running            0          79m
prometheus-operator-844d67c8d6-bmx9r                 1/1     Running            0          35d
prometheus-prometheus-0                              2/2     Running            0          35d
strimzi-cluster-operator-5fb779f7b4-ndwrl            1/1     Running            9          153m

Wonder how I could get out of this jam. It's been in this state for the last two hours now. Thanks!!

[scholzj]

I don't know why it is in this state => certainly not without any logs etc.

[yxiang92128]

After six hours, the cluster seems to heal itself.

kubectl get pods -n kafka
NAME                                                 READY   STATUS    RESTARTS   AGE
grafana-f5797fcc9-6jnch                              1/1     Running   0          36d
pod-kafka-cluster-entity-operator-5cdffbdc77-zlzcc   3/3     Running   0          21h
pod-kafka-cluster-kafka-0                            1/1     Running   0          20h
pod-kafka-cluster-kafka-1                            1/1     Running   0          20h
pod-kafka-cluster-kafka-2                            1/1     Running   0          20h
pod-kafka-cluster-kafka-exporter-f8969f8fd-5r6g5     1/1     Running   9          20h
pod-kafka-cluster-zookeeper-0                        1/1     Running   0          20h
pod-kafka-cluster-zookeeper-1                        1/1     Running   0          22h
pod-kafka-cluster-zookeeper-2                        1/1     Running   0          21h
prometheus-operator-844d67c8d6-bmx9r                 1/1     Running   0          36d
prometheus-prometheus-0                              2/2     Running   0          36d
strimzi-cluster-operator-5fb779f7b4-ndwrl            1/1     Running   10         23h