Strimzi Cluster Operator down after upgrading to 0.35.1 from 0.27.1

[Sai-Charan-Madhvaraj]

Hi Team,

We are trying to upgrade strimzi from version 0.27.1 to 0.35.1 directly and Strimzi Cluster operator pod was down with error.

Basically its complaining for access for all api resources of kafka.strimzi.io and core.strimzi.io, Please see below error in logs.

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.1.0.1:443/apis/kafka.strimzi.io/v1beta2/kafkamirrormaker2s?resourceVersion=0. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. kafkamirrormaker2s.kafka.strimzi.io is forbidden: User "system:serviceaccount:emp-kafka:strimzi-cluster-operator" cannot list resource "kafkamirrormaker2s" in API group "kafka.strimzi.io" at the cluster scope.

2023-07-12 13:14:04 ERROR InformerUtils:29 - Caught exception in the StrimziPodSet informer which is not started
java.util.concurrent.CompletionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.1.0.1:443/apis/core.strimzi.io/v1beta2/strimzipodsets?resourceVersion=0. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. strimzipodsets.core.strimzi.io is forbidden: User "system:serviceaccount:emp-kafka:strimzi-cluster-operator" cannot list resource "strimzipodsets" in API group "core.strimzi.io" at the cluster scope.

Upon investigating, we found that there is a new Cluster role strimzi-cluster-operator-watched where all API permissions has moved to this cluster role instead of strimzi-cluster-operator-namespaced which was case in 0.27.1.

Below are the changes in strimzi cluster operator deployment file

           - name: STRIMZI_NAMESPACE
              value: "*"
            - name: STRIMZI_LEADER_ELECTION_ENABLED
              value: "true"
            - name: STRIMZI_LEADER_ELECTION_LEASE_NAME
              value: "strimzi-cluster-operator"
            - name: STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: STRIMZI_LEADER_ELECTION_IDENTITY
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

[scholzj]

0.27->0.35 is a big jump. So yes, it is likely that there were some RBAC resource changes. You need to make sure the RBAC resources are installed properly according to the version you are upgrading to.

[scholzj]

Well, if you followed the guide, then you should have this cluster role and this (cluster) role binding. Is that the case? Does the binding point to the right service account?

[Sai-Charan-Madhvaraj]

@scholzj , yes above Cluster role and cluster role bindings are present and pointing to right service account .

Strimzi Cluster operator came up running once i changed the environment variable "STRIMZI_NAMESPACE" to metadata.namespace instead of *.

If we configure strimzi cluster deployment with below config, operator fails with error mentioned above.

           - name: STRIMZI_NAMESPACE
              value: "*"

when i changed this to below, operator pod came up successfully.

            - name: STRIMZI_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

[scholzj]

That suggests that you did not change the RoleBinding to ClusterRoleBinding as you should if you want to watch the whole cluster.

[Sai-Charan-Madhvaraj]

@scholzj , It works now after change from Role Binding to Cluster role binding. Thanks

The same has not been explicitly mentioned in the documentation (upgrade section) at https://strimzi.io/docs/operators/latest/deploying.html#assembly-upgrade-str , Do you think this needs to update in docs as most of them will face issue while upgrading from previous versions (0.27.1 and earlier) where the RBAC are completely different.

[scholzj]

You customized the operator when installing it in the first place - to watch the whole cluster. So when you do that, you have to redo this everytime you upgrade it. Otherwise you revert from the customized to the default settings. It looks like you kept your customized deployment with STRIMZI_NAMESPACE set to *. But you have to done the customization to the RBAC resources to make them work for the whole cluster.