Kafka not getting in Ready state due to KubernetesClientException

[AaruniAggarwal]

I am creating Strimzi-cluster-operator for my Scale tests.
All the pods are in Ready state , but Kafka is throwing error.

venv) [root@aaru-ocs-bf1b-bastion-0 ocs-ci]# oc get pods
NAME                                        READY   STATUS    RESTARTS   AGE
my-cluster-zookeeper-0                      1/1     Running   0          54s
my-cluster-zookeeper-1                      1/1     Running   0          54s
my-cluster-zookeeper-2                      1/1     Running   0          54s
strimzi-cluster-operator-78cb6c5fb8-dkgbj   1/1     Running   0          101s

But when I described Kafka , It was in NotReady state.

[root@aaru-ocs-bf1b-bastion-0 ocs-ci]# oc get Kafka my-cluster -o yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  creationTimestamp: "2021-03-29T13:52:28Z"
  generation: 1
  managedFields:
  - apiVersion: kafka.strimzi.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:entityOperator:
          .: {}
          f:topicOperator: {}
          f:userOperator: {}
        f:kafka:
          .: {}
          f:config:
            .: {}
            f:inter.broker.protocol.version: {}
            f:log.message.format.version: {}
            f:offsets.topic.replication.factor: {}
            f:transaction.state.log.min.isr: {}
            f:transaction.state.log.replication.factor: {}
          f:listeners: {}
          f:replicas: {}
          f:storage:
            .: {}
            f:type: {}
          f:version: {}
        f:zookeeper:
          .: {}
          f:replicas: {}
          f:storage:
            .: {}
            f:class: {}
            f:size: {}
            f:type: {}
    manager: kubectl-create
    operation: Update
    time: "2021-03-29T13:52:28Z"
  - apiVersion: kafka.strimzi.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:kafka:
          f:storage:
            f:volumes: {}
      f:status:
        .: {}
        f:conditions: {}
        f:observedGeneration: {}
    manager: okhttp
    operation: Update
    time: "2021-03-29T13:52:29Z"
  name: my-cluster
  namespace: myproject
  resourceVersion: "7386686"
  selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/myproject/kafkas/my-cluster
  uid: a04fc1cf-eafc-461f-8040-73f96bfb94d3
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      inter.broker.protocol.version: "2.7"
      log.message.format.version: "2.7"
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    replicas: 3
    storage:
      type: jbod
      volumes:
      - class: ocs-storagecluster-ceph-rbd
        deleteClaim: false
        id: 0
        size: 100Gi
        type: persistent-claim
    version: 2.7.0
  zookeeper:
    replicas: 3
    storage:
      class: ocs-storagecluster-ceph-rbd
      deleteClaim: false
      size: 100Gi
      type: persistent-claim
status:
  conditions:
  - lastTransitionTime: "2021-03-29T13:53:18.823Z"
    message: 'Failure executing: GET at: https://172.30.0.1/apis/networking.k8s.io/v1/namespaces/myproject/ingresses?labelSelector=strimzi.io%2Fcluster%3Dmy-cluster%2Cstrimzi.io%2Fname%3Dmy-cluster-kafka%2Cstrimzi.io%2Fkind%3DKafka. Message: Forbidden!Configured service account doesn''t have access. Service account may have been revoked. ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:myproject:strimzi-cluster-operator" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "myproject".'
    reason: KubernetesClientException
    status: "True"
    type: NotReady
  observedGeneration: 1

[AaruniAggarwal]

This is the error message ->
Message: Failure executing: GET at: https://172.30.0.1/apis/networking.k8s.io/v1/namespaces/myproject/ingresses?labelSelector=strimzi.io%2Fcluster%3Dmy-cluster%2Cstrimzi.io%2Fname%3Dmy-cluster-kafka%2Cstrimzi.io%2Fkind%3DKafka. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:myproject:strimzi-cluster-operator" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "myproject".

[AaruniAggarwal]

After creating strimzi-cluster-operator , All these resources got created :

[root@aaru-ocs-bf1b-bastion-0 ocs-ci]# oc get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/my-cluster-zookeeper-0                      1/1     Running   0          4m36s
pod/my-cluster-zookeeper-1                      1/1     Running   0          4m36s
pod/my-cluster-zookeeper-2                      1/1     Running   0          4m36s
pod/strimzi-cluster-operator-78cb6c5fb8-dkgbj   1/1     Running   0          5m23s

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/my-cluster-kafka-bootstrap    ClusterIP   172.30.13.144   <none>        9091/TCP,9092/TCP,9093/TCP   3m50s
service/my-cluster-kafka-brokers      ClusterIP   None            <none>        9091/TCP,9092/TCP,9093/TCP   3m50s
service/my-cluster-zookeeper-client   ClusterIP   172.30.42.88    <none>        2181/TCP                     4m36s
service/my-cluster-zookeeper-nodes    ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP   4m36s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/strimzi-cluster-operator   1/1     1            1           5m23s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/strimzi-cluster-operator-78cb6c5fb8   1         1         1       5m23s

NAME                                    READY   AGE
statefulset.apps/my-cluster-zookeeper   3/3     4m36s

And below are the service accounts:
[root@aaru-ocs-bf1b-bastion-0 ocs-ci]# oc get sa

NAME                       SECRETS   AGE
builder                    2         4m53s
default                    2         4m53s
deployer                   2         4m53s
my-cluster-kafka           2         3m16s
my-cluster-zookeeper       2         4m3s
strimzi-cluster-operator   2         4m51s

[AaruniAggarwal]

This is the testcase that I am running in my cluster which is inturn creating Strimzi-operator : https://github.com/red-hat-storage/ocs-ci/blob/master/tests/e2e/scale/test_scale_amq.py

And this is the file which is creating Strimzi-operator : https://github.com/red-hat-storage/ocs-ci/blob/master/ocs_ci/ocs/amq.py

[AaruniAggarwal]

@alok87 I saw you mentioned something wrong about my-project namespace in #3581. Could you please tell me what was the issue ? May be I am also having the same.

[scholzj]

What version of Kubernetes / OCP and what version of Strimzi are you using? This looks like you have either old version of the RBAC files with new operator version or you are running it on old unsupported Kubernetes version (We support Kube 1.16+)

[scholzj]

Anyway ... the RBAC above is what seems to be missing ... so just check the strimzi-cluster-operator-namespaced ClusterRole to see if it is there. And if not, maybe you can try to use the one from our repo to fix it.

[AaruniAggarwal]

Checked ClusterRole for strimzi-cluster-operator-namespaced :

Name:         strimzi-cluster-operator-namespaced
Labels:       app=strimzi
Annotations:  <none>
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  events                                          []                 []              [create]
  configmaps                                      []                 []              [get list watch create delete patch update]
  endpoints                                       []                 []              [get list watch create delete patch update]
  persistentvolumeclaims                          []                 []              [get list watch create delete patch update]
  pods                                            []                 []              [get list watch create delete patch update]
  secrets                                         []                 []              [get list watch create delete patch update]
  serviceaccounts                                 []                 []              [get list watch create delete patch update]
  services                                        []                 []              [get list watch create delete patch update]
  deploymentconfigs.apps.openshift.io/finalizers  []                 []              [get list watch create delete patch update]
  deploymentconfigs.apps.openshift.io/scale       []                 []              [get list watch create delete patch update]
  deploymentconfigs.apps.openshift.io/status      []                 []              [get list watch create delete patch update]
  deploymentconfigs.apps.openshift.io             []                 []              [get list watch create delete patch update]
  deployments.apps/scale                          []                 []              [get list watch create delete patch update]
  deployments.apps/status                         []                 []              [get list watch create delete patch update]
  deployments.apps                                []                 []              [get list watch create delete patch update]
  replicasets.apps                                []                 []              [get list watch create delete patch update]
  statefulsets.apps                               []                 []              [get list watch create delete patch update]
  buildconfigs.build.openshift.io/instantiate     []                 []              [get list watch create delete patch update]
  buildconfigs.build.openshift.io                 []                 []              [get list watch create delete patch update]
  builds.build.openshift.io                       []                 []              [get list watch create delete patch update]
  deployments.extensions/scale                    []                 []              [get list watch create delete patch update]
  deployments.extensions                          []                 []              [get list watch create delete patch update]
  ingresses.extensions                            []                 []              [get list watch create delete patch update]
  networkpolicies.extensions                      []                 []              [get list watch create delete patch update]
  replicasets.extensions                          []                 []              [get list watch create delete patch update]
  replicationcontrollers.extensions               []                 []              [get list watch create delete patch update]
  imagestreams.image.openshift.io/status          []                 []              [get list watch create delete patch update]
  imagestreams.image.openshift.io                 []                 []              [get list watch create delete patch update]
  kafkabridges.kafka.strimzi.io/status            []                 []              [get list watch create delete patch update]
  kafkabridges.kafka.strimzi.io                   []                 []              [get list watch create delete patch update]
  kafkaconnectors.kafka.strimzi.io/status         []                 []              [get list watch create delete patch update]
  kafkaconnectors.kafka.strimzi.io                []                 []              [get list watch create delete patch update]
  kafkaconnects.kafka.strimzi.io/status           []                 []              [get list watch create delete patch update]
  kafkaconnects2is.kafka.strimzi.io/status        []                 []              [get list watch create delete patch update]
  kafkaconnects2is.kafka.strimzi.io               []                 []              [get list watch create delete patch update]
  kafkaconnects.kafka.strimzi.io                  []                 []              [get list watch create delete patch update]
  kafkamirrormaker2s.kafka.strimzi.io/status      []                 []              [get list watch create delete patch update]
  kafkamirrormaker2s.kafka.strimzi.io             []                 []              [get list watch create delete patch update]
  kafkamirrormakers.kafka.strimzi.io/status       []                 []              [get list watch create delete patch update]
  kafkamirrormakers.kafka.strimzi.io              []                 []              [get list watch create delete patch update]
  kafkarebalances.kafka.strimzi.io/status         []                 []              [get list watch create delete patch update]
  kafkarebalances.kafka.strimzi.io                []                 []              [get list watch create delete patch update]
  kafkas.kafka.strimzi.io/status                  []                 []              [get list watch create delete patch update]
  kafkas.kafka.strimzi.io                         []                 []              [get list watch create delete patch update]
  networkpolicies.networking.k8s.io               []                 []              [get list watch create delete patch update]
  poddisruptionbudgets.policy                     []                 []              [get list watch create delete patch update]
  rolebindings.rbac.authorization.k8s.io          []                 []              [get list watch create delete patch update]
  roles.rbac.authorization.k8s.io                 []                 []              [get list watch create delete patch update]
  routes.route.openshift.io/custom-host           []                 []              [get list watch create delete patch update]
  routes.route.openshift.io                       []                 []              [get list watch create delete patch update]

[AaruniAggarwal]

There is one with the name ingresses.extensions

[scholzj]

Yeah, that is old one ... on OCP 4.7 / Kube 1.20 you need ingresses.networking... ... so I guess you can try to update the role with the file I linked to above and that should help.

[AaruniAggarwal]

Thank you very much @scholzj .