[Bug]: Kafka-init: User "system:serviceaccount:cluster-kafka" cannot get resource "nodes" in API group "" at the cluster scope.

[thanhhoan062]

Bug Description

Hi, I have problem when create kafka. Kafka-init not working

Steps to reproduce

1. Create Kafka:

kind: Kafka
apiVersion: kafka.strimzi.io/v1beta2
metadata:
  name: kafka-cluster
  namespace: app-kafka
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    resources:
      requests:
        cpu: 1000m
        memory: 1Gi
      limits:
        memory: 8Gi
        cpu: 4000m
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        tls: false
        type: nodeport
        configuration:
          bootstrap:
            nodePort: 31850
          brokers:
          - broker: 0
            nodePort: 31851
          - broker: 1
            nodePort: 31852
          - broker: 2
            nodePort: 31853
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.4'
    storage:
      class: thin
      deleteClaim: true
      size: 20Gi
      type: persistent-claim
  zookeeper:
    replicas: 3
    resources:
      requests:
        cpu: 1000m
        memory: 1Gi
      limits:
        cpu: 4000m
        memory: 8Gi
    storage:
      class: thin
      deleteClaim: true
      size: 10Gi
      type: persistent-claim
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

2. Zookeeper running success
3. Init kafka cluster

Expected behavior

No response

Strimzi version

0.34.0

Kubernetes version

OpenShift 4.10.38

Installation method

No response

Infrastructure

OpenShift

Configuration files and logs

2023-05-30 02:03:35 INFO  Main:26 - Init-kafka 0.35.0 is starting
2023-05-30 02:03:36 INFO  Main:31 - Init-kafka started with config: InitWriterConfig(nodeName=dr-m3-ocp-dev02-kz4jw-worker-wllzg,rackTopologyKey=null,externalAddress=true,initFolder=/opt/kafka/init,addressType=null)
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1:443/api/v1/nodes/dr-m3-ocp-dev02-kz4jw-worker-wllzg. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. nodes "dr-m3-ocp-dev02-kz4jw-worker-wllzg" is forbidden: User "system:serviceaccount:app-kafka:kafka-cluster-kafka" cannot get resource "nodes" in API group "" at the cluster scope.
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:517)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:534)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:481)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:745)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:186)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:142)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:93)
	at io.strimzi.kafka.init.InitWriter.writeExternalAddress(InitWriter.java:71)
	at io.strimzi.kafka.init.Main.main(Main.java:42)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1:443/api/v1/nodes/dr-m3-ocp-dev02-kz4jw-worker-wllzg. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. nodes "dr-m3-ocp-dev02-kz4jw-worker-wllzg" is forbidden: User "system:serviceaccount:app-kafka:cluster-kafka" cannot get resource "nodes" in API group "" at the cluster scope.
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:669)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:649)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:596)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:559)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$9(StandardHttpClient.java:136)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$9(StandardHttpClient.java:136)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$retryWithExponentialBackoff$13(StandardHttpClient.java:193)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844)
	at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Additional context

No response

[scholzj]

This means that the operator (or in this case the init container) does not have enough RBAC rights to get the information from the node. The Cluster operator normally grants it the rights through a ClusterRoleBinding. So my guess based on previous issues is that some tooling in your cluster deletes the ClusterRoleBinding causing this issue. If you provide a DEBUG log from the Cluster Operator, we should be able to see it from there.