Unable to deploy Kafka when createGlobalResources is set to false

[mohitruhaan]

Hello Experts,

While trying to deploy Strimzi version 0.27, we tried to set the flag: createGlobalResources to "false" in Values.yaml.
To our understanding, it means strimzi should work fine when used for managing Kafka within a namespace, this not needing global resources.
But we are not able to deploy Kafka within a namespace. We observe service account and authorization related errors as below:

2022-02-01 10:55:53 ERROR AbstractOperator:432 - Reconciliation #2(watch) Kafka(databus/databus-cluster): Reconciliation completion handler failed
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/databus/kafkas/databus-cluster. Message: Forbidden!Configured _**service account doesn't have access. Service account may have been revoked.**_ kafkas.kafka.strimzi.io "databus-cluster" is forbidden: User "system:serviceaccount:databus:strimzi-cluster-operator" cannot get resource "kafkas" in API group "kafka.strimzi.io" in the namespace "databus": **_RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found]._**

Could you please suggest? Thanks!

[scholzj]

To our understanding, it means strimzi should work fine when used for managing Kafka within a namespace, this not needing global resources.

That is not correct. IIRC, what this mode does is that it will not install the global resources such as the ClusterRoles or ClusterRoleBindings. But it expects you to do so outside of the Helm Installation (at least for the ClusterRoles and CRDs). See #1436 for more details.

You can in theory not install the ClusterRoleBindings and have some basic functionality. But there will be some limitations to what the operator can do (no rack awareness, storage resizing, node port listeners). You would also still need to install the ClusterRoles for it.

[scholzj]

How does your Kafka / KafkaConnect custom resource look like?

[mohitruhaan]

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name:  {{ include "databus.fullname" . }}-cluster
spec:
  kafka:
    template:
      persistentVolumeClaim:
        metadata:
          annotations:
            <<removed>>
    version: 2.8.0
    replicas: {{.Values.replicas }}
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      # - name: tls
      #   port: 9093
      #   type: internal
      #   tls: true
      #   configuration:
      #     brokerCertChainAndKey:
      #       secretName: databus-ca
      #       certificate: ca.crt
      #       key: ca.key
      - name: external
        port: 9094
        type: nodeport
        tls: false
        configuration:
          # brokerCertChainAndKey:
          #   secretName: databus-ca
          #   certificate: ca.crt
          #   key: ca.key
          bootstrap:
            nodePort: 32100
          {{- with .Values.brokerConfig }}
            {{- toYaml . | nindent 10 }}
          {{- end }}
    {{- with .Values.kafkaResources }}
    resources:
      {{- toYaml . | nindent 6 }}
    {{- end }}
    {{- with .Values.kafkaConfig }}
    config:
      {{- toYaml . | nindent 6 }}
    {{- end }}
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: {{  .Values.kafkaStorage.size  }}
        deleteClaim: false
  zookeeper:
    template:
      persistentVolumeClaim:
        metadata:
          annotations:
              << removed >>
    replicas: {{.Values.zookeeper.replicas }}
     {{- with .Values.zookeeperResources }}
    resources:
      {{- toYaml . | nindent 6 }}
    {{- end }}
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    storage:
      type: persistent-claim
      size: {{  .Values.zookeeperStorage.size  }}
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

[scholzj]

Well, as I said, you cannot use some of the features when you don't install it properly. In this case, the node port listener would for sure be the issue. You have to remove it / use another listener type instead. Rack awareness would be another issue for example which would not work.

[deshpande-pavan]

Dear @scholzj, hope you are doing well. Continuing from where Mohit left,
 
I have changed the ClusterRoles/ClusterRoleBindings to Role/RoleBindings in templates folder. When I deployed in the strimzi- cluster-operator, I got the error mentioned by Mohit.
 
Tracing steps:-
I traced it back to createOrUpdate() method of AssemblyOperator class, which goes to one of its implementations, i.e., KafkaAssemblyOperator.
Further traced it to reconcile method, where it is invoking kafkaInitClusterRoleBinding. This method generates the clusterRoleBinding for namespace passed as argument. While generating, it checks rack and isExposedWithNodePort(). I see that rack is null and isExposedWithNodePort() is true.
If rack is null, we don't want kafka init to create clusterRoleBindings. So I added a condition to check if rack is null, if its null, then I am returning null. As a result, desired variable is null. When desired is passed to withIgnoreRbacError method, it returns Future{result=null}. In turn kafkaInitClusterRoleBinding method of class returns Future{unresolved}.
 
When I do this, I see that the strimzi-cluster-operator pod is up with below errors-
 
ERROR AbstractOperator:245 - Reconciliation #1(watch) Kafka(databus/databus-cluster): createOrUpdate failed
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/nodes. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. nodes is forbidden: User "system:serviceaccount:databus:strimzi-cluster-operator" cannot list resource "nodes" in API group "" at the cluster scope.
 
Could you please help me to check if the direction I am going is the right one to achieve the result?

FYI - we are using strimzi-cluster-operator 0.27.1.

[scholzj]

I'm not entirely sure I follow your question TBH. The Strimzi operator installation files use the minimum required permissions which Strimzi needs. We do not ask for these permissions just for fun, we need them. So when you play with the RBAC files and decide to remove some of the permissions as you did, some features will not work. This for example includes node port listeners, rack awareness or storage resizing. So you simply cannot use these features if you don't give the operator the permissions.

[planetf1]

I also initially tried this.

There are two scenarios where it would be useful (related)

a) In development -- two sub-scenarios (where I have my own cluster/full permissions as needed)
i) Just testing my apps (helm charts). Typically one-per namespace. Sharing cluster resources ok - BUT by default the second one will fail, it would be useful to have a 'create if does not exist, but do not fail' option
ii) Testing my charts fully. One per namespace, but wanting to keep everything independent, including the deployment of the strimzi operator (could be different versions) and no cluster scope resources
b) Tutorials/education

• limited access. For example I'm trying to build a course on a hosted platform & I get generated a k8s namespace to use. But I cannot deploy cluster scoped resources at all (similar to a i) above in effect)

One possible mitigation would be use of helm conditions so that my parent chart could at least use a value to specify whether to include the strimzi chart or not. Then in the documentation the steps could be either:
a) Default - just install everything as we do today (strimzi operator/crd, & our own app)
b) Skip strimzi - so i) User installs strimzi manually as they like (or by admin) ii) they then install our app.
though this approach doesn't fully address the cluster scope resource limitation

[planetf1]

.. but this is a desired end-state. Given the limitation in k8s that customresourcedefinitions are inherently cluster scoped there's always a limitation. Maybe the helm condition is the way forward for us (and also simple - but needs a strimzi change), and ensure that in the hosted/limited environments we get strimzi setup once by admins.

[jnt2007]

Hi. Just trying to find a correct solution for our needs.

We need to have 2 operators in one cluster in 2 different namespaces. Deployment configuration should be the same. When I'm trying to install the Strimzi operator into a second namespace I'm getting an error with a naming conflict that cluster resources owned by the first namespace. I understand that I can set createGlobalResources to false for the second namespace or create global resources externally...

But what about to add into global resources name preffix/suffix which can be set by values of the helm chart?