Removing operator permission to create ClusterRoleBindings?

[bradw456]

Hi,

I wanted to remove the permission that grants the operator the ability to create/patch/update/delete ClusterRoleBindings. This config is too permissive to run in our org's multi-tenant environment.

I understand this is needed by the operator to generate a ClusterRoleBinding for each new kafka cluster. And I understand it allows the broker init sequence to do a "get" on nodes so that it can configure rack information. We definitely want to continue to be able to do this so we will use other methods to grant clusters this permission.

I've tried removing create/patch/update/delete from the strimzi-cluster-operator-global ClusterRole but have run into
some trouble.

It seems that, in addition to the warning messages pasted below, it also seems that the failures are causing reconciliation to fail. So spinning up a new kafka cluster stops and the brokers do not start.

Based on some earlier issues and PRs it seems like removing this permission is desirable. It almost seems like the code should allow this because of the use of
withIgnoreRbacError but sadly it appears it only applies in the case of deletes or when rack/nodeport is not enabled.

What is the feasibility of always ignoring this rbac error or adding a flag to do so? Perhaps even better, a flag to avoid trying to create this rbac at all?

Thanks!
Brad

Steps to reproduce after a vanilla install on minikube:

# Modify the cluster role to remove crud ops
$ k edit clusterroles strimzi-cluster-operator-global

$ k get clusterrole strimzi-cluster-operator-global -o yaml | grep -A5 -B3 clusterrolebindings
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  verbs:
  - get
  - list
  - watch
- apiGroups:
$

# Use a rack configuration

$ k label node minikube topology.kubernetes.io/zone=1
node/minikube labeled

$ diff -u kafka-persistent-single.yaml kafka-persistent-single-rack.yaml
--- kafka-persistent-single.yaml	2021-04-29 09:42:57.612434399 -0400
+++ kafka-persistent-single-rack.yaml	2021-04-29 09:40:49.387446440 -0400
@@ -5,6 +5,8 @@
 spec:
   kafka:
     version: 2.7.0
+    rack:
+      topologyKey: topology.kubernetes.io/zone
     replicas: 1
     listeners:
       - name: plain

$ k apply -f kafka-persistent-single-rack.yaml
kafka.kafka.strimzi.io/my-cluster created

$ k logs --tail 50 -n kafka strimzi-cluster-operator-56bf6d9bf-sz9sr
2021-04-29 14:15:35 DEBUG KafkaAssemblyOperator:3399 - Rolling update of kafka/my-cluster-zookeeper: pod my-cluster-zookeeper-0 has strimzi.io/custom-listener-cert-thumbprints=; sts has strimzi.io/custom-listener-cert-thumbprints=
2021-04-29 14:15:35 DEBUG KafkaAssemblyOperator:2700 - Reconciliation #381(watch) Kafka(kafka/my-cluster): Checking readiness of pod my-cluster-zookeeper-0.
2021-04-29 14:15:35 DEBUG Util:105 - Waiting for Pods resource my-cluster-zookeeper-0 in namespace kafka to get ready
2021-04-29 14:15:35 DEBUG Util:127 - Pods resource my-cluster-zookeeper-0 in namespace kafka is ready
2021-04-29 14:15:35 DEBUG KafkaAssemblyOperator:1389 - Reconciliation #381(watch) Kafka(kafka/my-cluster): Verifying that Zookeeper is configured to run with 1 replicas
2021-04-29 14:15:35 DEBUG ZookeeperScaler:64 - Creating Zookeeper Scaler for cluster my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181
2021-04-29 14:15:35 DEBUG Util:105 - Waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181 to get connected
2021-04-29 14:15:35 DEBUG ZookeeperScaler:142 - Received event WatchedEvent state:SyncConnected type:None path:null from ZooKeeperAdmin client connected to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181
2021-04-29 14:15:36 DEBUG Util:127 - ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181 is connected
2021-04-29 14:15:36 DEBUG ZookeeperScaler:202 - Current Zookeeper configuration is {server.1=my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2888:3888:participant;127.0.0.1:12181}
2021-04-29 14:15:36 DEBUG ZookeeperScaler:185 - The Zookeeper server configuration is already up to date
2021-04-29 14:15:36 DEBUG ZookeeperScaler:142 - Received event WatchedEvent state:Closed type:None path:null from ZooKeeperAdmin client connected to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181
2021-04-29 14:15:36 DEBUG Util:105 - Waiting for Endpoints resource my-cluster-zookeeper-client in namespace kafka to get ready
2021-04-29 14:15:36 DEBUG Util:127 - Endpoints resource my-cluster-zookeeper-client in namespace kafka is ready
2021-04-29 14:15:36 DEBUG Util:105 - Waiting for Endpoints resource my-cluster-zookeeper-nodes in namespace kafka to get ready
2021-04-29 14:15:36 DEBUG Util:127 - Endpoints resource my-cluster-zookeeper-nodes in namespace kafka is ready
2021-04-29 14:15:36 DEBUG NetworkPolicyOperator:106 - NetworkPolicy kafka/my-cluster-network-policy-kafka already exists, patching it
2021-04-29 14:15:36 DEBUG NetworkPolicyOperator:188 - NetworkPolicy my-cluster-network-policy-kafka in namespace kafka has been patched
2021-04-29 14:15:36 DEBUG PvcOperator:106 - PersistentVolumeClaim kafka/data-0-my-cluster-kafka-0 already exists, patching it
2021-04-29 14:15:36 DEBUG PvcOperator:188 - PersistentVolumeClaim data-0-my-cluster-kafka-0 in namespace kafka has been patched
2021-04-29 14:15:36 DEBUG ServiceAccountOperator:106 - ServiceAccount kafka/my-cluster-kafka already exists, patching it
2021-04-29 14:15:36 DEBUG ServiceAccountOperator:34 - ServiceAccount my-cluster-kafka in namespace kafka has not been patched: patching service accounts generates new tokens which should be avoided.
2021-04-29 14:15:36 DEBUG ClusterRoleBindingOperator:94 - ClusterRoleBinding strimzi-kafka-my-cluster-kafka-init does not exist, creating it
2021-04-29 14:15:36 DEBUG ClusterRoleBindingOperator:188 - Caught exception while creating ClusterRoleBinding strimzi-kafka-my-cluster-kafka-init
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.96.0.1/apis/rbac.authorization.k8s.io/v1/clusterrolebindings. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kafka:strimzi-cluster-operator" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:250) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:871) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:366) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:85) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractNonNamespacedResourceOperator.internalCreate(AbstractNonNamespacedResourceOperator.java:184) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1]
	at io.strimzi.operator.common.operator.resource.AbstractNonNamespacedResourceOperator.lambda$reconcile$0(AbstractNonNamespacedResourceOperator.java:95) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-29 14:15:36 ERROR AbstractOperator:275 - Reconciliation #381(watch) Kafka(kafka/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: Failure executing: POST at: https://10.96.0.1/apis/rbac.authorization.k8s.io/v1/clusterrolebindings. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kafka:strimzi-cluster-operator" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope.
2021-04-29 14:15:36 DEBUG CustomResource:157 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-04-29 14:15:36 DEBUG StatusDiff:43 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2021-04-29T14:15:36.748Z"}
2021-04-29 14:15:36 DEBUG AbstractOperator:355 - Reconciliation #381(watch) Kafka(kafka/my-cluster): Status did not change
2021-04-29 14:15:36 DEBUG CustomResource:157 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-04-29 14:15:36 DEBUG AbstractOperator:544 - Reconciliation #381(watch) Kafka(kafka/my-cluster): Updated metric strimzi.resource.state[tag(kind=Kafka),tag(name=my-cluster),tag(resource-namespace=kafka)] = 0
2021-04-29 14:15:36 WARN  AbstractOperator:508 - Reconciliation #381(watch) Kafka(kafka/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: Failure executing: POST at: https://10.96.0.1/apis/rbac.authorization.k8s.io/v1/clusterrolebindings. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kafka:strimzi-cluster-operator" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope.
2021-04-29 14:15:36 DEBUG AbstractOperator:405 - Reconciliation #381(watch) Kafka(kafka/my-cluster): Lock lock::kafka::Kafka::my-cluster released
$

[scholzj]

It is fairly simple ... if you want to use the features which need the ClusterRoleBindings, you have to give the operator the rights to create it. If you do not want to give it the permissions, then fine, do not do it, but do not use the features which need it. In your case, rackAwareness is such feature. So you cannot have both.

This config is too permissive to run in our org's multi-tenant environment.

The ClusterRoleBinding RBAC allows the operator to only grant the right the operator has. So in this case it allows the cluster operator to give the Kafka brokers / Connect clusters the right to get nodes (or in theory get the storage classes). It does not allow it to grant anything else. I guess whether this is too permissive or not would be individual. But without these rights you cannot use the features depending on them.

[bradw456]

Ah ok thanks for clarifying. I missed the safeguard that it can't expand upon its own permissions, thanks for pointing that out. That should limit our concerns then. Thank you.

[scholzj]

Just FYI: Here are the official docs for it - https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping.