[RBAC] Strimzi-Cluster-Operator: Full permission set on /status needed?

[timsn]

Hi,

currently we are having some trouble to get the strimzi-operator to run in our (a bit more restrictive) managed cluster.
The problem we are facing is related to RBAC and we are observing errors in the logs. The error is attached below.

If I understand it correctly the operator is trying to create and bind roles that include the verbs: ["list", "watch", "create", "delete"] for resources like kafkatopics/status, kafkausers/status and a few others for the creation of the EntityOperator.

But in our case the permissions/roles the ServiceAccount is granted are auto generated by our managed cluster. We assume by using the CLI "rbac-tool". This generated ruleset only contains ["get", "patch", "update"] for the stated resources from above (see attached command below). And therefore we are not able to grant more permissions than that from inside the running operator.

So I guess the main question here is, is this full permission set including: ["list", "watch", "create", "delete"] really needed for /status /scale resources and maybe others? Or are they only part of the permission set out of convenience?

Additionally it would be interesting to understand where the role the cluster-operator creates originates from. And whether it can be configured/altered in some way?

Any input would be highly appreciated.

The rbac-tool outputs the following example Role: rbac-tool show --for-groups=kafka.strimzi.io

[...]
- apiGroups:
  - kafka.strimzi.io
  resources:
  - kafkatopics/status
  verbs:
  - get
  - patch
  - update
[...]

Source: https://github.com/alcideio/rbac-tool/

strimzi-cluster-operator log:

2023-06-29 12:03:58 WARN  AbstractOperator:532 - Reconciliation #8(timer) Kafka(<namespace>/strimzi): Failed to reconcile
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://<ip>/apis/rbac.authorization.k8s.io/v1/namespaces/<namespace>/roles/strimzi-entity-operator. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. roles.rbac.authorization.k8s.io "strimzi-entity-operator" is forbidden: user "system:serviceaccount:strimzi-operator:strimzi-cluster-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:strimzi-operator" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["kafka.strimzi.io"], Resources:["kafkatopics/status"], Verbs:["list" "watch" "create" "delete"]}
{APIGroups:["kafka.strimzi.io"], Resources:["kafkausers/status"], Verbs:["list" "watch" "create" "delete"]}.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:610) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:555) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:411) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:372) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handlePatch(BaseOperation.java:654) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$patch$3(HasMetadataOperation.java:243) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:248) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:258) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:43) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.Patchable.patch(Patchable.java:35) ~[io.fabric8.kubernetes-client-5.12.2.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.internalPatch(AbstractResourceOperator.java:245) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
	at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.internalPatch(AbstractResourceOperator.java:239) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
	at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$reconcile$0(AbstractResourceOperator.java:116) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

[scholzj]

I'm not sure I really understand which permissions it does or does not need. TBH, I cannot confirm out of my head that the list operation is not needed under any circumstances.

But the particular error you get is I think because you badly edited the rights. You took it from the Cluster Operator, but not from the Cluster Roles it assigns to the other components. So if you are going to remove it, you should remove it from everywhere.

[scholzj]

The cluster roles are part of the installation files. I guess this one is likely related to the particular error you have: https://github.com/strimzi/strimzi-kafka-operator/blob/main/install/cluster-operator/031-ClusterRole-strimzi-entity-operator.yaml. So you should not need to edit the source codes. You just need to edit the role and keep it in sync with the rights you gave tot he operator.

[scholzj]

As I said ... I do not understand how you decided what roles does it need and which not -> so what I'm suggesting just solves the particular error you shared above. No idea if it is really going to work or runs into another error next.

[timsn]

We seem to have found the source of our problem. The roles generated by the cluster-operator seem to be hard-coded in the EntityOperator.generateRole(). Can you confirm this?

That would mean the cluster-operator tries to override our customized role which leads to the error seen above.

Do you see any option to customize this role as well or to disable the role creation by the cluster-operator? We really would like to avoid building our own image just to adapt the role definition.

[scholzj]

Hmm, you are right. I forgot about that I guess you would need to modify the resources and rebuild the operator from the sources.

[timsn]

Alright, we will then rebuild the operator from the sources. Thank you very much for the clarification and your help!

[scholzj]

If you want, you can also open an issue on the RBAC rights which you think are not needed and someone can investigate it and eventually remove it if they are really not needed.