Deploying multiple cluster operators to single cluster using Helm

[poksiala]

Hi.

TL;DR: What are the best practices settings for helm installation so that everyone can have their namespace scoped Strimzi installation which can be upgraded independently?

We have k8s 1.20 cluster for development, where each developer has their own namespace for testing purposes. We are using Helm v3 for distribution and deployment so it would be easiest to just add Strimzi to chart dependencies. All trials below are using Strimzi chart version 0.21.1.

Even though method in Trial no. 3 works, it looks to me like this is some kind of oversight or unintended behaviour.

Trial no. 1: no configuration

When running helm install -n alice alice strimzi/strimzi-kafka-operator cluster operator starts without problems in alice namespace. Problems arise when running helm install -n bob bob strimzi/strimzi-kafka-operator

W0210 15:30:12.117785   55410 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0210 ...
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "strimzi-cluster-operator-namespaced" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "bob": current value is "alice"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "bob": current value is "alice"

Trial no. 2: createGlobalResources=false

The first trial indicates that there is some conflicting global resources so let's try helm install -n alice alice strimzi/strimzi-kafka-operator --set createGlobalResources=false and helm install -n bob bob strimzi/strimzi-kafka-operator --set createGlobalResources=false

Both installations succeed but in both namespaces strimzi-cluster-operator is crashlooping.
It seems to boil down to missing ClusterRoles.

Status: 403 - kafkaconnects.kafka.strimzi.io is forbidden: User "system:serviceaccount:alice:strimzi-cluster-operator" cannot watch resource "kafkaconnects" in API group "kafka.strimzi.io" in the namespace "alice": RBAC: [clusterrole.rbac.authorization.k8s.io "strimzi-topic-operator" not found, clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" not found, clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" not found]

Trial no. 3: Create global resources in one installation

Running helm install -n alice alice strimzi/strimzi-kafka-operator and helm install -n bob bob strimzi/strimzi-kafka-operator --set createGlobalResources=false both succeed and both cluster operators work and are able to create Kafka-clusters.

Though this behaviour seems like it is not the intended solution and I'm worried wether it might stop working at some point. Also, what happens if due to one namespace is upgraded and those global resources go out of sync?

[scholzj]

I'm not sure this is possible with our Helm Chart. There are some ClusterRoleBindings (for example https://github.com/strimzi/strimzi-kafka-operator/blob/master/helm-charts/helm3/strimzi-kafka-operator/templates/021-ClusterRoleBinding-strimzi-cluster-operator.yaml) which have a fixed name. And these will be created to point to the service account used by the operator. So when you install the Helm Chart multiple times, these resources will be overwriting each other. So only one operator will have the correct RBAC rights and the others will be getting errors.

I guess the createGlobalResources=false doesn't create the cluster-scoped resources at all and that is why nothing woks in trial 2. In trial 3 you create them for one of the operators but not for the others. I would expect the first operator to work completely fine. The second one would work to some extend => it will be missing the ClusterRoleBindings and the features which it needs them for will not work. This would include rack awareness or node-port listeners for example.

This can be in theory worked around by naming the ClusterRoleBinding resources differently - for example including the namespace of the operator in their name. But I guess the error from trial 1 would still prevent it from installing it properly. And TBH I do not know Helm well enough to understand if that can be worked around somehow.

[poksiala]

Okay, so the answer is "Currently not supported."

I think I could provide a PR to address this on the Helm side, but if there are some hard coded values in the application code I might not be of much help. If there is interest in supporting this scenario, that is.

[scholzj]

I think there are three kind of resources:

• The shared resources by all operators operators: CRDs and ClusterRoles => not entirely sure how to deal with these in Helm due to lack of Helm knowledge on my side. You need to have these installed ... and you need to make sure these are deleted only when the last operator is deleted (deleting the CRDs deletes all Kafka clusters)
• The non-namespaced resources specific to each operator (the ClusterRoleBindings) => I think naming them with the namespace included in the name should be an easy solution
• The namespaced resources should be fine as they are

The createGlobalResources option was originally designed for another purpose ... for developers running small non-production clusters. That is why it does not match your usecase really.

[scholzj]

Obviously, the shared-resources will be shared by the operators. For the ClusterRoles, we could in theory change that and make their names configurable. But the CRDs are Kubernetes limitation which cannot be fixed in Strimzi itself. If you need to change the way the operator deals with the ClusterRoles, I can help of course. But I'm not sure it would help you that much since you still would need to solve the same issue for the CRDs via Helm only.

[selsinan]

Hello @scholzj,

We are trying to deploy multiple kafka-connect with each one seperated unique namespaces. Only first installation works fine but the rest fails. Is there any limitation/suggestion that we should follow? Thanks a lot for your help,

Error Message:
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://[fdc1:13e1:68d5::1]:443/apis/coordination.k8s.io/v1/namespaces/ss-kafka/leases/strimzi-cluster-operator. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. leases.coordination.k8s.io "strimzi-cluster-operator" is forbidden: User "system:serviceaccount:ss-kafka:strimzi-cluster-operator" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ss-kafka".
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:742) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:722) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:671) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:601) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]

[scholzj]

I'm not sure I fully understand the context and the relation to the original discussion. The only thing you shared is an RBAC error. So the RBAC setup is what you should look into.