[helm] watchAnyNamespace flag seems to have no effect

[mjrepo2]

I have installed strimzi 0.20.1 with the supplied helm chart values (https://github.com/strimzi/strimzi-kafka-operator/blob/0.20.1/helm-charts/helm3/strimzi-kafka-operator/values.yaml) and changed watchAnyNamespace: false, to true.
Strimzi is installed into a kafka namespace, and I want to create a "kafkauser" resource in different namespace and have the strimzi operator consume the resource and create the user in kafka.
So far, this is not working in any namespace other than the namespace strimzi is installed into, even though I have the watchAnyNamespace flag set to true.

I can create the user in the same namespace strimzi is running, and it creates a secret from it and the user operator logs show it has seen and created the resources.
In any other namespace there is no effect and no log entry, am I missing something?

Kafka user yaml:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  namespace: newnamespace
  name: kafka-user
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  authentication:
    type: scram-sha-512

strimzi helm values yaml:

# Default values for strimzi-kafka-operator.

# If you set `watchNamespaces` to the same value as ``.Release.Namespace` (e.g. `helm ... --namespace $NAMESPACE`),
# the chart will fail because duplicate RoleBindings will be attempted to be created in the same namespace
watchNamespaces: []
watchAnyNamespace: true

image:
  repository: strimzi
  name: operator
  tag: 0.20.1
logVolume: co-config-volume
logConfigMap: strimzi-cluster-operator
logLevel: ${env:STRIMZI_LOG_LEVEL:-INFO}
fullReconciliationIntervalMs: 120000
operationTimeoutMs: 300000
kubernetesServiceDnsDomain: cluster.local

tolerations: []
affinity: {}
nodeSelector: {}

# Docker images that operator uses to provision various components of Strimzi.  To use your own registry prefix the
# repository name with your registry URL.
# Ex) repository: registry.xyzcorp.com/strimzi/zookeeper
zookeeper:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
kafka:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
kafkaConnect:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
kafkaConnects2i:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
topicOperator:
  image:
    repository: strimzi
    name: operator
    tag: 0.20.1
userOperator:
  image:
    repository: strimzi
    name: operator
    tag: 0.20.1
kafkaInit:
  image:
    repository: strimzi
    name: operator
    tag: 0.20.1
tlsSidecarEntityOperator:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
kafkaMirrorMaker:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
kafkaBridge:
  image:
    repository: strimzi
    name: kafka-bridge
    tag: 0.19.0
kafkaExporter:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
jmxTrans:
  image:
    repository: strimzi
    name: jmxtrans
    tag: 0.20.1
kafkaMirrorMaker2:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
cruiseControl:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
tlsSidecarCruiseControl:
  image:
    repository: strimzi
    name: kafka
    tagPrefix: 0.20.1
resources:
  limits:
    memory: 384Mi
    cpu: 1000m
  requests:
    memory: 384Mi
    cpu: 200m
livenessProbe:
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  initialDelaySeconds: 10
  periodSeconds: 30

# Override the docker image repository used by all Strimzi images
imageRepositoryOverride: ""
# Override the docker image tag used by all Strimzi images
imageTagOverride: ""
createGlobalResources: true

[scholzj]

The Helm Chart deploys the so called Cluster Operator (CO). CO operates all of the resources apart from KafkaUser and KafkaTopic. When deploying the Kafka cluster using the Kafka resource, you can deploy the User and Topic Operator as part of the Kafka cluster and only then you can use the KafkaUser and KafkaTopic resources. So if you haven't done it, you cannot really use them since there is nobody to operate them yet.

And even once you do it, the Topic and User operators (at least right now) operate on only one namespace. It is by default the same namespace in which the Kafka cluster is deployed or you can configure another namespace. But it will be always one. The reason is that Kafka does not have any concept of namespaces, so having the users or topics work across multiple namespaces is relatively complex and complicated.

[mjrepo2]

I have the CO up and running and the entity operator pod (which contains the user and topic operators), everything is green as far as I can tell.
But - you supplied the answer I think: I just want to confirm that strimzi cannot operate (in terms of topic/user operator) in multiple namespaces? So the kafka user would have to be in the same namespace as strimzi is installed?
Is it possible to generate the user the in one namespace, and copy the secret over to another namespace and it would work? I assume on the fly updating is no longer possible, but will strimzi/kafka accept the secret still?

Basically I have strimzi/kafka deployed in a kafka namespace, and have an application in a different namespace that I want to access kafka, however the kafkauser secret has to be in the same namespace as the application. This is the problem I am having - since strimzi will not work across multiple namespaces - how can I achieve the goal of having a kafkauser resource that is accessible from everywhere/application namespace?

Also to encapsulate - I know the helm portion is quite special and maybe not used by all: But why is there a flag to watch all namespaces when this doesn't/can't work? I'm a tad confused here :D

[scholzj]

Yes, the Topic and User Operators watch always only one namespace. It does not have to be the namespace where the Kafka cluster is (that is just default), but it is always only one namespace. There are different approaches how to deal with it. Most common is copying of the secrets to the application namespace - there are tools which can do this automatically. The https://github.com/strimzi/kafka-kubernetes-config-provider is another option how you can read the secrets across namespaces for example.

[mjrepo2]

Ok I will take a look at this. Thank you for your time, timely responses, and help!

[mjrepo2]

Is it possible to tell the user operator to watch a different namespace? I only see the option to tell strimzi to watchAnyNamespace or a specific one (which doesn't function properly as far as I can tell) where can I tell specifically the user/topic operator to look in a different namespace? I won't be creating users in the "kafka" namespace, so I don't mind if the operator doesn't look there, since I will create them in my applications namespace. Any tips here?

[scholzj]

Yes, check the watchedNamespace options here: https://strimzi.io/docs/operators/latest/full/using.html#proc-config-kafka-str