[Bug]: Helm Chart couldn't apply watchNamespaces option

[tankim]

Bug Description

When I tried to install strimzi-kafka-operator via helm chart with custom values file,
below option is not applied.

watchNamespaces: []
watchAnyNamespace: true

values file, adding something like this seems to work.

strimzi-kafka-operator:
  watchAnyNamespace: true

Steps to reproduce

# values.yaml
watchNamespaces: []
watchAnyNamespace: true

helm upgrade --install strimzi-kafka-operator . -f values.yaml -n kafka

When run as above, the operator will only monitor the kafka namespace.

# values.yaml
#watchNamespaces: []
#watchAnyNamespace: true
strimzi-kafka-operator:
  watchAnyNamespace: true

helm upgrade --install strimzi-kafka-operator . -f values.yaml -n kafka

When run as above, the operator will monitor all namespaces.

No response

Expected behavior

No response

Strimzi version

0.34.0

Kubernetes version

1.23

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Can you please share the exact steps to reproduce it?

[tankim]

@scholzj edited comment I wrote.

[scholzj]

I tried the following commands:

helm install cluster-operator strimzi/strimzi-kafka-operator -f values.yaml -n kafka

and

helm upgrade --install cluster-operator strimzi/strimzi-kafka-operator -f values.yaml -n kafka

and both seem to work completely fine. The values.yaml file is as you suggested:

watchNamespaces: []
watchAnyNamespace: true

It deploys the operator watching all namespaces.

[tankim]

can you share whole the values.yaml you tested?

[scholzj]

That ^^^ is exactly what is in my values.yaml file as that is what you suggested as a reproducer.

[tan-kim]

This is my whole values.yaml.

# Default values for strimzi-kafka-operator.

# If you set `watchNamespaces` to the same value as ``.Release.Namespace` (e.g. `helm ... --namespace $NAMESPACE`),
# the chart will fail because duplicate RoleBindings will be attempted to be created in the same namespace
watchNamespaces: []
watchAnyNamespace: true
#strimzi-kafka-operator:
#  watchAnyNamespace: true

image:
  registry: quay.io
  repository: strimzi
  name: operator
  tag: 0.34.0
logVolume: co-config-volume
logConfigMap: strimzi-cluster-operator
logLevel: ${env:STRIMZI_LOG_LEVEL:-INFO}
fullReconciliationIntervalMs: 120000
operationTimeoutMs: 3000000
kubernetesServiceDnsDomain: cluster.local
featureGates: ""

tolerations: []
affinity: {}
annotations: {}
labels: {}
nodeSelector: {}

podSecurityContext: {}
securityContext: {}

# Docker images that operator uses to provision various components of Strimzi.  To use your own registry prefix the
# repository name with your registry URL.
# Ex) repository: registry.xyzcorp.com/strimzi/zookeeper
zookeeper:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
kafka:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
kafkaConnect:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
topicOperator:
  image:
    registry: quay.io
    repository: strimzi
    name: operator
    tag: 0.34.0
userOperator:
  image:
    registry: quay.io
    repository: strimzi
    name: operator
    tag: 0.34.0
kafkaInit:
  image:
    registry: quay.io
    repository: strimzi
    name: operator
    tag: 0.34.0
tlsSidecarEntityOperator:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
kafkaMirrorMaker:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
kafkaBridge:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka-bridge
    tag: 0.20.2
kafkaExporter:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
jmxTrans:
  image:
    registry: quay.io
    repository: strimzi
    name: jmxtrans
    tag: 0.34.0
kafkaMirrorMaker2:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
cruiseControl:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
tlsSidecarCruiseControl:
  image:
    registry: quay.io
    repository: strimzi
    name: kafka
    tagPrefix: 0.34.0
kanikoExecutor:
  image:
    registry: quay.io
    repository: strimzi
    name: kaniko-executor
    tag: 0.34.0
resources:
  limits:
    memory: 384Mi
    cpu: 1000m
  requests:
    memory: 384Mi
    cpu: 200m
livenessProbe:
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  initialDelaySeconds: 10
  periodSeconds: 30

# Override the docker registry used by all Strimzi images
imageRegistryOverride: ""
# Override the docker image repository used by all Strimzi images
imageRepositoryOverride: ""
# Override the docker image tag used by all Strimzi images
imageTagOverride: ""
createGlobalResources: true
# Override the exclude pattern for exclude some labels
labelsExclusionPattern: ""
# Controls whether Strimzi generates network policy resources (By default true)
generateNetworkPolicy: true
# Override the value for Connect build timeout
connectBuildTimeoutMs: 3000000

And this is log in strimzi-cluster-operator.
2023-05-22 23:37:11 INFO ClusterOperator:136 - Setting up periodic reconciliation for namespace kafka

[tan-kim]

# values.yaml
watchAnyNamespace: true

# strimzi operator logs
Main:82 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[kafka],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000

# values.yaml
strimzi-kafka-operator:
  watchAnyNamespace: true

# strimzi operator logs
Main:82 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[*],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000

[scholzj]

I guess a start would be to share the full log from the operator & full log from the installation and the YAMLs it generated. You should probably also consider removing from the YAML file the things which you are not changing. That will make it much more obvious what you changed.

[tankim]

I'm passing along the full log here with watchAnyNamespace value only.
strimzi-cluster-operator-795f85cf5f-fz6wd.log

The first time I tested it, I included values that didn't change, but after that I didn't include any settings in the YAML file other than watchAnyNamespace.

[scholzj]

I think there is some issue in your environment. Because for me it simply works:

$
$ minikube start
😄  minikube v1.27.0 on Darwin 13.3.1 (arm64)
❗  Kubernetes 1.25.0 has a known issue with resolv.conf. minikube is using a workaround that should work for most use cases.
❗  For more information, see: https://github.com/kubernetes/kubernetes/issues/112135
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=7788MB) ...

🧯  Docker is nearly out of disk space, which may cause deployments to fail! (94% of capacity). You can pass '--force' to skip this check.
💡  Suggestion:

    Try one or more of the following to free up space on the device:

    1. Run "docker system prune" to remove unused Docker data (optionally with "-a")
    2. Increase the storage allocated to Docker for Desktop by clicking on:
    Docker icon > Preferences > Resources > Disk Image Size
    3. Run "minikube ssh -- docker system prune" if using the Docker container runtime
🍿  Related issue: https://github.com/kubernetes/minikube/issues/9024

🐳  Preparing Kubernetes v1.25.0 on Docker 20.10.17 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$
$
$ kubectl create ns kafka
namespace/kafka created
$
$
$ helm version
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/scholzj/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/scholzj/.kube/config
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.18.2"}
$
$
$
$ helm repo add strimzi https://strimzi.io/charts/
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/scholzj/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/scholzj/.kube/config
"strimzi" already exists with the same configuration, skipping
$
$
$ cat values.yaml
watchAnyNamespace: true
$
$
$ helm install cluster-operator strimzi/strimzi-kafka-operator -f values.yaml -n kafka
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/scholzj/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/scholzj/.kube/config
NAME: cluster-operator
LAST DEPLOYED: Tue May 23 09:41:35 2023
NAMESPACE: kafka
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing strimzi-kafka-operator-0.35.0

To create a Kafka cluster refer to the following documentation.

https://strimzi.io/docs/operators/latest/deploying.html#deploying-cluster-operator-helm-chart-str
$
$
$
$ kubectl get pods -n kafka
NAME                                        READY   STATUS              RESTARTS   AGE
strimzi-cluster-operator-7d679699bc-gtltz   0/1     ContainerCreating   0          8s
$
$
$
$
$ kubectl logs strimzi-cluster-operator-7d679699bc-gtltz -n kafka
Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath :lib/io.strimzi.cluster-operator-0.35.0.jar:lib/io.strimzi.api-0.35.0.jar:lib/io.fabric8.kubernetes-model-common-6.6.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.6.1.jar:lib/io.strimzi.config-model-0.35.0.jar:lib/io.strimzi.operator-common-0.35.0.jar:lib/io.strimzi.crd-annotations-0.35.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.openshift-client-6.6.1.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.6.1.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.6.1.jar:lib/io.fabric8.openshift-model-operator-6.6.1.jar:lib/io.fabric8.openshift-model-operatorhub-6.6.1.jar:lib/io.fabric8.openshift-model-machine-6.6.1.jar:lib/io.fabric8.openshift-model-whereabouts-6.6.1.jar:lib/io.fabric8.openshift-model-monitoring-6.6.1.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.6.1.jar:lib/io.fabric8.openshift-model-tuned-6.6.1.jar:lib/io.fabric8.openshift-model-console-6.6.1.jar:lib/io.fabric8.openshift-model-config-6.6.1.jar:lib/io.fabric8.openshift-model-machineconfig-6.6.1.jar:lib/io.fabric8.openshift-model-miscellaneous-6.6.1.jar:lib/io.fabric8.openshift-model-hive-6.6.1.jar:lib/io.fabric8.openshift-model-installer-6.6.1.jar:lib/io.fabric8.kubernetes-client-6.6.1.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:lib/io.fabric8.kubernetes-client-api-6.6.1.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.6.1.jar:lib/io.fabric8.kubernetes-model-resource-6.6.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.6.1.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.6.1.jar:lib/io.fabric8.kubernetes-model-batch-6.6.1.jar:lib/io.fabric8.kubernetes-model-certificates-6.6.1.jar:lib/io.fabric8.kubernetes-model-discovery-6.6.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.6.1.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.6.1.jar:lib/io.fabric8.kubernetes-model-metrics-6.6.1.jar:lib/io.fabric8.kubernetes-model-scheduling-6.6.1.jar:lib/io.fabric8.kubernetes-model-node-6.6.1.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.6.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.2.jar:lib/io.fabric8.kubernetes-model-core-6.6.1.jar:lib/io.fabric8.kubernetes-model-events-6.6.1.jar:lib/io.fabric8.kubernetes-model-policy-6.6.1.jar:lib/io.fabric8.kubernetes-model-rbac-6.6.1.jar:lib/io.fabric8.kubernetes-model-apps-6.6.1.jar:lib/io.fabric8.kubernetes-model-storageclass-6.6.1.jar:lib/io.fabric8.kubernetes-model-networking-6.6.1.jar:lib/io.fabric8.kubernetes-model-coordination-6.6.1.jar:lib/io.fabric8.openshift-model-6.6.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.2.jar:lib/org.yaml.snakeyaml-2.0.jar:lib/io.vertx.vertx-core-4.3.8.jar:lib/io.netty.netty-common-4.1.87.Final.jar:lib/io.netty.netty-buffer-4.1.87.Final.jar:lib/io.netty.netty-handler-4.1.87.Final.jar:lib/io.netty.netty-handler-proxy-4.1.87.Final.jar:lib/io.netty.netty-codec-http-4.1.87.Final.jar:lib/io.netty.netty-codec-http2-4.1.87.Final.jar:lib/io.netty.netty-resolver-4.1.87.Final.jar:lib/io.netty.netty-resolver-dns-4.1.87.Final.jar:lib/io.netty.netty-codec-dns-4.1.87.Final.jar:lib/io.strimzi.certificate-manager-0.35.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.12.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.12.0.jar:lib/io.strimzi.kafka-oauth-client-0.12.0.jar:lib/io.strimzi.kafka-oauth-common-0.12.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-4.1.87.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.87.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.87.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.8.jar:lib/io.netty.netty-codec-4.1.87.Final.jar:lib/io.netty.netty-codec-socks-4.1.87.Final.jar io.strimzi.operator.cluster.Main
2023-05-23 07:42:08 INFO  Main:78 - ClusterOperator 0.35.0 is starting
2023-05-23 07:42:08 INFO  Util:319 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.35.0
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	STRIMZI_LEADER_ELECTION_IDENTITY: strimzi-cluster-operator-7d679699bc-gtltz
	KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	STRIMZI_LEADER_ELECTION_ENABLED: true
	STRIMZI_VERSION: 0.35.0
	STRIMZI_NAMESPACE: *
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.35.0
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.35.0
	STRIMZI_FEATURE_GATES:
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
	STRIMZI_LEADER_ELECTION_LEASE_NAME: strimzi-cluster-operator
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	SHLVL: 0
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.25.0
	KUBERNETES_PORT: tcp://10.96.0.1:443
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.3.1=quay.io/strimzi/kafka:0.35.0-kafka-3.3.1
3.3.2=quay.io/strimzi/kafka:0.35.0-kafka-3.3.2
3.4.0=quay.io/strimzi/kafka:0.35.0-kafka-3.4.0

	JAVA_HOME: /usr/lib/jvm/jre-17
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KUBERNETES_SERVICE_HOST: 10.96.0.1
	JAVA_CLASSPATH: :lib/io.strimzi.cluster-operator-0.35.0.jar:lib/io.strimzi.api-0.35.0.jar:lib/io.fabric8.kubernetes-model-common-6.6.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.6.1.jar:lib/io.strimzi.config-model-0.35.0.jar:lib/io.strimzi.operator-common-0.35.0.jar:lib/io.strimzi.crd-annotations-0.35.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.openshift-client-6.6.1.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.6.1.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.6.1.jar:lib/io.fabric8.openshift-model-operator-6.6.1.jar:lib/io.fabric8.openshift-model-operatorhub-6.6.1.jar:lib/io.fabric8.openshift-model-machine-6.6.1.jar:lib/io.fabric8.openshift-model-whereabouts-6.6.1.jar:lib/io.fabric8.openshift-model-monitoring-6.6.1.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.6.1.jar:lib/io.fabric8.openshift-model-tuned-6.6.1.jar:lib/io.fabric8.openshift-model-console-6.6.1.jar:lib/io.fabric8.openshift-model-config-6.6.1.jar:lib/io.fabric8.openshift-model-machineconfig-6.6.1.jar:lib/io.fabric8.openshift-model-miscellaneous-6.6.1.jar:lib/io.fabric8.openshift-model-hive-6.6.1.jar:lib/io.fabric8.openshift-model-installer-6.6.1.jar:lib/io.fabric8.kubernetes-client-6.6.1.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:lib/io.fabric8.kubernetes-client-api-6.6.1.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.6.1.jar:lib/io.fabric8.kubernetes-model-resource-6.6.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.6.1.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.6.1.jar:lib/io.fabric8.kubernetes-model-batch-6.6.1.jar:lib/io.fabric8.kubernetes-model-certificates-6.6.1.jar:lib/io.fabric8.kubernetes-model-discovery-6.6.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.6.1.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.6.1.jar:lib/io.fabric8.kubernetes-model-metrics-6.6.1.jar:lib/io.fabric8.kubernetes-model-scheduling-6.6.1.jar:lib/io.fabric8.kubernetes-model-node-6.6.1.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.6.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.2.jar:lib/io.fabric8.kubernetes-model-core-6.6.1.jar:lib/io.fabric8.kubernetes-model-events-6.6.1.jar:lib/io.fabric8.kubernetes-model-policy-6.6.1.jar:lib/io.fabric8.kubernetes-model-rbac-6.6.1.jar:lib/io.fabric8.kubernetes-model-apps-6.6.1.jar:lib/io.fabric8.kubernetes-model-storageclass-6.6.1.jar:lib/io.fabric8.kubernetes-model-networking-6.6.1.jar:lib/io.fabric8.kubernetes-model-coordination-6.6.1.jar:lib/io.fabric8.openshift-model-6.6.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.2.jar:lib/org.yaml.snakeyaml-2.0.jar:lib/io.vertx.vertx-core-4.3.8.jar:lib/io.netty.netty-common-4.1.87.Final.jar:lib/io.netty.netty-buffer-4.1.87.Final.jar:lib/io.netty.netty-handler-4.1.87.Final.jar:lib/io.netty.netty-handler-proxy-4.1.87.Final.jar:lib/io.netty.netty-codec-http-4.1.87.Final.jar:lib/io.netty.netty-codec-http2-4.1.87.Final.jar:lib/io.netty.netty-resolver-4.1.87.Final.jar:lib/io.netty.netty-resolver-dns-4.1.87.Final.jar:lib/io.netty.netty-codec-dns-4.1.87.Final.jar:lib/io.strimzi.certificate-manager-0.35.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.12.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.12.0.jar:lib/io.strimzi.kafka-oauth-client-0.12.0.jar:lib/io.strimzi.kafka-oauth-common-0.12.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-4.1.87.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.87.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.87.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.8.jar:lib/io.netty.netty-codec-4.1.87.Final.jar:lib/io.netty.netty-codec-socks-4.1.87.Final.jar
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0
	TINI_VERSION: v0.19.0
	STRIMZI_OPERATION_TIMEOUT_MS: 300000
	KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
	STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE: kafka
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.3.1=quay.io/strimzi/kafka:0.35.0-kafka-3.3.1
3.3.2=quay.io/strimzi/kafka:0.35.0-kafka-3.3.2
3.4.0=quay.io/strimzi/kafka:0.35.0-kafka-3.4.0

	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.35.0
	KUBERNETES_SERVICE_PORT: 443
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.35.0
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	HOSTNAME: strimzi-cluster-operator-7d679699bc-gtltz
	STRIMZI_KAFKA_CONNECT_IMAGES: 3.3.1=quay.io/strimzi/kafka:0.35.0-kafka-3.3.1
3.3.2=quay.io/strimzi/kafka:0.35.0-kafka-3.3.2
3.4.0=quay.io/strimzi/kafka:0.35.0-kafka-3.4.0

	STRIMZI_KAFKA_IMAGES: 3.3.1=quay.io/strimzi/kafka:0.35.0-kafka-3.3.1
3.3.2=quay.io/strimzi/kafka:0.35.0-kafka-3.3.2
3.4.0=quay.io/strimzi/kafka:0.35.0-kafka-3.4.0

	KUBERNETES_PORT_443_TCP_PORT: 443
	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0
	STRIMZI_OPERATOR_NAME: strimzi-cluster-operator-7d679699bc-gtltz
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	STRIMZI_OPERATOR_NAMESPACE: kafka

2023-05-23 07:42:08 INFO  Main:81 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[*],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{3.3.1={proto: 3.3 msg: 3.3 kafka-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.1 connect-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.1 mirrormaker-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.1 mirrormaker2-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.1}, 3.3.2={proto: 3.3 msg: 3.3 kafka-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.2 connect-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.2 mirrormaker-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.2 mirrormaker2-image: quay.io/strimzi/kafka:0.35.0-kafka-3.3.2}, 3.4.0={proto: 3.4 msg: 3.4 kafka-image: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0 connect-image: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0 mirrormaker-image: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0 mirrormaker2-image: quay.io/strimzi/kafka:0.35.0-kafka-3.4.0}},imagePullPolicy=null,imagePullSecrets=null,operatorNamespace=kafka,operatorNamespaceLabels=null,customResourceSelector=null,featureGates=FeatureGates(UseKRaft=false,StableConnectIdentities=false),zkAdminSessionTimeoutMs=10000,dnsCacheTtlSec=30,podSetReconciliationOnly=false,podSetControllerWorkQueueSize=1024,operatorName=strimzi-cluster-operator-7d679699bc-gtltz,podSecurityProviderClass=io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider,leaderElectionConfig=LeaderElectionConfig{leaseName='strimzi-cluster-operator', namespace='kafka', identity='strimzi-cluster-operator-7d679699bc-gtltz', leaseDuration=PT15S, renewDeadline=PT10S, retryPeriod=PT2S})
2023-05-23 07:42:09 INFO  Main:340 - Health and metrics server is ready on port 8080)
2023-05-23 07:42:09 INFO  Main:249 - Waiting to become a leader
2023-05-23 07:42:10 INFO  LeaderElectionManager:119 - The new leader is strimzi-cluster-operator-7d679699bc-gtltz
2023-05-23 07:42:10 INFO  LeaderElectionManager:100 - Started being a leader
2023-05-23 07:42:10 INFO  Main:237 - I'm the new leader
2023-05-23 07:42:10 WARN  PlatformFeaturesAvailability:162 - API Group route.openshift.io is not supported
2023-05-23 07:42:10 WARN  PlatformFeaturesAvailability:162 - API Group build.openshift.io is not supported
2023-05-23 07:42:10 WARN  PlatformFeaturesAvailability:162 - API Group image.openshift.io is not supported
2023-05-23 07:42:10 INFO  Main:126 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.25,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2023-05-23 07:42:10 INFO  PodSecurityProviderFactory:43 - Found PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2023-05-23 07:42:10 INFO  PodSecurityProviderFactory:62 - Initializing PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2023-05-23 07:42:10 INFO  ClusterOperator:94 - Creating ClusterOperator for namespace *
2023-05-23 07:42:10 INFO  ClusterOperator:109 - Starting ClusterOperator for namespace *
2023-05-23 07:42:10 INFO  StrimziPodSetController:518 - Starting the StrimziPodSet controller
2023-05-23 07:42:10 INFO  StrimziPodSetController:487 - Starting StrimziPodSet controller for namespace *
2023-05-23 07:42:10 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2023-05-23 07:42:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2023-05-23 07:42:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2023-05-23 07:42:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2023-05-23 07:42:10 INFO  StrimziPodSetController:490 - Waiting for informers to sync
2023-05-23 07:42:10 INFO  ClusterOperator:124 - Opened watch for Kafka operator
2023-05-23 07:42:10 INFO  ClusterOperator:124 - Opened watch for KafkaMirrorMaker operator
2023-05-23 07:42:10 INFO  ClusterOperator:124 - Opened watch for KafkaConnect operator
2023-05-23 07:42:10 INFO  ClusterOperator:124 - Opened watch for KafkaBridge operator
2023-05-23 07:42:10 INFO  ClusterOperator:124 - Opened watch for KafkaMirrorMaker2 operator
2023-05-23 07:42:10 INFO  ClusterOperator:136 - Setting up periodic reconciliation for namespace *
2023-05-23 07:42:10 INFO  Main:207 - Cluster Operator verticle started in namespace * without label selector
2023-05-23 07:42:11 INFO  StrimziPodSetController:495 - Informers are in-sync
2023-05-23 07:44:10 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace *
2023-05-23 07:44:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2023-05-23 07:44:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2023-05-23 07:44:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
2023-05-23 07:44:10 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'
2023-05-23 07:46:10 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace *
$

[tankim]

Thank you so much for sharing the details.
I'll dig a little deeper and share what's going on.

[scholzj]

It could be a different version of something ... I don't know. But maybe if you can try to repeat the steps from above, we can see if with your versions it works or not and get closer to it.

[scholzj]

Triaged on 1.6.2023: Seems more like an individual issue rather than a bug in the Helm Chart. Should be converted to a discussion for the time being.