[Bug]: No tls-sidecar container in entity-operator pod

[maazamalik]

Bug Description

After the strimzi operator upgrade from v0.38.0 to v0.39.0 using helm and applying crds after that, we noticed that the entity operator does not contain tls-sidecar container anymore (just the user-operator and the topic-operator container). No changes were made apart from the version bump. Its not that the container is crash looping or OOM killed. Its just not there. No error logs either in the strimzi operator pod, neither from the entity-operator pod that would point to the problem. We specifically tried to add the tls-sidecar config in kafka crd, but even then the entity-operator deployment does not contain tls-sidecar. Maybe some configuration is overriding it?
kafka version: 3.6.0

Steps to reproduce

upgrade strimzi helm chart from 0.38.0 to 0.39.0

Expected behavior

all pods and containers in running state

Strimzi version

0.39.0

Kubernetes version

1.28

Installation method

Helm chart

Infrastructure

Amazon EKS

Configuration files and logs

kafka yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    meta.helm.sh/release-name: kafka-cluster
    meta.helm.sh/release-namespace: kafka
  creationTimestamp: "2021-12-09T13:14:58Z"
  generation: 39
  labels:
    app.kubernetes.io/managed-by: Helm
  name: sandbox
  namespace: kafka
  resourceVersion: "291374484"
  uid: xxxxx
spec:
  entityOperator:
    tlsSidecar:
      image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
      resources:
        limits:
          memory: 128Mi
        requests:
          cpu: 200m
          memory: 64Mi
    topicOperator: {}
    userOperator: {}
  kafka:
    authorization:
      superUsers:
      - admin
      type: simple
    config:
      auto.create.topics.enable: "false"
      log.retention.hours: 120
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    jvmOptions:
      -XX:
        G1HeapRegionSize: 16M
        InitiatingHeapOccupancyPercent: 35
        MaxGCPauseMillis: 20
        MaxMetaspaceFreeRatio: 80
        MetaspaceSize: 96m
        MinMetaspaceFreeRatio: 50
        UseG1GC: true
      -Xms: 800m
      -Xmx: 800m
    listeners:
    - authentication:
        type: scram-sha-512
      configuration:
        useServiceDnsDomain: true
      name: plain
      port: 9095
      tls: false
      type: internal
    - authentication:
        type: scram-sha-512
      configuration:
        bootstrap:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
        brokers:
        - annotations:
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
          broker: 0
        - annotations:
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
          broker: 1
        - annotations:
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
          broker: 2
        finalizers:
        - service.k8s.aws/resources
      name: external
      port: 9094
      tls: true
      type: loadbalancer
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-metrics
    replicas: 3
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: "0.1"
        memory: 1Gi
    storage:
      type: jbod
      volumes:
      - class: gp3
        deleteClaim: false
        id: 0
        size: 100Gi
        type: persistent-claim
    template:
      pod:
        topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: kafka
              app.kubernetes.io/part-of: strimzi-sandbox
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
    version: 3.6.1
  kafkaExporter:
    enableSaramaLogging: true
    groupRegex: .*
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    logging: info
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      limits:
        memory: 128Mi
      requests:
        cpu: 200m
        memory: 64Mi
    template:
      pod:
        imagePullSecrets:
        - name: my-docker-credentials
        securityContext:
          fsGroup: 0
          runAsUser: 1000001
        terminationGracePeriodSeconds: 120
    topicRegex: .*
  zookeeper:
    jvmOptions:
      -Xms: 800m
      -Xmx: 800m
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: zookeeper-metrics-config.yml
          name: kafka-metrics
    replicas: 3
    resources:
      limits:
        memory: 0.8Gi
      requests:
        cpu: "0.1"
        memory: 0.8Gi
    storage:
      class: gp3
      deleteClaim: false
      size: 100Gi
      type: persistent-claim
    template:
      pod:
        topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: zookeeper
              app.kubernetes.io/part-of: strimzi-sandbox
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule

Additional context

No response

[scholzj]

Strimzi 0.39 uses the new Unidirectional Topic Operator by default. That does not use ZooKeeper anymore and does not need the sidecar. So it is not there anymore. You can find more details about it here: https://strimzi.io/blog/2023/11/02/unidirectional-topic-operator/

[maazamalik]

Thank you very much.

[scholzj]

FYI: I only now realized that the alert is part of our samples. I opened a PR to remove it: #9622