Issue with using FluxCD Gitops to manage Strimzi Kafka resource manifests

[misurin]

Hi,

We are a mid-size data engineering team working in a Singapore E-commerce company. Kubernetes has been used widely across many of our services and we use FluxCD to manage the K8S deployments.

We onboarded Kafka to our tech-stack a few months ago, We thought we could give Strimzi Operator a try and the actual installation process was very smooth, we're able to set up all features that we need. Until recently when we start migrating more use-cases to Kafka, more Kafka connectors need to set up, we realized that every time we make a change to the Kafka manifest Git repo, the whole cluster restarts even though the change is only a Kafka Connector config.

The restarting process starts with Rolling Restart of Zookeeper cluster, then Kafka Cluster, and then all Kafka Connect clusters.

I'm pretty sure that we made no changes to the main Kafka cluster manifest. When trying to compare the created Kafka cluster resource by FluxCD, we found that the only change is in the label kustomize.toolkit.fluxcd.io/checksum.

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  annotations:
  creationTimestamp: "2021-01-15T04:46:53Z"
  generation: 2
  labels:
    **kustomize.toolkit.fluxcd.io/checksum: f9380059f5f92fd6ccf96192922ddd8a2ce58c7c**
    kustomize.toolkit.fluxcd.io/name: data-eks-kafka
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: prod
  namespace: kafka
spec:

Is there any configuration in the operator to prevent this behavior? Or under which condition the operator decides to restart the whole cluster?

As a workaround, we have to block all Kafka connector creations using the KafkaConnector resources :(.

[scholzj]

I'm sorry, I'm not sure how does KafkaConnectors resources relate to the Kafka resource. But the labels from the Kafka resource are mirrored to the operand resources. So if the checksum is changing, it is probably triggering the rolling updates. I'm not sure why the checksum is needed and what does it checksum (maybe it checksums the connectors as well?). But you should be able to use the envoronment variable STRIMZI_LABELS_EXCLUSION_PATTERN in the Strimzi cluster operator to configure it to ignore these labels and not mirror them tot he operands (https://strimzi.io/docs/operators/latest/full/using.html#ref-operator-cluster-str). That should make sure that even when they change, they do not trigger the rolling update. IIRC this is supported from Strimzi 0.23 I think?

[misurin]

Thank you very much for the answer. I guess we'll need to upgrade the operator to have this label exclusion feature.

The checksum label is controlled by FluxCD Kustomize controller for its resource management and it can not be removed. Because off Kafka resources are managed and deployed by FluxCD, all will have this checksum label.

[misurin]

FluxCD authors probably realized that the checksum label is causing conflicts, they changed from label to annotation. The commit was just 2 weeks ago: fluxcd/kustomize-controller@6979dba