Operation process when updating Kafka Connect

[cstmgl]

Hi all,

I have a question just to clarify my understanding and maybe some pointers to the documentation that I may have overlooked.

Basically I'm updating my Kafka Connector to fix a bug, I'm using Strimzi over the Redhat distribution, and I can see that the Strimzi operator registers the change in the internal topics for the configuration.
But I was however expecting Strimzi to stop the old pods with the workers and start the new pods with the new workers and this is not working.
I was almost sure this worked in the past, but maybe I just got confused with something.
Is the normal process to do this manually and not rely on the Strimzi Operator?

UPDATE:
I'll try to explain my setup and what I'm trying to.

I have a docker image with my Kafka Connect and all their content. Let's say I am in version 1.0.0
I work on providing my CRDs with the KafkaConnect and KafkaConnectors with the configurations and all those things. Still part of my version 1.0.0.

Let's say I find a bug and I need to release version 1.1.0 and this will have a new docker image.
I can see that the strimzi updates and manages my configurations changes in the internal topics.
But at the pod level my pods with the workers for the version 1.0.0 remain running and the ones for the version 1.1.0 do not start.

I always have to preform a "oc delete kafkaconnect --all" and "oc delete kafkaconnector --all" to have Strimzi launch the new pods.
I'm wondering if I miss configured something or if this is expected. I was expecting Strimzi to stop the old pods and start the new ones.
Or at least that I only needed to scale down the KafkaConnect and then Strimzi would scale up the new pods?

Thank you for any feedback

I see an error in the operator lock like:

Failed to acquire lock lock::insight-data-flow-nightly::KafkaConnect::data-flow-kc within 10000ms.

[scholzj]

Can you maybe describe what exactly are you doing? I'm afraid it is not completely clear from the description.

[cstmgl]

I'm still not completely sure what exactly are you doing. But when you use a custom image with your connectors for Kafka Connect, you probably specify it in the KafkaConnect custom resource in the .spec.image field. When you change the tag of your image in this field, the operator should automatically update it in Kubernetes and Kubernetes should automatically roll all pods of the Connect deployment.

First off all thank you for the continuous feedback it's really appreciated I know you must receive a lot of this each day.

So long story short:
I update my KafkaConnect & KafkaConnector YML
In Openshift I see that they both get updated.
I expect that KafkaConnect will be picked up by the Strimzi Operator and transformed into a Deployment.
By some reason this Deployment does not get updated once I update KafkaConnect

I believe this is something that is wrong in the operator but might be related to the log I mentioned above but I would not know why that would be.

Failed to acquire lock lock::insight-data-flow-nightly::KafkaConnect::data-flow-kc within 10000ms.

[scholzj]

Sorry, but this is not specific enough. What do you update? Please share the YAMLs. The lock message on its own doesn't mean anything (https://strimzi.io/docs/operators/latest/full/using.html#what_do_the_failed_to_acquire_lock_warnings_in_the_log_mean), you have the share the full log, ideally on DEBUG level, but even with the default settings might show what is happening.

[cstmgl]

Sorry, but this is not specific enough. What do you update? Please share the YAMLs. The lock message on its own doesn't mean anything (https://strimzi.io/docs/operators/latest/full/using.html#what_do_the_failed_to_acquire_lock_warnings_in_the_log_mean), you have the share the full log, ideally on DEBUG level, but even with the default settings might show what is happening.

:)

so this is my yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: data-flow-kc
  labels:
    app.kubernetes.io/component: messaging
    strimzi.io/cluster: strimzi-kafka
    app.kubernetes.io/part-of: data-flow
    app.kubernetes.io/instance: data-flow-connector-base
    app.kubernetes.io/name: data-flow-connector-base
    app.kubernetes.io/version: 1.8.2-dev.1-25a6f44
    app: data-flow
  annotations:
    strimzi.io/use-connector-resources: 'true'
spec:
  image: data-flow-connector-base:1.8.2-dev.1-25a6f44
  replicas: 3
  template:
    pod:
      metadata:
        labels:
          app.kubernetes.io/component: messaging
        annotations:
          prometheus.io/scrape: 'true'
          prometheus.io/scheme: http
          prometheus.io/port: '9404'
    deployment:
      metadata:
        labels:
          app.kubernetes.io/part-of: data-flow
          app.kubernetes.io/instance: data-flow-connector-base
          app.kubernetes.io/name: data-flow-connector-base
          app.kubernetes.io/version: 1.8.2-dev.1-25a6f44
          app: data-flow
  authentication:
    type: tls
    certificateAndKey:
      certificate: user.crt
      key: user.key
      secretName: data-flow-connector-user
  bootstrapServers: strimzi-kafka-kafka-bootstrap:9093
  tls:
    trustedCertificates:
    - secretName: strimzi-kafka-cluster-ca-cert
      certificate: ca.crt
  config:
    group.id: com.dataflow.kafkaconnect
    offset.storage.topic: com.dataflow.kafkaconnect.offset
    config.storage.topic: com.dataflow.kafkaconnect.configs
    status.storage.topic: com.dataflow.kafkaconnect.status
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    offset.flush.interval.ms: 60000
    offset.flush.timeout.ms: 25000
    connector.client.config.override.policy: All
  externalConfiguration:
    volumes:
    - name: connection-details
      secret:
        secretName: data-flow-connector-base-connection-details
        defaultMode: 256
    - name: application-keystore
      secret:
        secretName: data-flow-connector-base-appl-keystore
        defaultMode: 256
  resources:
    requests:
      cpu: 200m
      memory: 4Gi
    limits:
      cpu: 2
      memory: 8Gi
  logging:
    type: external
    name: data-flow-connector-base-logging
  readinessProbe:
    initialDelaySeconds: 120
    timeoutSeconds: 60
  livenessProbe:
    initialDelaySeconds: 120
    timeoutSeconds: 60
  jvmOptions:
    gcLoggingEnabled: false
  metrics:
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
    - pattern: kafka.(.+)<type=app-info, client-id=(.+)><>start-time-ms
      name: kafka_$1_start_time_seconds
      labels:
        clientId: $2
      help: Kafka $1 JMX metric start time seconds
      type: GAUGE
      valueFactor: 0.001
    - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>(commit-id|version): (.+)'
      name: kafka_$1_$3_info
      value: 1
      labels:
        clientId: $2
        $3: $4
      help: Kafka $1 JMX metric info version and commit-id
      type: GAUGE
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)><>(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead)
      name: kafka_$2_$6
      labels:
        clientId: $3
        topic: $4
        partition: $5
      help: Kafka $1 JMX metric type $2
      type: GAUGE
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+)><>(.+-total|compression-rate|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: $3
        topic: $4
      help: Kafka $1 JMX metric type $2
      type: GAUGE
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: $3
        nodeId: $4
      help: Kafka $1 JMX metric type $2
      type: UNTYPED
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
      name: kafka_$2_$4
      labels:
        clientId: $3
      help: Kafka $1 JMX metric type $2
      type: GAUGE
    - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status:
        ([a-z-]+)'
      name: kafka_connect_connector_status
      value: 1
      labels:
        connector: $1
        task: $2
        status: $3
      help: Kafka Connect JMX Connector status
      type: GAUGE
    - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
      name: kafka_connect_$1_$4
      labels:
        connector: $2
        task: $3
      help: Kafka Connect JMX metric type $1
      type: GAUGE
    - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
      name: kafka_connect_worker_$2
      labels:
        connector: $1
      help: Kafka Connect JMX metric $1
      type: GAUGE
    - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
      name: kafka_connect_worker_$1
      help: Kafka Connect JMX metric worker
      type: GAUGE
    - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
      name: kafka_connect_worker_rebalance_$1
      help: Kafka Connect JMX metric rebalance information
      type: GAUGE

let's say for example I'm updating the base image from 1.7.3 to this new one

spec:
  image: data-flow-connector-base:1.8.2-dev.1-25a6f44

What I notice is that the Deployment generated out of the Operator never changes so Openshift does not know it needs to update the pods.
Like I said I really don't find anything in the operator log that would indicate me what could be going wrong. Maybe I'm looking in the wrong place, should I be looking in the entity operator or in the cluster operator?

[scholzj]

The cluster operator handles this. So you should check its logs and share them if needed.

I gave it a try and changing the image works fine for me (Strimzi 0.25, but I'mm not aware of any recent changes / fixes around this) => the operator updates the deployment and Kubernetes roll the Connect pods when I change the image.

[cstmgl]

I could not find this issue anymore so I don't know the root cause but indeed it seems the operator does what it's supposed to be doing