Strimzi MM2 doesn't start the Source Connector tasks.

[AAR256]

I'm using the below MM2 configuration to replicate messages from Source and Destination Kafka clusters.

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mm2-cluster
  namespace: kafka
spec:
  version: 3.0.0
  replicas: 1
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "9404"
  connectCluster: dev-strimzi-kafka #"my-cluster-target"
  clusters:
    - alias: dev-ccloud-kafka #"my-cluster-source"
      authentication:
        type: plain
        username: XXXXXXXXXX
        passwordSecret:
          secretName: ccloud-admin
          password: password
      bootstrapServers: XXXXXXXX.azure.confluent.cloud:XXXX
    - alias: dev-strimzi-kafka #"my-cluster-target"
      authentication:
        type: scram-sha-512
        username: strimzi-admin
        passwordSecret:
          secretName: strimzi-admin
          password: password
      bootstrapServers: dev-strimzi-kafka-bootstrap.XXXX.net:XXX
      tls:
        trustedCertificates:
          - secretName: dev-strimzi-kafka-cluster-ca-cert
            certificate: ca.crt
      config:
        # -1 means it will use the default replication factor configured in the broker
        config.storage.replication.factor: 3
        offset.storage.replication.factor: 3
        status.storage.replication.factor: 3
  mirrors:
    - sourceCluster: dev-ccloud-kafka #"my-cluster-source"
      targetCluster: dev-strimzi-kafka #"my-cluster-target"
      sourceConnector:
        config:
          replication.factor: 3
          offset-syncs.topic.replication.factor: 3
          tasks.max: 6
      heartbeatConnector:
        config:
          heartbeats.topic.replication.factor: 3
      checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 3
          sync.group.offsets.enabled: true
      topicsPattern: "topic1"
      groupsPattern: ".*"

  resources:
    requests:
      memory: 4Gi
      cpu: "1"
    limits:
      memory: 8Gi
      cpu: "2"

After deployment of MM2, all the pods are coming up correctly along with the config topics and whitelist topics but replication isn't happening for those whitelist topics.

From the MM2 resource description, I'm seeing no tasks running for MirrorSourceConnector. -

Status:
  Conditions:
    Last Transition Time:  2023-04-18T10:00:29.043063Z
    Status:                True
    Type:                  Ready
  Connectors:
    Connector:
      State:      RUNNING
      worker_id:  10.200.0.81:8083
    Name:         dev-ccloud-kafka->dev-strimzi-kafka.MirrorCheckpointConnector
    Tasks:
      Id:         0
      State:      RUNNING
      worker_id:  10.200.0.81:8083
    Type:         source
    Connector:
      State:      RUNNING
      worker_id:  10.200.1.206:8083
    Name:         dev-ccloud-kafka->dev-strimzi-kafka.MirrorHeartbeatConnector
    Tasks:
      Id:         0
      State:      RUNNING
      worker_id:  10.200.1.206:8083
    Type:         source
    Connector:
      State:      RUNNING
      worker_id:  10.200.4.198:8083
    Name:         dev-ccloud-kafka->dev-strimzi-kafka.MirrorSourceConnector
    Tasks:
    Type:               source

From the MM2 pods logs also, no errors were observed -

ssl.truststore.password = [hidden]
	ssl.truststore.type = PKCS12
 (org.apache.kafka.clients.admin.AdminClientConfig) [Scheduler for MirrorHeartbeatConnector-creating internal topics]
2023-04-18 09:59:28,171 WARN The configuration 'metrics.context.connect.group.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [StartAndStopExecutor-connect-1-2]
2023-04-18 09:59:28,171 WARN The configuration 'metrics.context.connect.kafka.cluster.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [StartAndStopExecutor-connect-1-2]
2023-04-18 09:59:28,171 INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser) [StartAndStopExecutor-connect-1-2]
2023-04-18 09:59:28,171 INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser) [StartAndStopExecutor-connect-1-2]
2023-04-18 09:59:28,171 INFO Kafka startTimeMs: 1681811968171 (org.apache.kafka.common.utils.AppInfoParser) [StartAndStopExecutor-connect-1-2]
2023-04-18 09:59:28,173 INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser) [Scheduler for MirrorHeartbeatConnector-creating internal topics]
2023-04-18 09:59:28,174 INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser) [Scheduler for MirrorHeartbeatConnector-creating internal topics]
2023-04-18 09:59:28,174 INFO Kafka startTimeMs: 1681811968173 (org.apache.kafka.common.utils.AppInfoParser) [Scheduler for MirrorHeartbeatConnector-creating internal topics]
2023-04-18 09:59:28,180 INFO [Worker clientId=connect-1, groupId=ccloud-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]

Any help in identifying the root cause for this issue would be great.

[scholzj]

Sharing the full custom resource with the status in YAML might be useful as it is more readable then the describe output you shared above. Also the full MM2 log would be useful I guess.

You do not have any TLS configuration for the Confluent cluster. Are you sure it is not using any TLS encryption?

[AAR256]

MM2-Pod-Logs.txt

Full CR Status -

Name:         dev-ccloud-kafka
Namespace:    emp-kafka
Labels:       <none>
Annotations:  <none>
API Version:  kafka.strimzi.io/v1beta2
Kind:         KafkaMirrorMaker2
Metadata:
  Creation Timestamp:  2023-04-18T09:20:41Z
  Generation:          1
  Managed Fields:
    API Version:  kafka.strimzi.io/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:clusters:
        f:connectCluster:
        f:metricsConfig:
          .:
          f:type:
          f:valueFrom:
            .:
            f:configMapKeyRef:
              .:
              f:key:
              f:name:
        f:mirrors:
        f:replicas:
        f:resources:
          .:
          f:limits:
            .:
            f:cpu:
            f:memory:
          f:requests:
            .:
            f:cpu:
            f:memory:
        f:template:
          .:
          f:pod:
            .:
            f:affinity:
              .:
              f:nodeAffinity:
                .:
                f:requiredDuringSchedulingIgnoredDuringExecution:
                  .:
                  f:nodeSelectorTerms:
            f:metadata:
              .:
              f:annotations:
                .:
                f:prometheus.io/port:
                f:prometheus.io/scrape:
            f:tolerations:
        f:version:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2023-04-18T09:20:41Z
    API Version:  kafka.strimzi.io/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:connectors:
        f:labelSelector:
        f:observedGeneration:
        f:replicas:
        f:url:
    Manager:         okhttp
    Operation:       Update
    Subresource:     status
    Time:            2023-04-18T09:21:57Z
  Resource Version:  416581628
  UID:               a0b536ab-9ac5-4d20-839a-1aca6ce02623
Spec:
  Clusters:
    Alias:  dev-ccloud-kafka-kafka
    Authentication:
      Password Secret:
        Password:       password
        Secret Name:    dev-ccloud-kafka-admin
      Type:             plain
      Username:         ABCD
    Bootstrap Servers:  XXX-XXXXX.westeurope.azure.confluent.cloud:9092
    Tls:
      Trusted Certificates:
        Certificate:  ca.crt
        Secret Name:  dev-ccloud-kafka-cert
    Alias:            dev-strimzi-kafka
    Authentication:
      Password Secret:
        Password:       password
        Secret Name:    pp-shared-emp-admin
      Type:             scram-sha-512
      Username:         pp-shared-emp-admin
    Bootstrap Servers:  dev-strimzi-kafka-bootstrap.net:443
    Config:
      config.storage.replication.factor:  3
      config.storage.topic:               ccloud-cluster-configs
      group.id:                           ccloud-cluster
      offset.storage.replication.factor:  3
      offset.storage.topic:               ccloud-cluster-offsets
      status.storage.replication.factor:  3
      status.storage.topic:               ccloud-cluster-status
    Tls:
      Trusted Certificates:
        Certificate:  tls.crt
        Secret Name:  dev-strimzi-kafka-listener-cert
  Connect Cluster:    dev-strimzi-kafka
  Metrics Config:
    Type:  jmxPrometheusExporter
    Value From:
      Config Map Key Ref:
        Key:   metrics-config.yml
        Name:  mirror-maker-2-metrics
  Mirrors:
    Checkpoint Connector:
      Config:
        checkpoints.topic.replication.factor:  3
        sync.group.offsets.enabled:            true
    Groups Pattern:                            .*
    Heartbeat Connector:
      Config:
        heartbeats.topic.replication.factor:  3
    Source Cluster:                           dev-ccloud-kafka-kafka
    Source Connector:
      Config:
        offset-syncs.topic.replication.factor:  3
        replication.factor:                     3
        replication.policy.class:               io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy
        replication.policy.separator:           
        tasks.max:                              6
    Target Cluster:                             dev-strimzi-kafka
    Topics Pattern:                             Topic1
  Replicas:                                     3
  Resources:
    Limits:
      Cpu:     2
      Memory:  4Gi
    Requests:
      Cpu:     1
      Memory:  2Gi
  Template:
    Pod:
      Affinity:
        Node Affinity:
          Required During Scheduling Ignored During Execution:
            Node Selector Terms:
              Match Expressions:
                Key:       team
                Operator:  In
                Values:
                  ABC
      Metadata:
        Annotations:
          prometheus.io/port:    9404
          prometheus.io/scrape:  true
      Tolerations:
        Effect:    NoSchedule
        Key:       kubernetes.azure.com/scalesetpriority
        Operator:  Equal
        Value:     spot
        Effect:    NoSchedule
        Key:       CriticalAddonsOnly
        Operator:  Equal
        Value:     true
  Version:         3.0.0
Status:
  Conditions:
    Last Transition Time:  2023-04-18T10:00:29.043063Z
    Status:                True
    Type:                  Ready
  Connectors:
    Connector:
      State:      RUNNING
      worker_id:  10.200.0.81:8083
    Name:         dev-ccloud-kafka-kafka->dev-strimzi-kafka.MirrorCheckpointConnector
    Tasks:
      Id:         0
      State:      RUNNING
      worker_id:  10.200.0.81:8083
    Type:         source
    Connector:
      State:      RUNNING
      worker_id:  10.200.1.206:8083
    Name:         dev-ccloud-kafka-kafka->dev-strimzi-kafka.MirrorHeartbeatConnector
    Tasks:
      Id:         0
      State:      RUNNING
      worker_id:  10.200.1.206:8083
    Type:         source
    Connector:
      State:      RUNNING
      worker_id:  10.200.4.198:8083
    Name:         dev-ccloud-kafka-kafka->dev-strimzi-kafka.MirrorSourceConnector
    Tasks:
    Type:               source
  Label Selector:       strimzi.io/cluster=dev-ccloud-kafka,strimzi.io/name=dev-ccloud-kafka-mirrormaker2,strimzi.io/kind=KafkaMirrorMaker2
  Observed Generation:  1
  Replicas:             3
  URL:                  http://dev-ccloud-kafka-mirrormaker2-api.emp-kafka.svc:8083
Events:                 <none>

I've attached the full MM2 logs as a file attachment. Also, I've enabled TLS for confluent cluster using let's encrypt certificate. Also, this is an intermittent issue. 40-50% of the time it might work as in it actually create the start and start the Source connector. The rest of the time it doesn't work using the same configuration. Not sure why it's behaving like this.

[scholzj]

You should get the custom resource in YAML => that has a clear and readable structure => kubectl get kafkaconnect <name> -o yaml. If you Confluent cluster uses TLS, then your previous configuration was definitely invalid and could not have worked.