Kafka MirrorMaker2 TLS issues

[edenkoveshi]

Hi,
I am trying to deploy Kafka Mirror Maker 2 using the Strimzi Kafka Operator, to mirror two Kafka clusters, currently in the same namespace, both using TLS and SCRAM-SHA-512 authentication.
I am using this guide https://dalelane.co.uk/blog/?p=4303 to set this up.
In the Mirror Maker 2 pod, I get the following error:

Preparing MirrorMaker 2.0 cluster truststores and keystores
Preparing MirrorMaker 2.0 truststores and keystores for cluster source-cluster
with trusted certs my-cluster-source-cluster-ca-cert/ca.crt
with tls auth certs
with tls auth keys
Perparing truststore
keytool error: java.io.FileNotFoundException: /opt/kafka/mm2-certs/source-cluster/my-cluster-source-cluster-ca-cert/ca.crt

Looking at the pod details, I see the ca cert secret was created on a volume mounted on /opt/kafka/mm2-certs/my-cluster-source-cluster-ca-cert/ca.crt (no source-cluster alias in the path)

Any idea why this could happen?

[scholzj]

You will probably need to share the details ... the YAMLs of the custom resources, etc.

[edenkoveshi]

Mirror maker yaml:

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: data-migration
spec:
  replicas: 1
  clusters:
  - alias: source-cluster
    bootstrapServers: "source-cluster-kafka-bootstrap:9093"
    authentication:
      passwordSecret:
        password: password
        secretName: source-cluster-user
      type: scram-sha-512
      username: source-cluster-user
    tls:
      trustedCertificates:
        - certificate: ca.crt
          secretName: source-cluster-ca-cert
  - alias: "target-cluster"
    bootstrapServers: "target-cluster-kafka-bootstrap:9093"
    authentication:
      passwordSecret:
        password: password
        secretName: target-cluster-user
      type: scram-sha-512
      username: target-cluster-user
    tls:
      trustedCertificates:
        - certificate: ca.crt
          secretName: target-cluster-ca-cert
    config:
      offset.storage.topic: migration-connect-cluster-offsets
      config.storage.topic: migration-connect-cluster-configs
      status.storage.topic: migration-connect-cluster-status
  connectCluster: "target-cluster"
  mirrors:
    - sourceCluster: "source-cluster"
      targetCluster: "target-cluster"
      sourceConnector:
        config:
          replication.factor: 1
          sync.topic.acls.enabled: "false"
          offset-syncs.topic.replication.factor: 1
      checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 1
          refresh.groups.interval.seconds: 600
          emit.checkpoints.enabled: true
          sync.group.offsets.enabled: true
          sync.group.offsets.interval.seconds: 60
          emit.checkpoints.interval.seconds: 60
      topicsPattern: ".*"
      groupsPattern: ".*"

Kafka yaml:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: source-cluster
spec:
    kafka:
      version: 2.4.0
      replicas: 3
      listeners:
        plain:
          authentication:
             type: scram-sha-512
        tls:
          authentication:
             type: scram-sha-512
       external:
          type: route
          authentication:
             type: scram-sha-512
      ...

Kafka user yaml:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: source-cluster-user
spec:
  authentication:
    type: scram-sha-512

Kafka and KafkaUser objects of the target cluster are identical (except for the name of course)

[scholzj]

Sharing the full resources would make it easier to reproduce this. Also, what Strimzi version are you using? Kafka 2.4.0 seems rather old.

It looks to me like the path changed in #3082 (Strimzi 0.19). So you should double check you use the same version of the container image for your MM2 deployment and for your operator. The error would suggest you have a newer image to run Mirror Maker 2 and older Operator version.