Kafka Mirror Maker 2

[Flooze-IT]

Hello everyone,

I need help with a problem that I can't solve. I'm using Strimzi version 0.34 and facing an issue with Kafka Mirror Maker 2.

Context:
I have a source cluster which has 3 brokers and a target cluster which has 1 broker.

I tried to set up Kafka mirror maker 2 but I get an error (in debug mode): org.apache.kafka.common.errors.CoordinatorNotAvailableException: The coordinator is not available.

The reassuring element and not too much at the same time is that if I scale my target cluster with 3 brokers everything works perfectly.

Did I forget a particular configuration?

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: event-bus-mm2-migrator-xxx
  namespace: xxx
spec:
  clusters:
    - alias: my-source-cluster
      authentication:
        clientId: xxx
        clientSecret:
          key: client-secret
          secretName: client-secret-xx-security
        tokenEndpointUri: >-
          https://blauth.berger-levrault.com/auth/realms/saas/protocol/openid-connect/token
        type: oauth
      bootstrapServers: xxx:443
      config:
        config.storage.replication.factor: -1
        offset.storage.replication.factor: -1
        status.storage.replication.factor: -1
      tls:
        trustedCertificates:
          - certificate: ca.crt
            secretName: certs-cluster-xxx-prod
    - alias: my-target-cluster
      bootstrapServers: xx.xx.svc.cluster.local:9094
      config:
        config.storage.replication.factor: 1
        config.storage.topic: mirrormaker2-cluster-configs
        offset.storage.replication.factor: 1
        offset.storage.topic: mirrormaker2-cluster-offsets
        status.storage.replication.factor: 1
        status.storage.topic: mirrormaker2-cluster-status
  connectCluster: my-target-cluster
  livenessProbe:
    initialDelaySeconds: 45
    timeoutSeconds: 5
  logging:
    loggers:
      connect.root.logger.level: debug
    type: inline
  mirrors:
    - checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 1
          refresh.groups.interval.seconds: 60
          replication.policy.class: io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy
          sync.group.offsets.enabled: true
      groupsPattern: .*
      sourceCluster: my-source-cluster
      sourceConnector:
        config:
          offset-syncs.topic.replication.factor: 1
          refresh.topics.interval.seconds: 60
          replication.factor: 1
          replication.policy.class: io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy
          replication.policy.separator: ''
          sync.topic.acls.enabled: 'false'
        tasksMax: 1
      targetCluster: my-target-cluster
      topicsPattern: .*
  replicas: 1
  version: 3.4.0

2023-06-30 08:01:08,956 DEBUG [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Sending FindCoordinator request to broker kafka-cluster-gmao-kafka-0.kafka-cluster-gmao-kafka-brokers.gmao.svc:9094 (id: 0 rack: null) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-30 08:01:08,956 DEBUG [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Sending FIND_COORDINATOR request with header RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=connect-1, correlationId=35247, headerVersion=2) and timeout 40000 to node 0: FindCoordinatorRequestData(key='', keyType=0, coordinatorKeys=[mirrormaker2-cluster]) (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
2023-06-30 08:01:08,957 DEBUG [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Received FIND_COORDINATOR response from node 0 for request with header RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=connect-1, correlationId=35247, headerVersion=2): FindCoordinatorResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', nodeId=0, host='', port=0, coordinators=[Coordinator(key='mirrormaker2-cluster', nodeId=-1, host='', port=-1, errorCode=15, errorMessage='')]) (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
2023-06-30 08:01:08,957 DEBUG [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Received FindCoordinator response ClientResponse(receivedTimeMs=1688112068957, latencyMs=1, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=connect-1, correlationId=35247, headerVersion=2), responseBody=FindCoordinatorResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', nodeId=0, host='', port=0, coordinators=[Coordinator(key='mirrormaker2-cluster', nodeId=-1, host='', port=-1, errorCode=15, errorMessage='')])) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-30 08:01:08,957 DEBUG [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Group coordinator lookup failed:  (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-30 08:01:08,957 DEBUG [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Coordinator discovery failed, refreshing metadata (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
org.apache.kafka.common.errors.CoordinatorNotAvailableException: The coordinator is not available.