MirrorMaker2 throwing error - NOT_ENOUGH_REPLICAS

[AAR256]

I'm running Kafka version 3.0.0 on AKS clusters. The Kafka Cluster definition that I'm using for both the clusters is -

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cdt-north-1-kafka-cluster
  namespace: kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    resources:
      requests:
        memory: 500Mi
        cpu: "100m"
      limits:
        memory: 600Mi
        cpu: "1"
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            host: cdt-north-1-kafka-cluster-1-bootstrap.net
          brokers:
          - broker: 0
            host: cdt-north-1-kafka-cluster-1-kafka-0.net
          - broker: 1
            host: cdt-north-1-kafka-cluster-1-kafka-1.net
          - broker: 2
            host: cdt-north-1-kafka-cluster-1-kafka-2.net
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    authorization:
      type: simple
      superUsers:
        - admin
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "3.0"
      inter.broker.protocol.version: "3.0"
      default.replication.factor: 3
      min.insync.replicas: 3
      num.partitions: 1
      log.segment.bytes: 1073741824
      log.retention.check.interval.ms: 300000
      num.network.threads: 3
      num.io.threads: 8
      num.recovery.threads.per.data.dir: 1
      socket.send.buffer.bytes: 102400
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      group.initial.rebalance.delay.ms: 0
      replica.fetch.max.bytes: 1048576
      auto.create.topics.enable: false
      delete.topic.enable: true
      offsets.topic.num.partitions: 50
      offsets.commit.required.acks: -1
      queued.max.requests: 500
      replica.socket.receive.buffer.bytes: 65536
      log.roll.ms: 604800000
      log.retention.bytes: 1073741824
      log.retention.ms: 1680000
      log.segment.delete.delay.ms: 60000
      log.cleanup.policy: compact,delete
      log.cleaner.backoff.ms: 15000
      log.cleaner.delete.retention.ms: 86400000
      log.cleaner.dedupe.buffer.size: 134217728
      log.cleaner.io.buffer.load.factor: 0.9
      log.cleaner.threads: 8
      log.cleaner.io.max.bytes.per.second: 1.7976931348623157E308
      message.max.bytes: 1048576
      compression.type: producer
      log.flush.scheduler.interval.ms: 2000
      log.flush.interval.ms: 50000
      log.flush.interval.messages: 100000
      replica.lag.time.max.ms: 500
      auto.leader.rebalance.enable: true
      leader.imbalance.check.interval.seconds: 300
      leader.imbalance.per.broker.percentage: 10
      unclean.leader.election.enable: false
      group.initial.rebalance.delay.ms: 3000
      zookeeper.connection.timeout.ms: 6000
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
        class: kafka-storage
      - id: 1
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
        class: kafka-storage
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 2
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-north-1-kafka-cluster-kafka
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 200Mi
        cpu: "100m"
      limits:
        memory: 400Mi
        cpu: "200m"
    storage:
      type: persistent-claim
      size: 20Gi
      deleteClaim: true
      class: zookeeper-storage
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 2
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-north-1-kafka-cluster-zookeeper
  entityOperator:
    tlsSidecar:
      resources:
        requests:
          cpu: 100m
          memory: 64Mi
        limits:
          cpu: 200m
          memory: 128Mi
    topicOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "100m"
        limits:
          memory: 512Mi
          cpu: "200m"
    userOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "100m"
        limits:
          memory: 512Mi
          cpu: "200m"
  cruiseControl:
    tlsSidecar:
      resources:
        requests:
          memory: 512Mi
          cpu: "1"
        limits:
          memory: 2Gi
          cpu: "2"
    resources:
      requests:
        memory: 512Mi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "2"
  kafkaExporter:
    resources:
      requests:
        cpu: 100m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 128Mi
    topicRegex: ".*"
    groupRegex: ".*"

The MirrorMaker definition used here is -

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mm-cluster
  labels:
    app: my-mm-cluster
spec:
  version: 3.0.0
  replicas: 1
  connectCluster: cdt-west-2 #"my-cluster-target"
  clusters:
    - alias: cdt-north-1 #"my-cluster-source"
      authentication:
        type: scram-sha-512
        username: mm2-credentials
        passwordSecret:
          secretName: cdt-north-1-mm2-credentials
          password: password
      tls:
        trustedCertificates:
          - secretName: cdt-north-1-kafka-cluster-cluster-ca-cert
            certificate: ca.crt
      bootstrapServers: cdt-north-1-kafka-cluster-1-bootstrap.net:443
    - alias: cdt-west-2 #"my-cluster-target"
      authentication:
        type: scram-sha-512
        username: mm2-credentials
        passwordSecret:
          secretName: mm2-credentials
          password: password
      tls:
        trustedCertificates:
          - secretName: cdt-west-2-kafka-cluster-cluster-ca-cert
            certificate: ca.crt
      bootstrapServers: cdt-west-2-kafka-cluster-bootstrap.net:443
      config:
      # -1 means it will use the default replication factor configured in the broker
        config.storage.replication.factor: 1
        offset.storage.replication.factor: 1
        status.storage.replication.factor: 1
        offset.storage.topic: migration-connect-cluster-offsets
        config.storage.topic: migration-connect-cluster-configs
        status.storage.topic: migration-connect-cluster-status
  mirrors:
  - sourceCluster: "cdt-north-1"
    targetCluster: "cdt-west-2"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        replication.policy.separator: ""
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
        refresh.groups.interval.seconds: 600
        # migrates the consumer group offsets
        emit.checkpoints.enabled: true
        sync.group.offsets.enabled: true
        sync.group.offsets.interval.seconds: 60
        emit.checkpoints.interval.seconds: 60
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        replication.policy.separator: ""
    topicsPattern: ".*"
    groupsPattern: ".*"
  resources:
    requests:
      memory: 8Gi
      cpu: "1"
    limits:
      memory: 10Gi
      cpu: "2"

Now, the MM2 pod is up and running but when I view the logs, I get the below error warning.

2022-03-15 11:00:44,817 WARN [Producer clientId=producer-4] Got error produce response with correlation id 533 on topic-partition mm2-offset-syncs.cdt-west-2.internal-0, retrying (2147483472 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | producer-4]

I have tried increasing the min.insync.replicas in Kafka Cluster definition as well but that didn't helped. Any input here would be appreciated.

[scholzj]

This normally means that there are not enough replicas of given partition online and in-sync. But the reasons for that can be different.

[AAR256]

2022-08-05 08:10:27,063 WARN [Producer clientId=producer-3] Got error produce response with correlation id 1774 on topic-partition mirrormaker2-cluster-configs-0, retrying (2147481876 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | producer-3]

In the logs, we can see it's failing for topic mirrormaker2-cluster-configs which has a default replication factor 1. Can we somehow bypass this or increase it to 3? Note - This topic is present in destination Kafka Cluster.

[scholzj]

This looks like a Mirror Maker topics -> you configure the replication factor of it in the custom resource when you deploy MM2. For example in the custom resource above:

        config.storage.replication.factor: 1
        offset.storage.replication.factor: 1
        status.storage.replication.factor: 1

[AAR256]

Thanks, for the MM2 default topics we're able to increase it to 3 using the mentioned configuration but now it's throwing a different error -

2022-08-08 08:03:34,977 WARN [Consumer clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster] Received unknown topic or partition error in fetch for partition mirrormaker2-cluster-offsets-10 (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]

Also, another question is can we configure the replication factor for default cruise control topics as well? I don't see the option to configure those in the docs. Any guidance here would be great.

[IronforgeV]

stop the connector, delete the topic on the src cluster, and set
"offset-syncs.topic.replication.factor" to a reasonable number in the mirror source connector config, i.e.
"offset-syncs.topic.replication.factor": "3"

[wangjia007bond]

Hi @AAR256

Could you share your final MM2 config and summarise what cause the error?

Cheers,
Steven