Mirror Maker2 throw Error: NOT_ENOUGH_REPLICAS

[wangjia007bond]

Bidirectional data replication not happening in strimzi mirrormaker2

Hello, I am using Strimzi MirrorMaker2 for replicating data between two kafka clusters. Bidirectional data replication is not happening in my case.

Providing my kafka config file

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prod-ae-cluster
  namespace: kafka1
spec:
  cruiseControl: {}
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 3
      inter.broker.protocol.version: '3.3'
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
    replicas: 3
    storage:
      type: ephemeral
    version: 3.4.0
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prod-ase-cluster
  namespace: kafka2
spec:
  cruiseControl: {}
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 3
      inter.broker.protocol.version: '3.3'
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
    replicas: 3
    storage:
      type: ephemeral
    version: 3.4.0
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

mirror maker 2 config

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: prod-ase-to-ae-mm2
  namespace: kafka1
spec:
  clusters:
    - alias: my-cluster-source
      bootstrapServers: 'prod-ase-cluster-kafka-bootstrap.kafka2.svc.cluster.local:9092'
    - alias: my-cluster-target
      bootstrapServers: 'prod-ae-cluster-kafka-bootstrap.kafka1.svc.cluster.local:9092'
      config:
        config.storage.replication.factor: -1
        offset.storage.replication.factor: -1
        status.storage.replication.factor: -1
  connectCluster: my-cluster-target
  mirrors:
    - checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 1
      groupsPattern: .*
      heartbeatConnector:
        config:
          heartbeats.topic.replication.factor: 1
      sourceCluster: my-cluster-source
      sourceConnector:
        config:
          offset-syncs.topic.replication.factor: 1
          replication.factor: 1
          sync.topic.acls.enabled: 'false'
      targetCluster: my-cluster-target
      topicsPattern: .*
  replicas: 1
  version: 3.4.0

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: prod-ae-to-ase-mm2
  namespace: kafka2
spec:
  clusters:
    - alias: my-cluster-source
      bootstrapServers: 'prod-ae-cluster-kafka-bootstrap.kafka1.svc.cluster.local:9092'
    - alias: my-cluster-target
      bootstrapServers: 'prod-ase-cluster-kafka-bootstrap.kafka2.svc.cluster.local:9092'
      config:
        config.storage.replication.factor: -1
        offset.storage.replication.factor: -1
        status.storage.replication.factor: -1
  connectCluster: my-cluster-target
  mirrors:
    - checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 1
      groupsPattern: .*
      heartbeatConnector:
        config:
          heartbeats.topic.replication.factor: 1
      sourceCluster: my-cluster-source
      sourceConnector:
        config:
          offset-syncs.topic.replication.factor: 1
          replication.factor: 1
          sync.topic.acls.enabled: 'false'
      targetCluster: my-cluster-target
      topicsPattern: .*
  replicas: 1
  version: 3.4.0

and I get error like below:

2023-06-02 05:18:41,588 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 105 on topic-partition my-cluster-source.mirrormaker2-cluster-status-4, retrying (2147483635 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,599 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 106 on topic-partition my-cluster-source.mirrormaker2-cluster-configs-0, retrying (2147483635 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,607 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 107 on topic-partition my-cluster-source.steven-topic-0, retrying (2147483635 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,607 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 108 on topic-partition my-cluster-source.strimzi.cruisecontrol.partitionmetricsamples-1, retrying (2147483635 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,633 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 109 on topic-partition my-cluster-source.strimzi.cruisecontrol.partitionmetricsamples-19, retrying (2147483635 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,638 WARN [my-cluster-source->my-cluster-target.MirrorHeartbeatConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorHeartbeatConnector-0] Got error produce response with correlation id 11 on topic-partition heartbeats-0, retrying (2147483639 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorHeartbeatConnector-0]
2023-06-02 05:18:41,642 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 110 on topic-partition my-cluster-source.mirrormaker2-cluster-status-0, retrying (2147483635 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,649 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 111 on topic-partition my-cluster-source.strimzi.cruisecontrol.modeltrainingsamples-30, retrying (2147483632 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,654 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 112 on topic-partition my-cluster-source.strimzi.cruisecontrol.metrics-0, retrying (2147483632 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,690 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 113 on topic-partition my-cluster-source.mirrormaker2-cluster-status-1, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,690 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 113 on topic-partition my-cluster-source.mirrormaker2-cluster-status-4, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,700 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 114 on topic-partition my-cluster-source.mirrormaker2-cluster-configs-0, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,707 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 115 on topic-partition my-cluster-source.steven-topic-0, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,708 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 116 on topic-partition my-cluster-source.strimzi.cruisecontrol.partitionmetricsamples-1, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,734 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 117 on topic-partition my-cluster-source.strimzi.cruisecontrol.partitionmetricsamples-19, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,739 WARN [my-cluster-source->my-cluster-target.MirrorHeartbeatConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorHeartbeatConnector-0] Got error produce response with correlation id 12 on topic-partition heartbeats-0, retrying (2147483638 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorHeartbeatConnector-0]
2023-06-02 05:18:41,743 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 118 on topic-partition my-cluster-source.mirrormaker2-cluster-status-0, retrying (2147483634 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,748 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 119 on topic-partition my-cluster-source.strimzi.cruisecontrol.modeltrainingsamples-30, retrying (2147483631 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,755 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 120 on topic-partition my-cluster-source.strimzi.cruisecontrol.metrics-0, retrying (2147483631 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,791 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 121 on topic-partition my-cluster-source.mirrormaker2-cluster-status-1, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,791 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 121 on topic-partition my-cluster-source.mirrormaker2-cluster-status-4, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,801 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 122 on topic-partition my-cluster-source.mirrormaker2-cluster-configs-0, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,808 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 123 on topic-partition my-cluster-source.steven-topic-0, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,808 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 124 on topic-partition my-cluster-source.strimzi.cruisecontrol.partitionmetricsamples-1, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,835 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 125 on topic-partition my-cluster-source.strimzi.cruisecontrol.partitionmetricsamples-19, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,839 WARN [my-cluster-source->my-cluster-target.MirrorHeartbeatConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorHeartbeatConnector-0] Got error produce response with correlation id 13 on topic-partition heartbeats-0, retrying (2147483637 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorHeartbeatConnector-0]
2023-06-02 05:18:41,844 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 126 on topic-partition my-cluster-source.mirrormaker2-cluster-status-0, retrying (2147483633 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,849 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 127 on topic-partition my-cluster-source.strimzi.cruisecontrol.modeltrainingsamples-30, retrying (2147483630 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,856 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 128 on topic-partition my-cluster-source.strimzi.cruisecontrol.metrics-0, retrying (2147483630 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]
2023-06-02 05:18:41,892 WARN [my-cluster-source->my-cluster-target.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0] Got error produce response with correlation id 129 on topic-partition my-cluster-source.mirrormaker2-cluster-status-1, retrying (2147483632 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | connector-producer-my-cluster-source->my-cluster-target.MirrorSourceConnector-0]

Does anyone know which config is wrong across my kafka and mm2?

Cheers,
Steven

[kyguy]

Hi @wangjia007bond, what is the replication factor of the topic not being mirrored?