Mirrormaker2 Internal Topics Not Created

[towens]

Hello,

We are trying to discover why internal mm2 topics are not being created after an mm2 teardown. We dropped mm2 and the internal topics, now the internal/mm2 topics are not being created when re-submitting the mm2 spec.

Relevant log entries.
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

org.apache.kafka.connect.errors.ConnectException: Could not look up partition metadata for offset backing store topic in allotted period. This could indicate a connectivity issue, unavailable topic partitions, or if this is your first use of the topic it may have taken too long to create.

Which makes sense if the topics aren't created. They aren't :)

No errors on the ssl trust store. Source cert is at generation one. It has been pulled down and used to verify access with kcat

kcat -L -b remote-source-bootstrap.redacted.com:9094 -X security.protocol=ssl -X ssl.ca.location=ca.crt

This same cert is used as the trustedCertificate on the destination cluster. Assuming there would be an auth error if there were diffs in those certs.

Here is the mm2 spec

kind: KafkaMirrorMaker2
metadata:
  name: mm2
spec:
  version: 3.2.0
  replicas: 5
  resources:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 2Gi
  template:
    pod:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            strimzi.io/name: mm2
  connectCluster: "local-target"
  clusters:
  - alias: "remote-source"
    bootstrapServers: remote-source-bootstrap.redacted.com:9094
    tls:
      trustedCertificates:
      - secretName: remote-source-cluster-secret
        certificate: ca.crt
  - alias: "local-target"
    bootstrapServers: local-target-bootstrap.redacted.svc.cluster.local:9092
    config:
      config.storage.replication.factor: 3
      offset.storage.replication.factor: 3
      status.storage.replication.factor: 3
  mirrors:
  - sourceCluster: "remote-source"
    targetCluster: "local-target"
    sourceConnector:
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 3
        sync.topic.acls.enabled: "false"
        replication.policy.separator: ""
        replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
        offset.storage.replication.factor: 3
        status.storage.replication.factor: 3
        # same as policy comment above
        replication.policy.separator: ""
        replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
        sync.group.offsets.enabled: "true"
    topicsPattern: "the-topic"
    groupsPattern: ".*"
  logging:
    type: inline
    loggers:
      connect.root.logger.level: "ERROR"

Hoping we didn't miss something obvious from the docs and examples. Thanks for any guidance.

[towens]

One moment. Wider problems with Kafka than MM2.

[towens]

Silly me. I was looking for errors and had to log level DEBUG to find an error with an upgrade from 0.28 to 0.29. False alarm.