MirrorMaker2 custom probes

[scottc-git]

Question:
Is there a way to provide my own Readiness probe to MirrorMaker2?

Background
I am using a few SMTs for the MirrorMaker2 - one of them is to copy the Avro schemas from the source Kafka to target (Strimzi) Kafka environment upon mirroring. The problem is the timing issue when SchemaRegistry becomes ready sometime after Mirroring started. The SMT will fail if the target schema registry is not ready yet and the MirrorMaker worker thread will eventually stop. I will have to restart the MM2 pod manually and then it will resume the mirroring. The whole Strimzi Kafka (mirror) environment can be built a few times a day and this problem has been making the test environments unreliable to my users.

Error Logs

2022-10-03 18:58:51,981 ERROR WorkerSourceTask{id=mirrortxn->strimzi-zxrt.MirrorSourceConnector-6} 
Task threw an uncaught and unrecoverable exception. 
Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) 
[task-thread-mirrortxn->strimzi-zxrt.MirrorSourceConnector-6]

Real Question
According to the Kafka documentation, setting errors.tolerance to none (default) seems to be a solution to this but the tasks do not come back up as the error message says. In the meantime, Pod status is still Running (with the dead threads) and Kubernetes will not bother recreating the pod either. Is there a way to configure the KMM2 to overcome this situation? I was going to override the Readiness probe to httpGet against http://schema-registry-client:8081 but it seems the [Probe](https://strimzi.io/docs/operators/0.27.1/using.html#type-Probe-reference) only checks on its own service (mm2) being running.

KMM2 spec

    mirrors:
    - checkpointConnector:
        config:
          sync.group.offsets.enabled: false
      groupsPattern: do-not-mirror-groups
      sourceCluster: mirrorref
      sourceConnector:
        config:
          checkpoints.topic.replication.factor: 1
          errors.log.enable: "true"
          errors.log.include.messages: "true"
          errors.tolerance: none           <== Do Not Ignore Errors
          heartbeats.topic.replication.factor: 1
          offset-syncs.topic.replication.factor: 3
          refresh.topics.interval.seconds: 60
          replication.factor: 1
          replication.policy.class: io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy
          replication.policy.separator: ""
          sync.topic.acls.enabled: "false"
          transforms: AvroSchemaTransfer
          transforms.AvroSchemaTransfer.dest.schema.registry.url: http://schema-registry-client:8081
          transforms.AvroSchemaTransfer.schema.capacity: 5000
          transforms.AvroSchemaTransfer.src.schema.registry.url: http://schema-registry-client.tdp-mirror-az-stg:8081
          transforms.AvroSchemaTransfer.transfer.message.keys: "false"
          transforms.AvroSchemaTransfer.type: cricket.jmoore.kafka.connect.transforms.SchemaRegistryTransfer
        tasksMax: 10
      targetCluster: strimzi-zwim
      topicsExcludePattern: __consumer_offsets, _schemas, heartbeats
      topicsPattern: ^master-data.(bar.v1|foo.v1)$

Alternative considered
I was thinking of using init-container for more extensive health check but previous discussions suggested that it is going to be more rough path and it is not supported.

Version
Strimzi 0.27.1

Thanks in advance.

[scholzj]

To answer your question, no, you cannot customize the healthcheck commands. You can only change the timings. I do not think customizing the healthcheck commands would make sense. The operator needs to know what they do, what to expect from them and that it can rely on them. You cannot set them to whatever you want. Tying the readiness to some completely different service does not seem to make much sense to me TBH.

TBH, I don't follow the whole idea to have some SMT to mirror the schemas. Shouldn't you simply have other tool (a proper tool running in its own pod) to mirror the schema? Using an SMT for it seems like a complete hack.

[scottc-git]

Thanks for the comment @scholzj, as usual.

schema-registry-transfer-smt is the one that I have been using for years. It works fine except for the situations described above. Users will have to restart the MM2 pod when that happens - which is not so great. Having a separate tool just to mirror the schemas can be an option but IMHO it may suffer from the similar timing issue depending upon the frequency of the schema changes and etc. What if MM2 wins the race before the schema gets replicated? It will say 'Schema not found' and give up.

So looks like I am back to the drawing board. I will test with below retry settings and see if it works.

          errors.tolerance: none
          # retry for at most 10 minutes times waiting up to 30 seconds between consecutive failures
          errors.retry.timeout: 600000
          errors.retry.delay.max.ms: 30000

[scholzj]

Maybe I don't understand your use-case. But normally, you have your registry somewhere and it stores the schemas in some storage. So depending on what the storage is, if you need it mirrored you setup the mirroring (DB, NFS volume etc.) and it is an independent service. If it is a Kafka topic, you should be able to mirror it just as any other Kafka topic. There should be no need to piggy back this on some SMT.