MirrorMaker2 (sometimes?) not picking up newly-added topics, requiring a manual restart?

[frankgrimes97]

Hi, we're running strimzi-0.28.0 and kafka-3.0.0 and have noticed that when applying a change to topicsPattern to explicitly (not using wildcards) add a newly-created topic the mirroring doesn't start consistently.
We noticed that even after many hours it had not started the mirroring of the new topic, so it doesn't appear to be explained by a simple delay happening... it was only after a full restart of the MirrorMaker2 pods that we saw data start to be mirrored for the newly-added topic.

I found the following Kafka bug report upstream which indicates that Kafka itself doesn't gracefully handle this:

• Mirror Maker 2 not replicating new topics until restart - https://issues.apache.org/jira/browse/KAFKA-10582

Does or should Strimzi restart the MirrorMaker2 pods when topicsPattern changes are applied on the KafkaMirrorMaker2 CRD config?

Has anyone else hit this same problem? Is there extra configuration or monitoring required to ensure that all listed topics are in fact mirroring data?

Thanks!

[frankgrimes97]

@scholzj Is this something you've come across before? Any help/hints/insights would be most welcome! 🙏

[scholzj]

I'm not aware of any issue like this. But to be honest, it is more on the Mirror Maker 2 side rather than Strimzi. So I would not be the expert for it. It is also worth pointing that a lot has changed to Mirror Maker 2 since Kafka 3.0.0. So it might be worth trying if it works better in newer Kafka version (although the issue you linked does not seem to be fixed yet in any version).

[frankgrimes97]

FYI, I found the following fix in the Kafka-3.7.0 release notes (also fixed in the upcoming Kafka-3.6.2 release) that looks to me like it may be related: https://issues.apache.org/jira/browse/KAFKA-15372

[frankgrimes97]

FYI, it seems https://issues.apache.org/jira/browse/KAFKA-10582 has just been marked as closed and fixed in Kafka-3.5.0.

[scholzj]

You can't it changed in any way. We run Connect, that has its own REST API and that manages the connectors. So I do not think we face any problems like that.

[frankgrimes97]

So KIP-710 isn't applicable at all to Strimzi?
So a similar bug lies elsewhere?

[scholzj]

I do not think the KIP is related in any way to how Strimzi runs MM2.

[frankgrimes97]

So from KIP-382 it appears that there are 4 ways to run MirrorMaker 2.0:

• As a dedicated MirrorMaker cluster.
• As a Connector in a distributed Connect cluster.
• As a standalone Connect worker.
• In legacy mode using existing MirrorMaker scripts.

So just to confirm/clarify...

The Kafka MirrorMaker 2.0 bugs we found are all related to:

• As a dedicated MirrorMaker cluster.

Strimzi is running MirrorMaker 2.0 as follows:

• As a Connector in a distributed Connect cluster.

I've dug a bit more into the Strimzi code and found the following in the newer MM2 reconciliation code:

strimzi-kafka-operator/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaMirrorMaker2AssemblyOperator.java

Lines 310 to 315 in f7e7789

	var autoRestart = result.result().autoRestart;
	if (autoRestart != null) {
	autoRestart.setConnectorName(connectorName);
	mirrorMaker2Status.getAutoRestartStatuses().add(autoRestart);
	mirrorMaker2Status.getAutoRestartStatuses().sort(Comparator.comparing(AutoRestartStatus::getConnectorName));
	}

I see that this autoRestart code logic isn't present in Strimzi-0.28.0 but was added later.
As per https://strimzi.io/blog/2023/01/25/auto-restarting-connectors:

When enabled, the Strimzi Cluster Operator watches the connector and its tasks for failures in every reconciliation.

We're in the process of rolling out Strimzi-0.38.0 and Kafka-3.6.0 and are wondering if that might help.
However, we didn't notice any MM2 Connect task failures occur when the topicsPattern config fails to propagate so autoRestart won't likely occur in those cases, right?

Is there possibly something else non-obvious in the reconciliation code which might explain the lack of the latest config values being applied in the running MM2 Connectors? e.g. in maybeCreateOrUpdateConnector?

[scholzj]

The auto-restart feature was added only after Strimzi 0.28. But keep in mind that you have to enable it and if the connector / tasks never fail and just silently ignore some topics, it will likely not help. That said if you are running Strimzi 0.28 with Kafka 3.0 / 3.1, a lot changed / improved in Mirror MAker 2 since then. So the best start would be to see if you still have the issue there.

PS: 0.40.0 with Kafka 3.7.0 support is likely to be released soon ...