multiple MirrorMaker2 connecting to single kafka-cluster

[warmuuh]

Is your feature request related to a problem? Please describe.
our company works with several teams, each of them "owning" their own kafka-infrastructure. that is, they also manage mm2. now the need arises to mirror from several of those kafka-clusters into a single kafka-cluster (somewhere totally different).
We tried several things (further down) but it seems that this setup is not easy to support when there should be independent teams managing their mm2 instance separately.

Describe the solution you'd like
I would like to see support for running multiple mm2 instances vs a single kafka-cluster. for this, i think, it is necessary to customize all the values inside of KafkaMirrorMaker2Configuration (based on these docs), which is currently not possible (or is it?)

Describe alternatives you've considered
we tried to have fan-out where a mm2 instance writes from to but this is also not working as MM2 uses SourceConnectors.

The only option i currently see is have only one mirrormaker that is then modified by all the teams (i really would like to avoid this though)

[scholzj]

I'm not sure I understand what is the problem. AFAIK, It is possible to run multiple MM2s connected to the same cluster. You just need to agree on some naming rules to not conflict. Depending on what you do, this might also include naming of the internal MM2 topics etc.

[warmuuh]

thank you for your fast response. when setting up two mm2 to have the same connectCluster, they'll create intermediate topics ("mirrormaker2-cluster-offsets", "mirrormaker2-cluster-status", "mirrormaker2-cluster-configs".) and use the same group.id. and because i could find no way to modify these values in the KafkaMirrorMaker2 resource, these two mm2 instances think, they belong to the same KafkaConnect cluster and try to work together). as written in the docs, if those two should be totally independend, these 4 values have to be different.

[scholzj]

Have you tried to set it like here: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/mirror-maker/kafka-mirror-maker-2.yaml#L16-L18 ... basically just add this (with what ever values you want to use):

    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status

to the same place?

[warmuuh]

it worked... thank you ;) i was expecting the props to go closer to the connectCluster