MirrorSourceConnector that creates remote topics. (target topic name)

[mike-serchenia]

Hi
i was checking the documentation regarding mirror maker 2, looks like while replicating it prepends cluster-name to the terget topic. Is there a way to customize it? To prepend or append another prefix/postfix ? Or even provide custom dst topic name.

sth like

targetTopic:
  -  'oldTopicName:newTopicName'

The main motivation of it is to have possibility to rename topics during replication.

the only way I found to do that is:
https://github.com/opencore/mirrormaker_topic_rename

but it's way too slow. 50k messages in 3h

[scholzj]

i was checking the documentation regarding mirror maker 2, looks like while replicating it prepends cluster-name to the terget topic. Is there a way to customize it? To prepend or append another prefix/postfix ? Or even provide custom dst topic name.

There is a Identity Replication Policy which you can use if you do not want to have the cluster name prefixed. I'm not aware of anything to add some custom prefix / suffix. You would need to write it your self.

The link looks like something for Mirror Maker 1 where as you talk here about Mirror Maker 2.

[scholzj]

Why do you need the same name on both cluster with active/active mirroring? It does not work like that, so it should not be needed.

[vjvel]

we need to configure route53 to have both kafka endpoints, if any one cluster is down, My application (producer and consumer) should use the same topic name and redirect to the healthy cluster. @scholzj

what is the active/active does with different topic name ?

[scholzj]

But why do you need the same topic names? You can simply subscribe to topic pattern and that is it.

[vjvel]

I am not sure about producer side how it can work using topic pattern ?

I am begineer in kafka and sorry if my question is not clear. @scholzj

we need DR setup and it should do autofailover in case of one cluster is down. our application (producer and cosumer) will point to the other cluster using route53/cloudflare LB in case of failure. What is the best approch to do this ?

[scholzj]

Normally, you producer always produces only into the local topic. E.g. my-topic. Where as in the consumer, you will use the pattern to consume from my-topic but also from <prefix>.my-topic for example.