Active/Active Setup Consuming Topic Recommendation

[vjvel]

Hi @scholzj

We have deployed 2 cluster in different DC (active/active) setup using mirror maker at both side.

Currently Route53 pointing to primary DC so producer and consumer both use same Route53 to reach the primary DC kafka for the topic "my-topic-1" , In case of failure in primary DC, route53 automatically routes traffic to secondary dc but here topic will be "primary.my-topic-1" and "my-topic-1" . In this case how to consume the topic from the secondary dc when failover. If we use pattern topic in consumer, the offset will be tracked for the local topic and mirrored topics ?

Can you give some ideas on consuming topics for the active/active setup? Thanks

[scholzj]

Sorry, I don't follow what exactly are you asking. The topics will be mirrored into their prefixed mirrors from which you can consume the messages.

[vjvel]

If i use topic pattern to match local "my-topic-1" and mirror topic "primary.my-topic-1" , will this consume duplicate message from both topics. How this will work?

[scholzj]

That depends on what exactly you do and how. The active/active mirroring is not necessarily designed for switching the clients between the clusters all the time. The topics data will be mirrored into their prefixed versions in the other cluster. If you enable offset synchronization, the offsets will be synchronized as well. But they are mirrored only in one direction. So any offsets committed in the mirrored prefixed topic will nto be synchronized back. From how you describe it, I wonder if active-active mirroring is really what you need. if you just want to have a backup cluster you can switch to in case of some disaster in the primary DC, you probably want active-passive mirroring.

[scholzj]

PS: You should always have some level of duplicate detection as duplicates can occur anyway and the mirroring is eventually consistent.

[vjvel]

In case of active/passive mirroring , it will be one way data mirroring. If I again switchover from secondary to primary site, Secondary topic data won't be available right?

We need HA setup , in case of failure of one site it should work with another site and not to lose data.