Skip to content

Commit 591645e

Browse files
authored
Update and rename kafka-mirroraker-2-0-guide.md to kafka-mirrormaker-2-0-guide.md
1 parent 737785c commit 591645e

File tree

1 file changed

+20
-21
lines changed

1 file changed

+20
-21
lines changed

articles/hdinsight/kafka/kafka-mirroraker-2-0-guide.md renamed to articles/hdinsight/kafka/kafka-mirrormaker-2-0-guide.md

Lines changed: 20 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
---
2-
title: Apache Kafka MirrorMaker 2.0 Guide - Azure HDInsight
2+
title: Apache Kafka MirrorMaker 2.0 guide - Azure HDInsight
33
description: How to to use Kafka MirrorMaker 2.0 in data migration/replication and the use-cases.
44
ms.service: hdinsight
55
ms.topic: how-to
66
ms.custom: hdinsightactive
77
ms.date: 05/20/2022
88
---
99

10-
# How to use Kafka MirrorMaker 2.0 in data migration, replication and the use-cases.
10+
# How to use Kafka MirrorMaker 2.0 in data migration, replication and the use-cases
1111

1212
MirrorMaker 2 (MM2) is designed to make it easier to mirror or replicate topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify configuration and scaling. It dynamically detects changes to topics and ensures source and target topic properties are synchronized, including offsets and partitions.
1313

1414
In this article, you'll learn how to use Kafka MirrorMaker 2.0 in data migration/replication and the use-cases.
1515

16-
## Pre-requisites
16+
## Prerequisites
1717

1818
* Environment with at least two HDI Kafka clusters.
1919
* Kafka version higher than 2.4 (HDI 4.0)
@@ -29,31 +29,30 @@ Simulation of MirrorMaker 2.0 to replicate data points/offsets between two Kafka
2929

3030
The Mirrormaker2 tool is composed of different connectors. These connectors are standard Kafka Connect connectors, which can be used directly with Kafka Connect in standalone or distributed mode.
3131

32-
3332
The summary of the broker setup process is as follows:
3433

35-
1. MirrorSourceConnector :
34+
**MirrorSourceConnector :**
3635

37-
1. Replicates remote topics, topic ACLs & configs of a single source cluster.
38-
1. Emits offset-syncs to an internal topic.
36+
1. Replicates remote topics, topic ACLs & configs of a single source cluster.
37+
1. Emits offset-syncs to an internal topic.
3938

40-
1. MirrorSinkConnector:
39+
**MirrorSinkConnector:**
4140

42-
1. Consumes from the primary cluster and replicate topics to a single target cluster.
41+
1. Consumes from the primary cluster and replicate topics to a single target cluster.
4342

44-
1. MirrorCheckpointConnector:
43+
**MirrorCheckpointConnector:**
4544

46-
1. Consumes offset-syncsr.
47-
1.Emits checkpoints to enable failover points.
48-
49-
1. MirrorHeartBeatConnector:
45+
1.Consumes offset-syncsr.
46+
1.Emits checkpoints to enable failover points.
47+
48+
**MirrorHeartBeatConnector:**
5049

51-
1. Emits heartbeats to remote clusters, enabling monitoring of replication process.
50+
1. Emits heartbeats to remote clusters, enabling monitoring of replication process.
5251

5352
### Deployment
5453

5554
1. Connect-mirror-maker.sh script bundled with the Kafka library implements a distributed MM2 cluster, which manages the Connect workers internally based on a config file. Internally Mirrormaker driver creates and handles pairs of each connector – MirrorSourceConnector, MirrorSinkConnector, MirrorCheckpoint connector and MirrorHeartbeatConnector.
56-
1. Start the Mirormaker2.
55+
1. Start the Mirormaker 2.
5756

5857
```
5958
./bin/connect-mirror-maker.sh ./config/mirror-maker.properties
@@ -149,21 +148,21 @@ destination.sasl.mechanism=GSSAPI
149148

150149
**How will the consumers behave on migration, if that the destination cluster may have a different offset mapping to data points?**
151150

152-
Mirrormaker2’s MirrorCheckpointConnector automatically stores consumer group offset checkpoints for consumer groups on the source cluster. Each checkpoint contains a mapping of the last committed offset for each group in the source cluster to the equivalent offset in destination cluster. So on migration the consumers that start consuming from same topic on the destination cluster will be able to resume receiving messages from the last offset they committed on the source cluster.
151+
Mirrormaker 2’s MirrorCheckpointConnector automatically stores consumer group offset checkpoints for consumer groups on the source cluster. Each checkpoint contains a mapping of the last committed offset for each group in the source cluster to the equivalent offset in destination cluster. So on migration the consumers that start consuming from same topic on the destination cluster will be able to resume receiving messages from the last offset they committed on the source cluster.
153152

154153
**How can we retain the exact topic name in destination cluster, as the source alias is prefixed with all the topics replicated?**
155154

156-
This is the default behavior in Mirrormaker2.0 to avoid data overriding in complex mirroring topologies. Customization of this needd to be done carefully in terms of replication flow design and topic management to avoid data loss. This can be done by using a custom replication policy class against “replication.policy.class”.
155+
This is the default behavior in Mirrormaker2.0 to avoid data overriding in complex mirroring topologies. Customization of this needs to be done carefully in terms of replication flow design and topic management to avoid data loss. This can be done by using a custom replication policy class against “replication.policy.class”.
157156

158157
**Why do we see new internal topics created in my source and destination Kafka?**
159158

160-
Mirrormaker2 internal topics are created by the Connectors to keep track of the replication process, monitoring, offset mapping and checkpointing.
159+
Mirrormaker 2 internal topics are created by the Connectors to keep track of the replication process, monitoring, offset mapping and checkpointing.
161160

162161
**Why does the mirrormaker creates only two replicas of the topic in the destination cluster while the source has more?**
163162

164-
Mirrormaker2 doesn’t replicate the replication factor of topics to target clusters. This can be controlled from MM2 config, by specifying the required number of “replication.factor”. The default value for the same is two.
163+
Mirrormaker 2 doesn’t replicate the replication factor of topics to target clusters. This can be controlled from MM2 config, by specifying the required number of “replication.factor”. The default value for the same is two.
165164

166-
**How to use custom replication policy in Mirrormaker2?**
165+
**How to use custom replication policy in Mirrormaker 2?**
167166

168167
Custom Replication Policy can be created by implementing the interface below.
169168

0 commit comments

Comments
 (0)