You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: How to to use Kafka MirrorMaker 2.0 in data migration/replication and the use-cases.
4
4
ms.service: hdinsight
5
5
ms.topic: how-to
6
6
ms.custom: hdinsightactive
7
7
ms.date: 05/20/2022
8
8
---
9
9
10
-
# How to use Kafka MirrorMaker 2.0 in data migration, replication and the use-cases.
10
+
# How to use Kafka MirrorMaker 2.0 in data migration, replication and the use-cases
11
11
12
12
MirrorMaker 2 (MM2) is designed to make it easier to mirror or replicate topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify configuration and scaling. It dynamically detects changes to topics and ensures source and target topic properties are synchronized, including offsets and partitions.
13
13
14
14
In this article, you'll learn how to use Kafka MirrorMaker 2.0 in data migration/replication and the use-cases.
15
15
16
-
## Pre-requisites
16
+
## Prerequisites
17
17
18
18
* Environment with at least two HDI Kafka clusters.
19
19
* Kafka version higher than 2.4 (HDI 4.0)
@@ -29,31 +29,30 @@ Simulation of MirrorMaker 2.0 to replicate data points/offsets between two Kafka
29
29
30
30
The Mirrormaker2 tool is composed of different connectors. These connectors are standard Kafka Connect connectors, which can be used directly with Kafka Connect in standalone or distributed mode.
31
31
32
-
33
32
The summary of the broker setup process is as follows:
34
33
35
-
1.MirrorSourceConnector :
34
+
**MirrorSourceConnector :**
36
35
37
-
1. Replicates remote topics, topic ACLs & configs of a single source cluster.
38
-
1. Emits offset-syncs to an internal topic.
36
+
1. Replicates remote topics, topic ACLs & configs of a single source cluster.
37
+
1. Emits offset-syncs to an internal topic.
39
38
40
-
1.MirrorSinkConnector:
39
+
**MirrorSinkConnector:**
41
40
42
-
1. Consumes from the primary cluster and replicate topics to a single target cluster.
41
+
1. Consumes from the primary cluster and replicate topics to a single target cluster.
43
42
44
-
1.MirrorCheckpointConnector:
43
+
**MirrorCheckpointConnector:**
45
44
46
-
1.Consumes offset-syncsr.
47
-
1.Emits checkpoints to enable failover points.
48
-
49
-
1.MirrorHeartBeatConnector:
45
+
1.Consumes offset-syncsr.
46
+
1.Emits checkpoints to enable failover points.
47
+
48
+
**MirrorHeartBeatConnector:**
50
49
51
-
1. Emits heartbeats to remote clusters, enabling monitoring of replication process.
50
+
1. Emits heartbeats to remote clusters, enabling monitoring of replication process.
52
51
53
52
### Deployment
54
53
55
54
1. Connect-mirror-maker.sh script bundled with the Kafka library implements a distributed MM2 cluster, which manages the Connect workers internally based on a config file. Internally Mirrormaker driver creates and handles pairs of each connector – MirrorSourceConnector, MirrorSinkConnector, MirrorCheckpoint connector and MirrorHeartbeatConnector.
**How will the consumers behave on migration, if that the destination cluster may have a different offset mapping to data points?**
151
150
152
-
Mirrormaker2’s MirrorCheckpointConnector automatically stores consumer group offset checkpoints for consumer groups on the source cluster. Each checkpoint contains a mapping of the last committed offset for each group in the source cluster to the equivalent offset in destination cluster. So on migration the consumers that start consuming from same topic on the destination cluster will be able to resume receiving messages from the last offset they committed on the source cluster.
151
+
Mirrormaker 2’s MirrorCheckpointConnector automatically stores consumer group offset checkpoints for consumer groups on the source cluster. Each checkpoint contains a mapping of the last committed offset for each group in the source cluster to the equivalent offset in destination cluster. So on migration the consumers that start consuming from same topic on the destination cluster will be able to resume receiving messages from the last offset they committed on the source cluster.
153
152
154
153
**How can we retain the exact topic name in destination cluster, as the source alias is prefixed with all the topics replicated?**
155
154
156
-
This is the default behavior in Mirrormaker2.0 to avoid data overriding in complex mirroring topologies. Customization of this needd to be done carefully in terms of replication flow design and topic management to avoid data loss. This can be done by using a custom replication policy class against “replication.policy.class”.
155
+
This is the default behavior in Mirrormaker2.0 to avoid data overriding in complex mirroring topologies. Customization of this needs to be done carefully in terms of replication flow design and topic management to avoid data loss. This can be done by using a custom replication policy class against “replication.policy.class”.
157
156
158
157
**Why do we see new internal topics created in my source and destination Kafka?**
159
158
160
-
Mirrormaker2 internal topics are created by the Connectors to keep track of the replication process, monitoring, offset mapping and checkpointing.
159
+
Mirrormaker 2 internal topics are created by the Connectors to keep track of the replication process, monitoring, offset mapping and checkpointing.
161
160
162
161
**Why does the mirrormaker creates only two replicas of the topic in the destination cluster while the source has more?**
163
162
164
-
Mirrormaker2 doesn’t replicate the replication factor of topics to target clusters. This can be controlled from MM2 config, by specifying the required number of “replication.factor”. The default value for the same is two.
163
+
Mirrormaker 2 doesn’t replicate the replication factor of topics to target clusters. This can be controlled from MM2 config, by specifying the required number of “replication.factor”. The default value for the same is two.
165
164
166
-
**How to use custom replication policy in Mirrormaker2?**
165
+
**How to use custom replication policy in Mirrormaker 2?**
167
166
168
167
Custom Replication Policy can be created by implementing the interface below.
0 commit comments