You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/kafka/kafka-mirrormaker-2-0-guide.md
+12-12Lines changed: 12 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ ms.date: 05/20/2022
9
9
10
10
# How to use Kafka MirrorMaker 2.0 in data migration, replication and the use-cases
11
11
12
-
MirrorMaker 2 (MM2) is designed to make it easier to mirror or replicate topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify configuration and scaling. It dynamically detects changes to topics and ensures source and target topic properties are synchronized, including offsets and partitions.
12
+
MirrorMaker 2.0 (MM2) is designed to make it easier to mirror or replicate topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify configuration and scaling. It dynamically detects changes to topics and ensures source and target topic properties are synchronized, including offsets and partitions.
13
13
14
14
In this article, you'll learn how to use Kafka MirrorMaker 2.0 in data migration/replication and the use-cases.
15
15
@@ -27,7 +27,7 @@ Simulation of MirrorMaker 2.0 to replicate data points/offsets between two Kafka
27
27
28
28
### MM2 Internals
29
29
30
-
The Mirrormaker2 tool is composed of different connectors. These connectors are standard Kafka Connect connectors, which can be used directly with Kafka Connect in standalone or distributed mode.
30
+
The MirrorMaker 2.0 tool is composed of different connectors. These connectors are standard Kafka Connect connectors, which can be used directly with Kafka Connect in standalone or distributed mode.
31
31
32
32
The summary of the broker setup process is as follows:
33
33
@@ -51,8 +51,8 @@ The summary of the broker setup process is as follows:
51
51
52
52
### Deployment
53
53
54
-
1. Connect-mirror-maker.sh script bundled with the Kafka library implements a distributed MM2 cluster, which manages the Connect workers internally based on a config file. Internally Mirrormaker driver creates and handles pairs of each connector – MirrorSourceConnector, MirrorSinkConnector, MirrorCheckpoint connector and MirrorHeartbeatConnector.
55
-
1. Start the Mirormaker 2.
54
+
1. Connect-mirror-maker.sh script bundled with the Kafka library implements a distributed MM2 cluster, which manages the Connect workers internally based on a config file. Internally MirrorMaker driver creates and handles pairs of each connector – MirrorSourceConnector, MirrorSinkConnector, MirrorCheckpoint connector and MirrorHeartbeatConnector.
**How will the consumers behave on migration, if that the destination cluster may have a different offset mapping to data points?**
150
150
151
-
Mirrormaker 2’s MirrorCheckpointConnector automatically stores consumer group offset checkpoints for consumer groups on the source cluster. Each checkpoint contains a mapping of the last committed offset for each group in the source cluster to the equivalent offset in destination cluster. So on migration the consumers that start consuming from same topic on the destination cluster will be able to resume receiving messages from the last offset they committed on the source cluster.
151
+
The MirrorMaker 2.0 MirrorCheckpointConnector feature automatically stores consumer group offset checkpoints for consumer groups on the source cluster. Each checkpoint contains a mapping of the last committed offset for each group in the source cluster to the equivalent offset in destination cluster. So on migration the consumers that start consuming from same topic on the destination cluster will be able to resume receiving messages from the last offset they committed on the source cluster.
152
152
153
153
**How can we retain the exact topic name in destination cluster, as the source alias is prefixed with all the topics replicated?**
154
154
155
-
This is the default behavior in Mirrormaker2.0 to avoid data overriding in complex mirroring topologies. Customization of this needs to be done carefully in terms of replication flow design and topic management to avoid data loss. This can be done by using a custom replication policy class against “replication.policy.class”.
155
+
This is the default behavior in MirrorMaker 2.0 to avoid data overriding in complex mirroring topologies. Customization of this needs to be done carefully in terms of replication flow design and topic management to avoid data loss. This can be done by using a custom replication policy class against “replication.policy.class”.
156
156
157
157
**Why do we see new internal topics created in my source and destination Kafka?**
158
158
159
-
Mirrormaker 2 internal topics are created by the Connectors to keep track of the replication process, monitoring, offset mapping and checkpointing.
159
+
MirrorMaker 2.0 internal topics are created by the Connectors to keep track of the replication process, monitoring, offset mapping and checkpointing.
160
160
161
-
**Why does the mirrormaker creates only two replicas of the topic in the destination cluster while the source has more?**
161
+
**Why does MirrorMaker create only two replicas of the topic in the destination cluster while the source has more?**
162
162
163
-
Mirrormaker 2 doesn’t replicate the replication factor of topics to target clusters. This can be controlled from MM2 config, by specifying the required number of “replication.factor”. The default value for the same is two.
163
+
MirrormMker 2 doesn’t replicate the replication factor of topics to target clusters. This can be controlled from MM2 config, by specifying the required number of “replication.factor”. The default value for the same is two.
164
164
165
-
**How to use custom replication policy in Mirrormaker 2?**
165
+
**How to use custom replication policy in MirrorMaker 2.0?**
166
166
167
167
Custom Replication Policy can be created by implementing the interface below.
168
168
@@ -203,7 +203,7 @@ The implementation needs to be added to the Kafka classpath for the class refere
0 commit comments