|
| 1 | +--- |
| 2 | +title: Use Apache Kafka MirrorMaker 2 - Azure Event Hubs | Microsoft Docs |
| 3 | +description: This article provides information on how to use Kafka MirrorMaker 2 to replicate a Kafka cluster in Azure Event Hubs. |
| 4 | +ms.topic: how-to |
| 5 | +ms.date: 12/04/2022 |
| 6 | +--- |
| 7 | + |
| 8 | +# Replicate data from a Kafka cluster to Event Hubs using Apache Kafka Mirror Maker 2 |
| 9 | + |
| 10 | +This tutorial shows how to replicate data from an existing Kafka cluster to Azure Event Hubs using Mirror Maker 2. |
| 11 | + |
| 12 | +  |
| 13 | + |
| 14 | +> [!NOTE] |
| 15 | +> This sample is available on [GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/mirror-maker-2) |
| 16 | +
|
| 17 | + |
| 18 | +In this tutorial, you learn how to: |
| 19 | +> [!div class="checklist"] |
| 20 | +> * Create an Event Hubs namespace |
| 21 | +> * Set up or use an existing Kafka cluster |
| 22 | +> * Configure Kafka Mirror Maker 2 |
| 23 | +> * Run Kafka Mirror Maker 2 |
| 24 | +
|
| 25 | +## Introduction |
| 26 | +Apache Kafka MirrorMaker 2.0 (MM2) is designed to make it easier to mirror or replicate topics from one Kafka cluster to another. Mirror Maker uses the Kafka Connect framework to simplify configuration and scaling. For more detailed information on Kafka MirrorMaker, see the [Kafka Mirroring/MirrorMaker guide](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330). |
| 27 | + |
| 28 | +As Azure Event Hubs is compatible with Apache Kafka protocol, you can use Mirror Maker 2 to replicate data between an existing Kafka cluster and an Event Hubs namespace. |
| 29 | + |
| 30 | +Mirror Maker 2 dynamically detects changes to topics and ensures source and target topic properties are synchronized, including offsets and partitions. It can be used to replicated data bi-directionally between Kafka cluster and Event Hubs namespace. |
| 31 | + |
| 32 | +## Prerequisites |
| 33 | + |
| 34 | +To complete this tutorial, make sure you have: |
| 35 | + |
| 36 | +* Read through the [Event Hubs for Apache Kafka](event-hubs-for-kafka-ecosystem-overview.md) article. |
| 37 | +* An Azure subscription. If you don't have one, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin. |
| 38 | +* [Java Development Kit (JDK) 1.7+](/azure/developer/java/fundamentals/java-support-on-azure) |
| 39 | + * On Ubuntu, run `apt-get install default-jdk` to install the JDK. |
| 40 | + * Be sure to set the JAVA_HOME environment variable to point to the folder where the JDK is installed. |
| 41 | +* [Download](https://maven.apache.org/download.cgi) and [install](https://maven.apache.org/install.html) a Maven binary archive |
| 42 | + * On Ubuntu, you can run `apt-get install maven` to install Maven. |
| 43 | +* [Git](https://www.git-scm.com/downloads) |
| 44 | + * On Ubuntu, you can run `sudo apt-get install git` to install Git. |
| 45 | +* [Apache Kafka distribution](https://kafka.apache.org/downloads) |
| 46 | + * Download the preferred Apache Kafka distribution (which should contain the Mirror Maker 2 distribution.) |
| 47 | + |
| 48 | +> [!NOTE] |
| 49 | +> Apache Kafka Mirror Maker 2 requires log compaction support which is currently available only in Premium and Dedicated SKUs of Azure Event Hubs. Therefore to replicate data using Mirror Maker 2, you need to use either Premium of Dedicated SKU. |
| 50 | +
|
| 51 | +> [!WARNING] |
| 52 | +> Use of the Apache Mirror Maker 2 **not eligible for product support through Microsoft Azure**. |
| 53 | +> |
| 54 | +
|
| 55 | +## Create an Event Hubs namespace |
| 56 | + |
| 57 | +An Event Hubs namespace is required to send and receive from any Event Hubs service. See [Creating an event hub](event-hubs-create.md) for instructions to create a namespace and an event hub. Make sure to copy the Event Hubs connection string for later use. |
| 58 | + |
| 59 | + |
| 60 | + |
| 61 | +## Clone the example project |
| 62 | +Now that you have an Event Hubs connection string, clone the Azure Event Hubs for Kafka repository and navigate to the `mirror-maker-2` subfolder: |
| 63 | + |
| 64 | +```shell |
| 65 | +git clone https://github.com/Azure/azure-event-hubs-for-kafka.git |
| 66 | +cd azure-event-hubs-for-kafka/tutorials/mirror-maker-2 |
| 67 | +``` |
| 68 | + |
| 69 | +## Set up or use an existing Kafka cluster |
| 70 | + |
| 71 | +If you don't have an existing Kafka cluster, use the [Kafka quickstart guide](https://kafka.apache.org/quickstart) to set up a Kafka cluster with the desired settings (or use an existing Kafka cluster). For testing purposes, you can also create a couple of topics in the newly created Kafka cluster and publish data to them. |
| 72 | + |
| 73 | +If you already have an existing Kafka cluster on-premises or in a managed Kafka cloud service, then you can use it to replicate existing data to Event Hubs. |
| 74 | + |
| 75 | +## Configure Kafka Mirror Maker 2 |
| 76 | + |
| 77 | +Apache Kafka distribution comes with `connect-mirror-maker.sh` script that is bundled with the Kafka library that implements a distributed Mirror Maker 2 cluster. It manages the Connect workers internally based on a configuration file. Internally MirrorMaker driver creates and handles pairs of each connector – *MirrorSource Connector*, *MirrorSink Connector*, *MirrorCheckpoint Connector* and *MirrorHeartbeat Connector*. |
| 78 | + |
| 79 | +1. To configure Mirror Maker 2 to replicate data, you need to update Mirror Maker 2 configuration file `kafka-to-eh-connect-mirror-maker.properties` to define the replication topology. |
| 80 | +1. In the `kafka-to-eh-connect-mirror-maker.properties` config file, define cluster aliases that you plan to use for your Kafka cluster(source) and Event Hubs (destination). |
| 81 | + |
| 82 | + ```config |
| 83 | + # cluster aliases |
| 84 | + clusters = source, destination |
| 85 | + ``` |
| 86 | + |
| 87 | +1. Then specify the connection information for your source, which is your Kafka cluster. |
| 88 | + ```config |
| 89 | + source.bootstrap.servers = your-kafka-cluster-hostname:9092 |
| 90 | + #source.security.protocol=SASL_SSL |
| 91 | + #source.sasl.mechanism=PLAIN |
| 92 | + #source.sasl.jaas.config=<replace sasl jaas config of your Kafka cluster>; |
| 93 | + ``` |
| 94 | + |
| 95 | +1. Specify connection information for destination, which is the Event Hubs namespace that you created. |
| 96 | + ```config |
| 97 | + destination.bootstrap.servers = <your-enventhubs-namespace>.servicebus.windows.net:9093 |
| 98 | + destination.security.protocol=SASL_SSL |
| 99 | + destination.sasl.mechanism=PLAIN |
| 100 | + destination.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='$ConnectionString' password='<Your Event Hubs namespace connection string.>'; |
| 101 | + ``` |
| 102 | + |
| 103 | +1. Enable replication flow from source Kafka cluster to destination Event Hubs namespace. |
| 104 | + ```config |
| 105 | + source->destination.enabled = true |
| 106 | + source->destination.topics = .* |
| 107 | + ``` |
| 108 | + |
| 109 | +1. Update the replication factor of the remote topics and internal topics that Mirror Maker creates at the destination. |
| 110 | + ```config |
| 111 | + replication.factor=3 |
| 112 | + |
| 113 | + checkpoints.topic.replication.factor=3 |
| 114 | + heartbeats.topic.replication.factor=3 |
| 115 | + offset-syncs.topic.replication.factor=3 |
| 116 | +
|
| 117 | + offset.storage.replication.factor=3 |
| 118 | + status.storage.replication.factor=3 |
| 119 | + config.storage.replication.factor=3 |
| 120 | + ``` |
| 121 | + |
| 122 | +1. Then you copy `kafka-to-eh-connect-mirror-maker.properties` configuration file to the Kafka distribution's config directory and can run the Mirror Maker 2 script using the following command. |
| 123 | + ```bash |
| 124 | + ./bin/connect-mirror-maker.sh ./config/kafka-to-eh-connect-mirror-maker.properties |
| 125 | + ``` |
| 126 | +1. Upon the successful execution of the script, you should see the Kafka topics and events getting replicated to your Event Hubs namespace. |
| 127 | +1. To verify that events are making it to the Kafka-enabled Event Hubs, check out the ingress statistics in the [Azure portal](https://azure.microsoft.com/features/azure-portal/), or run a consumer against the Event Hubs. |
| 128 | + |
| 129 | + |
| 130 | +## Samples |
| 131 | +See the following samples on GitHub: |
| 132 | + |
| 133 | +- [Sample code for this tutorial on GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/mirror-maker-2) |
| 134 | + |
| 135 | +- If you are hosting Apache Kafka on Kubernetes using the CNCF Strimzi operator, you can use [Strimzi Mirror Maker 2 sample for Event Hubs](https://strimzi.io/blog/2020/06/09/mirror-maker-2-eventhub). |
| 136 | + |
| 137 | + |
| 138 | +## Next steps |
| 139 | +To learn more about Event Hubs for Kafka, see the following articles: |
| 140 | + |
| 141 | +- [Explore samples on our GitHub](https://github.com/Azure/azure-event-hubs-for-kafka) |
0 commit comments