You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/event-hubs/apache-kafka-developer-guide.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,15 +1,15 @@
1
1
---
2
2
title: Apache Kafka developer guide for Event Hubs
3
3
description: This article provides links to articles that describe how to integrate your Kafka applications with Azure Event Hubs.
4
-
ms.date: 11/09/2022
4
+
ms.date: 12/18/2024
5
5
ms.topic: article
6
6
---
7
7
8
8
# Apache Kafka developer guide for Azure Event Hubs
9
9
This article provides links to articles that describe how to integrate your Apache Kafka applications with Azure Event Hubs.
10
10
11
11
## Overview
12
-
Event Hubs provides a Kafka endpoint that can be used by your existing Kafka based applications as an alternative to running your own Kafka cluster. Event Hubs works with many of your existing Kafka applications. For more information, see [Event Hubs for Apache Kafka](azure-event-hubs-apache-kafka-overview.md)
12
+
Event Hubs provides a Kafka endpoint that your existing Kafka based applications can use as an alternative to running your own Kafka cluster. Event Hubs works with many of your existing Kafka applications. For more information, see [Event Hubs for Apache Kafka](azure-event-hubs-apache-kafka-overview.md)
13
13
14
14
## Quickstarts
15
15
You can find quickstarts in GitHub and in this content set that helps you quickly ramp up on Event Hubs for Kafka.
@@ -19,13 +19,13 @@ See the following quickstarts in the **azure-event-hubs-for-kafka** repo:
19
19
20
20
| Client language/framework | Description |
21
21
| ------------------------- | ----------- |
22
-
|[.NET](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/dotnet)| <p>This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in C# using .NET Core 2.0.</p><p>This sample is based on [Confluent's Apache Kafka .NET client](https://github.com/confluentinc/confluent-kafka-dotnet), modified for use with Event Hubs for Kafka.</p> |
23
-
|[Java](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java)| This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Java. |
24
-
|[Node.js](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/node)| <p>This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Node.</p><p>This sample uses the [node-rdkafka](https://github.com/Blizzard/node-rdkafka) library. </p>|
25
-
|[Python](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/python)| <p>This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in python.</p><p>This sample is based on [Confluent's Apache Kafka Python client](https://github.com/confluentinc/confluent-kafka-python), modified for use with Event Hubs for Kafka.</p>|
26
-
|[Go](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/go)| <p>This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Go.</p><p>This sample is based on [Confluent's Apache Kafka Golang client](https://github.com/confluentinc/confluent-kafka-go), modified for use with Event Hubs for Kafka.</p>|
27
-
|[Sarama kafka Go](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/go-sarama-client)| This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Go using the [Sarama Kafka client](https://github.com/Shopify/sarama) library. |
28
-
|[Kafka](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/kafka-cli)| This quickstart will show how to create and connect to an Event Hubs Kafka endpoint using the CLI that comes bundled with the Apache Kafka distribution.|
22
+
|[.NET](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/dotnet)| <p>This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in C# using .NET Core 2.0.</p><p>This sample is based on [Confluent's Apache Kafka .NET client](https://github.com/confluentinc/confluent-kafka-dotnet), modified for use with Event Hubs for Kafka.</p> |
23
+
|[Java](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java)| This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Java. |
24
+
|[Node.js](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/node)| <p>This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Node.</p><p>This sample uses the [node-rdkafka](https://github.com/Blizzard/node-rdkafka) library. </p>|
25
+
|[Python](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/python)| <p>This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in python.</p><p>This sample is based on [Confluent's Apache Kafka Python client](https://github.com/confluentinc/confluent-kafka-python), modified for use with Event Hubs for Kafka.</p>|
26
+
|[Go](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/go)| <p>This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Go.</p><p>This sample is based on [Confluent's Apache Kafka Golang client](https://github.com/confluentinc/confluent-kafka-go), modified for use with Event Hubs for Kafka.</p>|
27
+
|[Sarama kafka Go](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/go-sarama-client)| This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Go using the [Sarama Kafka client](https://github.com/Shopify/sarama) library. |
28
+
|[Kafka](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/kafka-cli)| This quickstart shows how to create and connect to an Event Hubs Kafka endpoint using the CLI that comes bundled with the Apache Kafka distribution.|
29
29
|[kcat](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/kafkacat)| kcat is a non-JVM command-line consumer and producer based on librdkafka, popular due to its speed and small footprint. This quickstart contains a sample configuration and several simple sample kafkacat commands. |
30
30
31
31
### Quickstarts in DOCS
@@ -40,17 +40,17 @@ See the following tutorials on GitHub:
40
40
| Tutorial | Description |
41
41
| ------------------------- | ----------- |
42
42
|[Akka](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/akka/java)| This tutorial shows how to connect Akka Streams to Kafka-enabled Event Hubs without changing your protocol clients or running your own clusters. There are two separate tutorials using **Java** and **Scala** programming languages. |
43
-
|[Connect](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/connect)| This document will walk you through integrating Kafka Connect with Azure Event Hubs and deploying basic FileStreamSource and FileStreamSink connectors. While these connectors are not meant for production use, they demonstrate an end-to-end Kafka Connect Scenario where Azure Event Hubs masquerades as a Kafka broker.|
44
-
|[Filebeat](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/filebeat)| This document will walk you through integrating Filebeat and Event Hubs via Filebeat's Kafka output. |
45
-
|[Flink](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/flink)| This tutorial will show how to connect Apache Flink to Kafka-enabled Event Hubs without changing your protocol clients or running your own clusters. |
46
-
|[FluentD](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/fluentd)| This document will walk you through integrating Fluentd and Event Hubs using the `out_kafka` output plugin for Fluentd. |
43
+
|[Connect](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/connect)| This document walks you through integrating Kafka Connect with Azure Event Hubs and deploying basic FileStreamSource and FileStreamSink connectors. While these connectors aren't meant for production use, they demonstrate an end-to-end Kafka Connect Scenario where Azure Event Hubs masquerades as a Kafka broker.|
44
+
|[Filebeat](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/filebeat)| This document walks you through integrating Filebeat and Event Hubs via Filebeat's Kafka output. |
45
+
|[Flink](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/flink)| This tutorial shows how to connect Apache Flink to Kafka-enabled Event Hubs without changing your protocol clients or running your own clusters. |
46
+
|[FluentD](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/fluentd)| This document walks you through integrating Fluentd and Event Hubs using the `out_kafka` output plugin for Fluentd. |
47
47
|[Interop](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/interop)| This tutorial shows you how to exchange events between consumers and producers using different protocols. |
48
-
|[Logstash](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/logstash)| This tutorial will walk you through integrating Logstash with Kafka-enabled Event Hubs using Logstash Kafka input/output plugins. |
48
+
|[Logstash](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/logstash)| This tutorial walks you through integrating Logstash with Kafka-enabled Event Hubs using Logstash Kafka input/output plugins. |
49
49
|[MirrorMaker](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/mirror-maker)| This tutorial shows how an event hub and Kafka MirrorMaker can integrate an existing Kafka pipeline into Azure by mirroring the Kafka input stream in the Event Hubs service. |
50
-
|[NiFi](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/nifi)| This tutorial will show how to connect Apache NiFi to an Event Hubs namespace. |
50
+
|[NiFi](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/nifi)| This tutorial shows how to connect Apache NiFi to an Event Hubs namespace. |
51
51
|[OAuth](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth)| Quickstarts show you how to create and connect to an Event Hubs Kafka endpoint using an example producer and consumer written in Go and Java programming languages. |
52
-
|[Confluent's Schema Registry](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/schema-registry)| This tutorial will walk you through integrating Schema Registry and Event Hubs for Kafka. |
53
-
|[Spark](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/spark)| This tutorial will show how to connect your Spark application to an event hub without changing your protocol clients or running your own Kafka clusters. |
52
+
|[Confluent's Schema Registry](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/schema-registry)| This tutorial walks you through integrating Schema Registry and Event Hubs for Kafka. |
53
+
|[Spark](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/spark)| This tutorial shows how to connect your Spark application to an event hub without changing your protocol clients or running your own Kafka clusters. |
54
54
55
55
### Tutorials in DOCS
56
56
Also, see the tutorial: [Process Apache Kafka for Event Hubs events using Stream analytics](event-hubs-kafka-stream-analytics.md) in this content set, which shows how to stream data into Event Hubs and process it with Azure Stream Analytics.
Copy file name to clipboardExpand all lines: articles/event-hubs/apache-kafka-migration-guide.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
title: Migrate to Azure Event Hubs for Apache Kafka
3
3
description: This article explains how to migrate clients from Apache Kafka to Azure Event Hubs.
4
4
ms.topic: article
5
-
ms.date: 09/20/2021
5
+
ms.date: 12/18/2024
6
6
---
7
7
8
8
# Migrate to Azure Event Hubs for Apache Kafka Ecosystems
@@ -14,23 +14,23 @@ Azure Event Hubs exposes an Apache Kafka endpoint, which enables you to connect
14
14
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.
15
15
16
16
### Create an Event Hubs namespace
17
-
Follow step-by-step instructions in the [Create an event hub](event-hubs-create.md) article to create an Event Hubs namespace and an event hub.
17
+
To create an Event Hubs namespace and an event hub, Follow step-by-step instructions in the [Create an event hub](event-hubs-create.md) article.
18
18
19
19
### Connection string
20
-
Follow steps from the [Get connection string from the portal](event-hubs-get-connection-string.md#azure-portal) article. And, note down the connection string for later use.
20
+
Follow steps from the article: [Get connection string from the portal](event-hubs-get-connection-string.md#azure-portal). And, note down the connection string for later use.
21
21
22
22
### Fully qualified domain name (FQDN)
23
-
You may also need the FQDN that points to your Event Hub namespace. The FQDN can be found within your connection string as follows:
23
+
You might also need the FQDN that points to your Event Hubs namespace. The FQDN can be found within your connection string as follows:
If your Event Hubs namespace is deployed on a non-public cloud, your domain name may differ (for example, \*.servicebus.chinacloudapi.cn, \*.servicebus.usgovcloudapi.net, or \*.servicebus.cloudapi.de).
27
+
If your Event Hubs namespace is deployed on a non-public cloud, your domain name might differ (for example, \*.servicebus.chinacloudapi.cn, \*.servicebus.usgovcloudapi.net, or \*.servicebus.cloudapi.de).
28
28
29
29
## Migration
30
30
31
31
### Update your Kafka client configuration
32
32
33
-
To connect to a Kafka-enabled Event Hub, you'll need to update the Kafka client configurations. If you're having trouble finding yours, try searching for where `bootstrap.servers` is set in your application.
33
+
To connect to a Kafka-enabled event hub, you need to update the Kafka client configurations. If you're having trouble finding yours, try searching for where `bootstrap.servers` is set in your application.
34
34
35
35
Insert the following configs wherever makes sense in your application. Make sure to update the `bootstrap.servers` and `sasl.jaas.config` values to direct the client to your Event Hubs Kafka endpoint with the correct authentication.
36
36
@@ -42,10 +42,10 @@ sasl.mechanism=PLAIN
42
42
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{CONNECTION STRING TO YOUR NAMESPACE}";
43
43
```
44
44
45
-
If `sasl.jaas.config` isn't a supported configuration in your framework, find the configurations that are used to set the SASL username and password and use them instead. Set the username to `$ConnectionString` and the password to your Event Hubs connection string.
45
+
If `sasl.jaas.config` isn't a supported configuration in your framework, find the configurations that are used to set the Simple Authentication and Security Layer (SASL) username and password and use them instead. Set the username to `$ConnectionString` and the password to your Event Hubs connection string.
46
46
47
47
## Post-migration
48
-
Run your Kafka application that sends events to the event hub. Then, verify that the event hub receives the events using the Azure portal. On the **Overview** page of your Event Hubs namespace, switch to the **Messages** view in the **Metrics** section. Refresh the page to update the chart. It may take a few seconds for it to show that the messages have been received.
48
+
Run your Kafka application that sends events to the event hub. Then, verify that the event hub receives the events using the Azure portal. On the **Overview** page of your Event Hubs namespace, switch to the **Messages** view in the **Metrics** section. Refresh the page to update the chart. It might take a few seconds for it to show that the messages are received.
49
49
50
50
[](./media/getstarted-dotnet-standard-send-v2/verify-messages-portal.png#lightbox)
Copy file name to clipboardExpand all lines: articles/event-hubs/event-hubs-kafka-akka-streams-tutorial.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
title: Using Akka Streams for Apache Kafka - Azure Event Hubs| Microsoft Docs
3
3
description: This article provides information on how to connect Akka Streams to an Azure event hub.
4
4
ms.topic: how-to
5
-
ms.date: 09/23/2021
5
+
ms.date: 12/18/2024
6
6
ms.devlang: java
7
7
---
8
8
@@ -25,7 +25,7 @@ In this tutorial, you learn how to:
25
25
To complete this tutorial, make sure you have the following prerequisites:
26
26
27
27
* Read through the [Event Hubs for Apache Kafka](azure-event-hubs-apache-kafka-overview.md) article.
28
-
* An Azure subscription. If you do not have one, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.
28
+
* An Azure subscription. If you don't have one, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.
29
29
*[Java Development Kit (JDK) 1.8+](/azure/developer/java/fundamentals/java-support-on-azure)
30
30
* On Ubuntu, run `apt-get install default-jdk` to install the JDK.
31
31
* Be sure to set the JAVA_HOME environment variable to point to the folder where the JDK is installed.
@@ -40,7 +40,7 @@ An Event Hubs namespace is required to send or receive from any Event Hubs servi
40
40
41
41
## Clone the example project
42
42
43
-
Now that you have a Event Hubs connection string, clone the Azure Event Hubs for Kafka repository and navigate to the `akka` subfolder:
43
+
Now that you have an Event Hubs connection string, clone the Azure Event Hubs for Kafka repository and navigate to the `akka` subfolder:
To run the producer from the command line, generate the JAR and then run from within Maven (or generate the JAR using Maven, then run in Java by adding the necessary Kafka JAR(s) to the classpath):
82
+
To run the producer from the command line, generate the JAR and then run from within Maven (or generate the JAR using Maven, then run in Java by adding the necessary Kafka Java Archive files (JARs) to the classpath):
83
83
84
84
```shell
85
85
mvn clean package
@@ -123,7 +123,7 @@ akka.kafka.consumer {
123
123
124
124
### Run consumer from the command line
125
125
126
-
To run the consumer from the command line, generate the JAR and then run from within Maven (or generate the JAR using Maven, then run in Java by adding the necessary Kafka JAR(s) to the classpath):
126
+
To run the consumer from the command line, generate the JAR and then run from within Maven (or generate the JAR using Maven, then run in Java by adding the necessary Kafka JARs to the classpath):
0 commit comments