You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: 'Quickstart: Data streaming with Azure Event Hubs using the Kafka protocol'
3
3
description: 'Quickstart: This article provides information on how to stream into Azure Event Hubs using the Kafka protocol and APIs.'
4
4
ms.topic: quickstart
5
-
ms.date: 09/22/2022
5
+
ms.date: 09/26/2022
6
6
ms.custom: mode-other
7
7
---
8
8
9
9
# Quickstart: Data streaming with Event Hubs using the Kafka protocol
10
-
This quickstart shows how to stream into Event Hubs without changing your protocol clients or running your own clusters. You learn how to use your producers and consumers to talk to Event Hubs with just a configuration change
11
-
in your applications.
10
+
11
+
This quickstart shows how to stream into Event Hubs without changing your protocol clients or running your own clusters. You learn how to use your producers and consumers to talk to Event Hubs with just a configuration change in your applications.
12
12
13
13
> [!NOTE]
14
14
> This sample is available on [GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java)
@@ -22,139 +22,141 @@ To complete this quickstart, make sure you have the following prerequisites:
22
22
*[Java Development Kit (JDK) 1.7+](/azure/developer/java/fundamentals/java-support-on-azure).
23
23
*[Download](https://maven.apache.org/download.cgi) and [install](https://maven.apache.org/install.html) a Maven binary archive.
24
24
*[Git](https://www.git-scm.com/)
25
-
* To run this quickstart using managed identity, you need to run it on an Azure virtual machine.
26
-
25
+
* To run this quickstart using managed identity, you need to run it on an Azure virtual machine.
27
26
28
27
## Create an Event Hubs namespace
28
+
29
29
When you create an Event Hubs namespace, the Kafka endpoint for the namespace is automatically enabled. You can stream events from your applications that use the Kafka protocol into event hubs. Follow step-by-step instructions in the [Create an event hub using Azure portal](event-hubs-create.md) to create an Event Hubs namespace. If you're using a dedicated cluster, see [Create a namespace and event hub in a dedicated cluster](event-hubs-dedicated-cluster-create-portal.md#create-a-namespace-and-event-hub-within-a-cluster).
30
30
31
31
> [!NOTE]
32
32
> Event Hubs for Kafka isn't supported in the **basic** tier.
33
33
34
34
## Send and receive messages with Kafka in Event Hubs
1. Managed identities for Azure resources provide Azure services with an automatically managed identity in Azure Active Directory. You can use this identity to authenticate to any service that supports Azure AD authentication, without having credentials in your code.
39
39
40
-
Azure Event Hubs supports using Azure Active Directory (Azure AD) to authorize requests to Event Hubs resources. With Azure AD, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, or an application service principal.
41
-
42
-
To use Managed Identity, you can create a (or configure an existing) virtual machine using a system-assigned managed identity. For more information about configuring managed identity on a VM, see [Configure managed identities for Azure resources on a VM using the Azure portal](https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm#system-assigned-managed-identity).
40
+
Azure Event Hubs supports using Azure Active Directory (Azure AD) to authorize requests to Event Hubs resources. With Azure AD, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, or an application service principal.
41
+
42
+
To use Managed Identity, you can create or configure a virtual machine using a system-assigned managed identity. For more information about configuring managed identity on a VM, see [Configure managed identities for Azure resources on a VM using the Azure portal](../active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm.md#system-assigned-managed-identity).
43
43
44
44
1. In the virtual machine that you configure managed identity, clone the [Azure Event Hubs for Kafka repository](https://github.com/Azure/azure-event-hubs-for-kafka).
45
-
1. Navigate to `azure-event-hubs-for-kafka/quickstart/java/producer`.
46
-
1. Update the configuration details for the producer in `src/main/resources/producer.config` as follows:
47
-
48
-
49
-
50
-
Once you configure the virtual machine with managed identity, you need to add managed identity to Event Hubs namespace. For that you need to follow these steps.
51
-
52
-
- In the Azure portal, navigate to your Event Hubs namespace. Go to "Access Control (IAM)" in the left navigation.
53
-
54
-
- Select + Add and select `Add role assignment`.
55
-
56
-
- In the Role tab, select `Azure Event Hubs Data Owner` and select the Next button.
57
-
58
-
- In the `Members` tab, select the `Managed Identity` radio button for type to assign access to.
59
-
60
-
- Select the `+Select members` link. In the Managed Identity dropdown, select Virtual Machine and select your virtual machine's managed identity.
61
-
62
-
- Select `Review + Assign`.
63
-
64
-
1. Once you configure managed identity, you can update `src/main/resources/producer.config` as shown below.
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/producer/src/main/java).
75
-
76
-
4. Run the producer code and stream events into Event Hubs:
77
-
78
-
```shell
79
-
mvn clean package
80
-
mvn exec:java -Dexec.mainClass="TestProducer"
81
-
```
82
-
83
-
5. Navigate to `azure-event-hubs-for-kafka/quickstart/java/consumer`.
84
-
85
-
6. Update the configuration details for the consumer in `src/main/resources/consumer.config` as follows:
86
-
1. Make sure you configure managed identity as mentioned in step 3 and use the following consumer configuration.
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/consumer/src/main/java).
97
-
98
-
You can find all the OAuth samples for Event Hubs for Kafka [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).
99
-
7. Run the consumer code and process events from event hub using your Kafka clients:
100
-
101
-
```java
102
-
mvn clean package
103
-
mvn exec:java -Dexec.mainClass="TestConsumer"
104
-
```
105
-
106
-
If your Event Hubs Kafka cluster has events, you now start receiving them from the consumer.
107
-
108
-
#### [Connection string](#tab/connection-string)
45
+
46
+
1. Navigate to *azure-event-hubs-for-kafka/quickstart/java/producer*.
47
+
48
+
1. Update the configuration details for the producer in *src/main/resources/producer.config* as follows:
49
+
50
+
After you configure the virtual machine with managed identity, you need to add managed identity to Event Hubs namespace. For that you need to follow these steps.
51
+
52
+
* In the Azure portal, navigate to your Event Hubs namespace. Go to **Access Control (IAM)** in the left navigation.
53
+
54
+
* Select **Add** and select `Add role assignment`.
55
+
56
+
* In the **Role** tab, select **Azure Event Hubs Data Owner**, then select **Next**=.
57
+
58
+
* In the **Members** tab, select the **Managed Identity** radio button for the type to assign access to.
59
+
60
+
* Select the **Select members** link. In the **Managed Identity** dropdown, select **Virtual Machine**, then select your virtual machine's managed identity.
61
+
62
+
* Select **Review + Assign**.
63
+
64
+
1. After you configure managed identity, you can update *src/main/resources/producer.config* as shown below.
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/producer/src/main/java).
75
+
76
+
1. Run the producer code and stream events into Event Hubs:
77
+
78
+
```shell
79
+
mvn clean package
80
+
mvn exec:java -Dexec.mainClass="TestProducer"
81
+
```
82
+
83
+
1. Navigate to *azure-event-hubs-for-kafka/quickstart/java/consumer*.
84
+
85
+
1. Update the configuration details for the consumer in *src/main/resources/consumer.config* as follows:
86
+
87
+
1. Make sure you configure managed identity as mentioned in step 3 and use the following consumer configuration.
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/consumer/src/main/java).
98
+
99
+
You can find all the OAuth samples for Event Hubs for Kafka [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).
100
+
101
+
1. Run the consumer code and process events from event hub using your Kafka clients:
102
+
103
+
```java
104
+
mvn clean package
105
+
mvn exec:java -Dexec.mainClass="TestConsumer"
106
+
```
107
+
108
+
If your Event Hubs Kafka cluster has events, you now start receiving them from the consumer.
109
+
110
+
### [Connection string](#tab/connection-string)
109
111
110
112
1. Clone the [Azure Event Hubs for Kafka repository](https://github.com/Azure/azure-event-hubs-for-kafka).
111
-
2. Navigate to `azure-event-hubs-for-kafka/quickstart/java/producer`.
112
-
3. Update the configuration details for the producer in `src/main/resources/producer.config` as follows:
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
124
-
125
-
4. Run the producer code and stream events into Event Hubs:
126
-
127
-
```shell
128
-
mvn clean package
129
-
mvn exec:java -Dexec.mainClass="TestProducer"
130
-
```
131
-
132
-
5. Navigate to `azure-event-hubs-for-kafka/quickstart/java/consumer`.
133
-
134
-
6. Update the configuration details for the consumer in `src/main/resources/consumer.config` as follows:
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
145
-
146
-
7. Run the consumer code and process events from event hub using your Kafka clients:
147
-
148
-
```java
149
-
mvn clean package
150
-
mvn exec:java -Dexec.mainClass="TestConsumer"
151
-
```
152
-
153
-
If your Event Hubs Kafka cluster has events, you now start receiving them from the consumer.
154
113
155
-
---
114
+
1. Navigate to *azure-event-hubs-for-kafka/quickstart/java/producer*.
115
+
116
+
1. Update the configuration details for the producer in *src/main/resources/producer.config* as follows:
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
156
127
128
+
1. Run the producer code and stream events into Event Hubs:
157
129
130
+
```shell
131
+
mvn clean package
132
+
mvn exec:java -Dexec.mainClass="TestProducer"
133
+
```
134
+
135
+
1. Navigate to *azure-event-hubs-for-kafka/quickstart/java/consumer*.
136
+
137
+
1. Update the configuration details for the consumer in *src/main/resources/consumer.config* as follows:
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
148
+
149
+
1. Run the consumer code and process events from event hub using your Kafka clients:
150
+
151
+
```java
152
+
mvn clean package
153
+
mvn exec:java -Dexec.mainClass="TestConsumer"
154
+
```
155
+
156
+
If your Event Hubs Kafka cluster has events, you will now start receiving them from the consumer.
157
+
158
+
---
158
159
159
160
## Next steps
161
+
160
162
In this article, you learned how to stream into Event Hubs without changing your protocol clients or running your own clusters. To learn more, see [Apache Kafka developer guide for Azure Event Hubs](apache-kafka-developer-guide.md).
0 commit comments