You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: 'Quickstart: Data streaming with Azure Event Hubs using the Kafka protocol'
3
3
description: 'Quickstart: This article provides information on how to stream into Azure Event Hubs using the Kafka protocol and APIs.'
4
4
ms.topic: quickstart
5
-
ms.date: 05/10/2021
5
+
ms.date: 09/22/2022
6
6
ms.custom: mode-other
7
7
---
8
8
@@ -18,108 +18,107 @@ in your applications.
18
18
To complete this quickstart, make sure you have the following prerequisites:
19
19
20
20
* Read through the [Event Hubs for Apache Kafka](event-hubs-for-kafka-ecosystem-overview.md) article.
21
-
* An Azure subscription. If you do not have one, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.
21
+
* An Azure subscription. If you don't have one, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.
22
22
*[Java Development Kit (JDK) 1.7+](/azure/developer/java/fundamentals/java-support-on-azure).
23
23
*[Download](https://maven.apache.org/download.cgi) and [install](https://maven.apache.org/install.html) a Maven binary archive.
24
24
*[Git](https://www.git-scm.com/)
25
25
26
26
27
27
## Create an Event Hubs namespace
28
-
When you create an Event Hubs namespace, the Kafka endpoint for the namespace is automatically enabled. You can stream events from your applications that use the Kafka protocol into event hubs. Follow step-by-step instructions in the [Create an event hub using Azure portal](event-hubs-create.md) to create an Event Hubs namespace. If you are using a dedicated cluster, see [Create a namespace and event hub in a dedicated cluster](event-hubs-dedicated-cluster-create-portal.md#create-a-namespace-and-event-hub-within-a-cluster).
28
+
When you create an Event Hubs namespace, the Kafka endpoint for the namespace is automatically enabled. You can stream events from your applications that use the Kafka protocol into event hubs. Follow step-by-step instructions in the [Create an event hub using Azure portal](event-hubs-create.md) to create an Event Hubs namespace. If you're using a dedicated cluster, see [Create a namespace and event hub in a dedicated cluster](event-hubs-dedicated-cluster-create-portal.md#create-a-namespace-and-event-hub-within-a-cluster).
29
29
30
30
> [!NOTE]
31
31
> Event Hubs for Kafka isn't supported in the **basic** tier.
32
32
33
33
## Send and receive messages with Kafka in Event Hubs
34
34
35
-
1. Clone the [Azure Event Hubs for Kafka repository](https://github.com/Azure/azure-event-hubs-for-kafka).
1. Clone the [Azure Event Hubs for Kafka repository](https://github.com/Azure/azure-event-hubs-for-kafka).
37
38
2. Navigate to `azure-event-hubs-for-kafka/quickstart/java/producer`.
38
-
39
39
3. Update the configuration details for the producer in `src/main/resources/producer.config` as follows:
40
40
41
+
Managed identities for Azure resources provide Azure services with an automatically managed identity in Azure Active Directory. You can use this identity to authenticate to any service that supports Azure AD authentication, without having credentials in your code.
Azure Event Hubs supports using Azure Active Directory (Azure AD) to authorize requests to Event Hubs resources. With Azure AD, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, or an application service principal.
47
-
48
-
If you want to run this sample locally with Azure AD authentication, be sure your user account has authenticated via Azure Toolkit for IntelliJ, Visual Studio Code Azure Account plugin, or Azure CLI. Also, be sure the account has been granted sufficient permissions.
49
-
50
-
> [!NOTE]
51
-
> You need to set the following data plane access roles: `Azure Event Hubs Data Sender` and `Azure Event Hubs Data Receiver`.
52
-
53
-
To authenticate using the Azure CLI, use the following steps.
54
-
55
-
1. First, use the following command to get the resource ID for your Azure Event Hubs namespace:
56
-
57
-
```azurecli
58
-
export AZURE_RESOURCE_ID=$(az resource show \
59
-
--resource-group $AZ_RESOURCE_GROUP \
60
-
--name $AZ_EVENTHUBS_NAMESPACE_NAME \
61
-
--resource-type Microsoft.EventHub/Namespaces \
62
-
--query "id" \
63
-
--output tsv)
64
-
```
65
-
66
-
1. Second, use the following command to get your user object ID of your Azure CLI user account:
67
-
68
-
```azurecli
69
-
export AZURE_ACCOUNT_ID=$(az ad signed-in-user show \
70
-
--query "id" --output tsv)
71
-
```
43
+
Azure Event Hubs supports using Azure Active Directory (Azure AD) to authorize requests to Event Hubs resources. With Azure AD, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, or an application service principal.
44
+
45
+
To use Managed Identity, you can create a (or configure an existing) virtual machine using a system-assigned managed identity. See [Configure managed identities for Azure resources on a VM using the Azure portal](https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm#system-assigned-managed-identity) for more details.
46
+
47
+
Once you configure the virtual machine with managed identity, you need to add managed idendity to Event Hubs namespace. For that you need to follow these steps.
48
+
49
+
- In the Azure Portal, navigate to your Event Hubs namespace. Go to "Access Control (IAM)" in the left navigation.
50
+
51
+
- Click + Add and select `Add role assignment`.
52
+
53
+
- In the Role tab, select `Azure Event Hubs Data Owner` and click the Next button.
54
+
55
+
- In the `Members` tab, select the `Managed Identity` radio button for type to assign access to.
56
+
57
+
- Click the `+Select members` link. In the Managed Identity dropdown, select Virtual Machine and select your virtual machine's managed identity.
58
+
59
+
- Click `Review + Assign`.
72
60
73
-
1. Then, use the following commands to assign the `Azure Event Hubs Data Sender` and `Azure Event Hubs Data Receiver` roles to your account.
61
+
1. Once you configure managed identity you can update `src/main/resources/producer.config` as shown below.
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/producer/src/main/java).
74
72
75
-
```azurecli
76
-
az role assignment create \
77
-
--assignee $AZURE_ACCOUNT_ID \
78
-
--role "Azure Event Hubs Data Receiver" \
79
-
--scope $AZURE_RESOURCE_ID
73
+
4. Run the producer code and stream events into Event Hubs:
80
74
81
-
az role assignment create \
82
-
--assignee $AZURE_ACCOUNT_ID \
83
-
--role "Azure Event Hubs Data Sender" \
84
-
--scope $AZURE_RESOURCE_ID
85
-
```
86
-
87
-
For more information about granting access roles, see [Authorize access to Event Hubs resources using Azure Active Directory](/azure/event-hubs/authorize-access-azure-active-directory).
88
-
89
-
Once your user account is authenticated, you can update use following configuration in `src/main/resources/producer.config` as shown below.
75
+
```shell
76
+
mvn clean package
77
+
mvn exec:java -Dexec.mainClass="TestProducer"
78
+
```
79
+
80
+
5. Navigate to `azure-event-hubs-for-kafka/quickstart/java/consumer`.
6. Update the configuration details for the consumer in `src/main/resources/consumer.config` as follows:
83
+
1. Make sure you configure managed identity as mentioned in step 3 and use the following consumer configuration.
98
84
99
-
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/producer/src/main/java).
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/consumer/src/main/java).
94
+
95
+
You can find all the OAuth samples for Event Hubs for Kafka [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).
96
+
7. Run the consumer code and process events from event hub using your Kafka clients:
100
97
98
+
```java
99
+
mvn clean package
100
+
mvn exec:java -Dexec.mainClass="TestConsumer"
101
+
```
101
102
102
-
---
103
+
If your Event Hubs Kafka cluster has events, you now start receiving them from the consumer.
103
104
104
105
#### [Connection string](#tab/connection-string)
105
106
106
-
**TLS/SSL:**
107
+
1. Clone the [Azure Event Hubs for Kafka repository](https://github.com/Azure/azure-event-hubs-for-kafka).
108
+
2. Navigate to `azure-event-hubs-for-kafka/quickstart/java/producer`.
109
+
3. Update the configuration details for the producer in `src/main/resources/producer.config` as follows:
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
117
-
118
-
---
119
-
120
-
121
-
122
-
118
+
119
+
> [!IMPORTANT]
120
+
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
121
+
123
122
4. Run the producer code and stream events into Event Hubs:
124
123
125
124
```shell
@@ -130,46 +129,16 @@ You can find the source code for the sample handler class CustomAuthenticateCall
130
129
5. Navigate to `azure-event-hubs-for-kafka/quickstart/java/consumer`.
131
130
132
131
6. Update the configuration details for the consumer in `src/main/resources/consumer.config` as follows:
You can find the source code for the sample handler class CustomAuthenticateCallbackHandler on GitHub [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/appsecret/consumer/src/main/java).
149
-
150
-
You can find all the OAuth samples for Event Hubs for Kafka [here](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
> Replace `{YOUR.EVENTHUBS.CONNECTION.STRING}` with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see [Get an Event Hubs connection string](event-hubs-get-connection-string.md). Here's an example configuration: `sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";`
173
142
174
143
7. Run the consumer code and process events from event hub using your Kafka clients:
If your Event Hubs Kafka cluster has events, you now start receiving them from the consumer.
182
151
152
+
---
153
+
154
+
155
+
183
156
## Next steps
184
157
In this article, you learned how to stream into Event Hubs without changing your protocol clients or running your own clusters. To learn more, see [Apache Kafka developer guide for Azure Event Hubs](apache-kafka-developer-guide.md).
0 commit comments