You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/kafka/rest-proxy.md
+26-13Lines changed: 26 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,15 +6,15 @@ ms.author: hrasheed
6
6
ms.reviewer: hrasheed
7
7
ms.service: hdinsight
8
8
ms.topic: conceptual
9
-
ms.date: 12/17/2019
9
+
ms.date: 04/01/2020
10
10
---
11
11
# Interact with Apache Kafka clusters in Azure HDInsight using a REST proxy
12
12
13
13
Kafka REST Proxy enables you to interact with your Kafka cluster via a REST API over HTTP. This means that your Kafka clients can be outside of your virtual network. Additionally, clients can make simple HTTP calls to send and receive messages to the Kafka cluster, instead of relying on Kafka libraries. This tutorial will show you how to create a REST proxy enabled Kafka cluster and provide a sample code that shows how to make calls to REST proxy.
14
14
15
15
## REST API reference
16
16
17
-
For the full specification of operations supported by the Kafka REST API, please see [HDInsight Kafka REST Proxy API Reference](https://docs.microsoft.com/rest/api/hdinsight-kafka-rest-proxy).
17
+
For the full specification of operations supported by the Kafka REST API, see [HDInsight Kafka REST Proxy API Reference](https://docs.microsoft.com/rest/api/hdinsight-kafka-rest-proxy).
18
18
19
19
## Background
20
20
@@ -24,25 +24,32 @@ For the full specification of operations supported by the API, please see [Apach
24
24
25
25
### REST Proxy endpoint
26
26
27
-
Creating an HDInsight Kafka cluster with REST proxy creates a new public endpoint for your cluster, which you can find in your HDInsight cluster “Properties” on the Azure portal.
27
+
Creating an HDInsight Kafka cluster with REST proxy creates a new public endpoint for your cluster, which you can find in your HDInsight cluster "Properties" on the Azure portal.
28
28
29
29
### Security
30
30
31
-
Access to the Kafka REST proxy is managed with Azure Active Directory security groups. When creating the Kafka cluster with the REST proxy enabled, you will provide the Azure Active Directory security group that should have access to the REST endpoint. The Kafka clients (applications) that need access to the REST proxy should be registered to this group by the group owner. The group owner can do this via the Portal or via Powershell.
31
+
Access to the Kafka REST proxy is managed with Azure Active Directory security groups. When creating the Kafka cluster with the REST proxy enabled, you'll provide the Azure Active Directory security group that should have access to the REST endpoint. The Kafka clients (applications) that need access to the REST proxy should be registered to this group by the group owner. The group owner can do this via the Portal or via PowerShell.
32
32
33
-
Before making requests to the REST proxy endpoint, the client application should get an OAuth token to verify membership of the right security group. Please find a [Client application sample](#client-application-sample) below that shows how to get an OAuth token. Once the client application has the OAuth token, they must pass that token in the HTTP request made to the REST proxy.
33
+
Before making requests to the REST proxy endpoint, the client application should get an OAuth token to verify membership of the right security group. Find a [Client application sample](#client-application-sample) below that shows how to get an OAuth token. Once the client application has the OAuth token, they must pass that token in the HTTP request made to the REST proxy.
34
34
35
35
> [!NOTE]
36
36
> See [Manage app and resource access using Azure Active Directory groups](../../active-directory/fundamentals/active-directory-manage-groups.md), to learn more about AAD security groups. For more information on how OAuth tokens work, see [Authorize access to Azure Active Directory web applications using the OAuth 2.0 code grant flow](../../active-directory/develop/v1-protocols-oauth-code.md).
37
37
38
38
## Prerequisites
39
39
40
40
1. Register an application with Azure AD. The client applications that you write to interact with the Kafka REST proxy will use this application's ID and secret to authenticate to Azure.
41
-
1. Create an Azure AD security group and add the application that you have registered with Azure AD to the security group. This security group will be used to control which applications are allowed to interact with the REST proxy. For more information on creating Azure AD groups, see [Create a basic group and add members using Azure Active Directory](../../active-directory/fundamentals/active-directory-groups-create-azure-portal.md).
41
+
42
+
1. Create an Azure AD security group and add the application that you've registered with Azure AD to the security group as a "member" of the group. This security group will be used to control which applications are allowed to interact with the REST proxy. For more information on creating Azure AD groups, see [Create a basic group and add members using Azure Active Directory](../../active-directory/fundamentals/active-directory-groups-create-azure-portal.md).
1. During the Kafka cluster creation workflow, in the “Security + networking” tab, check the “Enable Kafka REST proxy” option.
52
+
1. During the Kafka cluster creation workflow, in the "Security + networking" tab, check the "Enable Kafka REST proxy" option.
46
53
47
54

48
55
@@ -63,18 +70,18 @@ You can use the python code below to interact with the REST proxy on your Kafka
63
70
1. Save the sample code on a machine with Python installed.
64
71
1. Install required python dependencies by executing `pip3 install adal` and `pip install msrestazure`.
65
72
1. Modify the code section *Configure these properties* and update the following properties for your environment:
66
-
1.*Tenant ID* – The Azure tenant where your subscription is.
67
-
1.*Client ID* – The ID for the application that you registered in the security group.
68
-
1.*Client Secret* – The secret for the application that you registered in the security group
69
-
1.*Kafkarest_endpoint* – get this value from the “properties” tab in the cluster overview as described in the [deployment section](#create-a-kafka-cluster-with-rest-proxy-enabled). It should be in the following format – `https://<clustername>-kafkarest.azurehdinsight.net`
70
-
3. From the command line, execute the python file by executing `python <filename.py>`
73
+
1.*Tenant ID* – The Azure tenant where your subscription is.
74
+
1.*Client ID* – The ID for the application that you registered in the security group.
75
+
1.*Client Secret* – The secret for the application that you registered in the security group
76
+
1.*Kafkarest_endpoint* – get this value from the "properties" tab in the cluster overview as described in the [deployment section](#create-a-kafka-cluster-with-rest-proxy-enabled). It should be in the following format – `https://<clustername>-kafkarest.azurehdinsight.net`
77
+
1. From the command line, execute the python file by executing `python <filename.py>`
71
78
72
79
This code does the following:
73
80
74
81
1. Fetches an OAuth token from Azure AD
75
82
1. Shows how to make a request to Kafka REST proxy
76
83
77
-
For more information on getting OAuth tokens in python, see [Python AuthenticationContext class](https://docs.microsoft.com/python/api/adal/adal.authentication_context.authenticationcontext?view=azure-python). You might see a delay while topics that are not created or deleted through the Kafka REST proxy are reflected there. This delay is due to cache refresh.
84
+
For more information on getting OAuth tokens in python, see [Python AuthenticationContext class](https://docs.microsoft.com/python/api/adal/adal.authentication_context.authenticationcontext?view=azure-python). You might see a delay while topics that aren't created or deleted through the Kafka REST proxy are reflected there. This delay is due to cache refresh.
Find below another sample on how to get a token from Azure for REST proxy using a curl command. Notice that we need the `resource=https://hib.azurehdinsight.net` specified while getting a token.
131
+
132
+
```cmd
133
+
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d 'client_id=<clientid>&client_secret=<clientsecret>&grant_type=client_credentials&resource=https://hib.azurehdinsight.net' 'https://login.microsoftonline.com/<tenantid>/oauth2/token'
134
+
```
135
+
123
136
## Next steps
124
137
125
138
*[Kafka REST proxy API reference documents](https://docs.microsoft.com/rest/api/hdinsight-kafka-rest-proxy/)
0 commit comments