You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/stream-analytics/kafka-output.md
+18-1Lines changed: 18 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,14 +5,28 @@ author: enkrumah
5
5
ms.author: ebnkruma
6
6
ms.service: stream-analytics
7
7
ms.topic: conceptual
8
-
ms.date: 01/18/2024
8
+
ms.date: 02/20/2024
9
9
---
10
10
11
11
# Kafka output from Azure Stream Analytics (Preview)
12
12
13
13
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka output is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
14
14
Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
15
15
16
+
## Steps
17
+
This article shows how to set up Kafka as an output from Azure Stream Analytics. There are six steps:
18
+
19
+
1. Create an Azure Stream Analytics job.
20
+
2. Configure your Azure Stream Analytics job to use managed identity if you are using mTLS or SASL_SSl security protocols.
21
+
3. Configure Azure Key vault if you are using mTLS or SASL_SSl security protocols.
22
+
4. Upload certificates as secrets into Azure Key vault.
23
+
5. Grant Azure Stream Analytics permissions to access the uploaded certificate.
24
+
6. Configure Kafka output in your Azure Stream Analytics job.
25
+
26
+
> [!NOTE]
27
+
> Depending on how your Kafka cluster is configured and the type of Kafka cluster you are using, some of the above steps may not apply to you. Examples are: if you are using confluent cloud Kafka, you will not need to upload a certificate to use the Kafka connector. If your Kafka cluster is inside a virtual network (VNET) or behind a firewall, you may have to configure your Azure Stream Analytics job to access your Kafka topic using a private link or a dedicated networking configuration.
28
+
29
+
16
30
## Configuration
17
31
The following table lists the property names and their description for creating a Kafka output:
18
32
@@ -26,6 +40,9 @@ The following table lists the property names and their description for creating
| Kafka event compression type | The compression type used for outgoing data streams, such as Gzip, Snappy, Lz4, Zstd, or None. |
28
42
43
+
:::image type="content" source="./media/kafka/kafka-output.png" alt-text="Screenshot showing how to configure kafka output for a stream analytics job." lightbox="./media/kafka/kafka-output.png" :::
44
+
45
+
29
46
## Authentication and encryption
30
47
31
48
You can use four types of security protocols to connect to your Kafka clusters:
Copy file name to clipboardExpand all lines: articles/stream-analytics/stream-analytics-define-kafka-input.md
+19-2Lines changed: 19 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@ author: enkrumah
5
5
ms.author: ebnkruma
6
6
ms.service: stream-analytics
7
7
ms.topic: conceptual
8
-
ms.date: 01/18/2024
8
+
ms.date: 02/20/2024
9
9
---
10
10
11
11
# Stream data from Kafka into Azure Stream Analytics (Preview)
@@ -21,6 +21,21 @@ The following are the major use cases:
21
21
22
22
Azure Stream Analytics lets you connect directly to Kafka clusters to ingest data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka input is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions. Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
23
23
24
+
25
+
## Steps
26
+
This article shows how to set up Kafka as an input source for Azure Stream Analytics. There are six steps:
27
+
28
+
1. Create an Azure Stream Analytics job.
29
+
2. Configure your Azure Stream Analytics job to use managed identity if you are using mTLS or SASL_SSl security protocols.
30
+
3. Configure Azure Key vault if you are using mTLS or SASL_SSl security protocols.
31
+
4. Upload certificates as secrets into Azure Key vault.
32
+
5. Grant Azure Stream Analytics permissions to access the uploaded certificate.
33
+
6. Configure Kafka input in your Azure Stream Analytics job.
34
+
35
+
> [!NOTE]
36
+
> Depending on how your Kafka cluster is configured and the type of Kafka cluster you are using, some of the above steps may not apply to you. Examples are: if you are using confluent cloud Kafka, you will not need to upload a certificate to use the Kafka connector. If your Kafka cluster is inside a virtual network (VNET) or behind a firewall, you may have to configure your Azure Stream Analytics job to access your Kafka topic using a private link or a dedicated networking configuration.
37
+
38
+
24
39
## Configuration
25
40
The following table lists the property names and their description for creating a Kafka Input:
26
41
@@ -36,6 +51,8 @@ The following table lists the property names and their description for creating
36
51
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
37
52
| Event Serialization format | The serialization format (JSON, CSV, Avro, Parquet, Protobuf) of the incoming data stream. |
38
53
54
+
:::image type="content" source="./media/kafka/kafka-input.png" alt-text="Screenshot showing how to configure kafka input for a stream analytics job." lightbox="./media/kafka/kafka-input.png" :::
55
+
39
56
40
57
## Authentication and encryption
41
58
@@ -157,7 +174,7 @@ Use the following steps to grant special permissions to your stream analytics jo
157
174
158
175
### VNET integration
159
176
160
-
If your Kafka is inside a virtual network (VNET) or behind a firewall, you must configure your Azure Stream Analytics job to access your Kafka topic.
177
+
If your Kafka cluster is inside a virtual network (VNET) or behind a firewall, you may have to configure your Azure Stream Analytics job to access your Kafka topic using a private link or a dedicated networking configuration.
161
178
Visit the [Run your Azure Stream Analytics job in an Azure Virtual Network documentation](../stream-analytics/run-job-in-virtual-network.md) for more information.
0 commit comments