Skip to content

Commit 0b397ea

Browse files
Merge pull request #266753 from enkrumah/patch-55
Update stream-analytics-define-kafka-input.md
2 parents 426de12 + 0eb793c commit 0b397ea

File tree

2 files changed

+37
-3
lines changed

2 files changed

+37
-3
lines changed

articles/stream-analytics/kafka-output.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,28 @@ author: enkrumah
55
ms.author: ebnkruma
66
ms.service: stream-analytics
77
ms.topic: conceptual
8-
ms.date: 01/18/2024
8+
ms.date: 02/20/2024
99
---
1010

1111
# Kafka output from Azure Stream Analytics (Preview)
1212

1313
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka output is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
1414
Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
1515

16+
## Steps
17+
This article shows how to set up Kafka as an output from Azure Stream Analytics. There are six steps:
18+
19+
1. Create an Azure Stream Analytics job.
20+
2. Configure your Azure Stream Analytics job to use managed identity if you are using mTLS or SASL_SSl security protocols.
21+
3. Configure Azure Key vault if you are using mTLS or SASL_SSl security protocols.
22+
4. Upload certificates as secrets into Azure Key vault.
23+
5. Grant Azure Stream Analytics permissions to access the uploaded certificate.
24+
6. Configure Kafka output in your Azure Stream Analytics job.
25+
26+
> [!NOTE]
27+
> Depending on how your Kafka cluster is configured and the type of Kafka cluster you are using, some of the above steps may not apply to you. Examples are: if you are using confluent cloud Kafka, you will not need to upload a certificate to use the Kafka connector. If your Kafka cluster is inside a virtual network (VNET) or behind a firewall, you may have to configure your Azure Stream Analytics job to access your Kafka topic using a private link or a dedicated networking configuration.
28+
29+
1630
## Configuration
1731
The following table lists the property names and their description for creating a Kafka output:
1832

@@ -26,6 +40,9 @@ The following table lists the property names and their description for creating
2640
| Partition key | Azure Stream Analytics assigns partitions using round partitioning. |
2741
| Kafka event compression type | The compression type used for outgoing data streams, such as Gzip, Snappy, Lz4, Zstd, or None. |
2842

43+
:::image type="content" source="./media/kafka/kafka-output.png" alt-text="Screenshot showing how to configure kafka output for a stream analytics job." lightbox="./media/kafka/kafka-output.png" :::
44+
45+
2946
## Authentication and encryption
3047

3148
You can use four types of security protocols to connect to your Kafka clusters:

articles/stream-analytics/stream-analytics-define-kafka-input.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ author: enkrumah
55
ms.author: ebnkruma
66
ms.service: stream-analytics
77
ms.topic: conceptual
8-
ms.date: 01/18/2024
8+
ms.date: 02/20/2024
99
---
1010

1111
# Stream data from Kafka into Azure Stream Analytics (Preview)
@@ -21,6 +21,21 @@ The following are the major use cases:
2121

2222
Azure Stream Analytics lets you connect directly to Kafka clusters to ingest data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka input is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions. Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
2323

24+
25+
## Steps
26+
This article shows how to set up Kafka as an input source for Azure Stream Analytics. There are six steps:
27+
28+
1. Create an Azure Stream Analytics job.
29+
2. Configure your Azure Stream Analytics job to use managed identity if you are using mTLS or SASL_SSl security protocols.
30+
3. Configure Azure Key vault if you are using mTLS or SASL_SSl security protocols.
31+
4. Upload certificates as secrets into Azure Key vault.
32+
5. Grant Azure Stream Analytics permissions to access the uploaded certificate.
33+
6. Configure Kafka input in your Azure Stream Analytics job.
34+
35+
> [!NOTE]
36+
> Depending on how your Kafka cluster is configured and the type of Kafka cluster you are using, some of the above steps may not apply to you. Examples are: if you are using confluent cloud Kafka, you will not need to upload a certificate to use the Kafka connector. If your Kafka cluster is inside a virtual network (VNET) or behind a firewall, you may have to configure your Azure Stream Analytics job to access your Kafka topic using a private link or a dedicated networking configuration.
37+
38+
2439
## Configuration
2540
The following table lists the property names and their description for creating a Kafka Input:
2641

@@ -36,6 +51,8 @@ The following table lists the property names and their description for creating
3651
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
3752
| Event Serialization format | The serialization format (JSON, CSV, Avro, Parquet, Protobuf) of the incoming data stream. |
3853

54+
:::image type="content" source="./media/kafka/kafka-input.png" alt-text="Screenshot showing how to configure kafka input for a stream analytics job." lightbox="./media/kafka/kafka-input.png" :::
55+
3956

4057
## Authentication and encryption
4158

@@ -157,7 +174,7 @@ Use the following steps to grant special permissions to your stream analytics jo
157174

158175
### VNET integration
159176

160-
If your Kafka is inside a virtual network (VNET) or behind a firewall, you must configure your Azure Stream Analytics job to access your Kafka topic.
177+
If your Kafka cluster is inside a virtual network (VNET) or behind a firewall, you may have to configure your Azure Stream Analytics job to access your Kafka topic using a private link or a dedicated networking configuration.
161178
Visit the [Run your Azure Stream Analytics job in an Azure Virtual Network documentation](../stream-analytics/run-job-in-virtual-network.md) for more information.
162179

163180

0 commit comments

Comments
 (0)