Skip to content

Commit a25b168

Browse files
JamesJBarnettJamesJBarnett
authored
Merge pull request #252788 from enkrumah/kafkainput
added kafka input and output doc
2 parents 2127ad5 + eef3ee5 commit a25b168

File tree

3 files changed

+169
-0
lines changed

3 files changed

+169
-0
lines changed

‎articles/stream-analytics/TOC.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,8 @@
101101
href: stream-analytics-define-inputs.md
102102
- name: Reference data inputs
103103
href: stream-analytics-use-reference-data.md
104+
- name: Kafka input
105+
href: stream-analytics-define-kafka-input.md
104106
- name: Validate input schema
105107
href: input-validation.md
106108
- name: Read inputs in any format using custom deserializers
@@ -150,6 +152,8 @@
150152
href: azure-functions-output.md
151153
- name: Azure Data Lake Storage Gen 1
152154
href: azure-data-lake-storage-gen1-output.md
155+
- name: Kafka output
156+
href: kafka-output.md
153157
- name: Extend queries with functions
154158
items:
155159
- name: Functions overview

‎articles/stream-analytics/kafka-output.md

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
title: Stream data from Azure Stream Analytics into Kafka
3+
description: Learn about setting up Azure Stream Analytics as a producer to kafka
4+
author: enkrumah
5+
ms.author: ebnkruma
6+
ms.service: stream-analytics
7+
ms.topic: conceptual
8+
ms.date: 09/26/2023
9+
---
10+
11+
# Kafka output from Azure Stream Analytics (Preview)
12+
13+
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The Kafka Adapters are backward compatible and support all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
14+
Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
15+
16+
## Authentication and encryption
17+
18+
You can use four types of security protocols to connect to your Kafka clusters:
19+
20+
|Property name |Description |
21+
|----------|-----------|
22+
|mTLS |encryption and authentication |
23+
|SASL_SSL |It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and SSL (Secure Sockets Layer) - to ensure both authentication and encryption are in place for data transmission. |
24+
|Kafka topic |A unit of your Kafka cluster you want to write events to. |
25+
|SASL_PLAINTEXT |standard authentication with username and password without encryption |
26+
|None |The serialization format (JSON, CSV, Avro, Parquet) of the incoming data stream. |
27+
28+
29+
> [!IMPORTANT]
30+
> Confluent Cloud supports authenticating using API Keys, OAuth, or SAML single sign-on (SSO). Azure Stream Analytics does not currently support these authentication options.
31+
>
32+
33+
34+
### Key vault integration
35+
36+
> [!NOTE]
37+
> When using trust store certificates with mTLS or SASL_SSL security protocols, you must have Azure Key vault and managed identity configured for your Azure Stream Analytics job.
38+
>
39+
Azure Stream Analytics integrates seamlessly with Azure Key vault to access stored secrets needed for authentication and encryption when using mTLS or SASL_SSL security protocols. Your Azure Stream Analytics job connects to Azure Key vault using managed identity to ensure a secure connection and avoid the exfiltration of secrets.
40+
41+
You can store the certificates as Key vault certificates or Key vault secrets. Private keys are in PEM format.
42+
43+
### VNET integration
44+
When configuring your Azure Stream Analytics job to connect to your Kafka clusters, depending on your configuration, you may have to configure your job to access your Kafka clusters, which are behind a firewall or inside a virtual network. You can visit the Azure Stream Analytics VNET documentation to learn more about configuring private endpoints to access resources inside a virtual network or behind a firewall.
45+
46+
47+
### Configuration
48+
The following table lists the property names and their description for creating a Kafka output:
49+
50+
| Property name | Description |
51+
|------------------------------|-------------------------------------------------------------------------------------------------------------------------|
52+
| Input/Output Alias | A friendly name used in queries to reference your input or output |
53+
| Bootstrap server addresses | A list of host/port pairs to establish the connection to the Kafka cluster. |
54+
| Kafka topic | A unit of your Kafka cluster you want to write events to. |
55+
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
56+
| Event Serialization format | The serialization format (JSON, CSV, Avro) of the outgoing data stream. |
57+
| Partition key | Azure Stream Analytics assigns partitions using round partitioning. |
58+
| Kafka event compression type | The compression type used for outgoing data streams, such as Gzip, Snappy, Lz4, Zstd, or None. |
59+
60+
### Limitations
61+
* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with at least six (6) streaming units.
62+
* When using mTLS or SASL_SSL with Azure Key vault, you must convert your Java Key Store to PEM format.
63+
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.
64+
65+
> [!NOTE]
66+
> For direct help with using the Azure Stream Analytics Kafka adapter, please reach out to [[email protected]](mailto:[email protected]).
67+
>
68+
69+
## Next steps
70+
> [!div class="nextstepaction"]
71+
> [Quickstart: Create a Stream Analytics job by using the Azure portal](stream-analytics-quick-create-portal.md)
72+
73+
<!--Link references-->
74+
[stream.analytics.developer.guide]: ../stream-analytics-developer-guide.md
75+
[stream.analytics.scale.jobs]: stream-analytics-scale-jobs.md
76+
[stream.analytics.introduction]: stream-analytics-introduction.md
77+
[stream.analytics.get.started]: stream-analytics-real-time-fraud-detection.md
78+
[stream.analytics.query.language.reference]: /stream-analytics-query/stream-analytics-query-language-reference

‎articles/stream-analytics/stream-analytics-define-kafka-input.md

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
title: Stream data from Kafka into Azure Stream Analytics
3+
description: Learn about setting up Azure Stream Analytics as a consumer from Kafka
4+
author: enkrumah
5+
ms.author: ebnkruma
6+
ms.service: stream-analytics
7+
ms.topic: conceptual
8+
ms.date: 09/26/2023
9+
---
10+
11+
# Stream data from Kafka into Azure Stream Analytics (Preview)
12+
13+
Kafka is a distributed streaming platform used to publish and subscribe to streams of records. Kafka is designed to allow your apps to process records as they occur. It is an open-source system developed by the Apache Software Foundation and written in Java and Scala.
14+
15+
The following are the major use cases:
16+
* Messaging
17+
* Website Activity Tracking
18+
* Metrics
19+
* Log Aggregation
20+
* Stream Processing
21+
22+
Azure Stream Analytics lets you connect directly to Kafka clusters to ingest data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The Kafka Adapters are backward compatible and support all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions. Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
23+
24+
## Authentication and encryption
25+
26+
You can use four types of security protocols to connect to your Kafka clusters:
27+
28+
|Property name |Description |
29+
|----------|-----------|
30+
|mTLS |encryption and authentication |
31+
|SASL_SSL |It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and SSL (Secure Sockets Layer) - to ensure both authentication and encryption are in place for data transmission. |
32+
|Kafka topic |A unit of your Kafka cluster you want to write events to. |
33+
|SASL_PLAINTEXT |standard authentication with username and password without encryption |
34+
|None |The serialization format (JSON, CSV, Avro, Parquet) of the incoming data stream. |
35+
36+
37+
> [!IMPORTANT]
38+
> Confluent Cloud supports authenticating using API Keys, OAuth, or SAML single sign-on (SSO). Azure Stream Analytics does not currently support these authentication options.
39+
40+
41+
### Key vault integration
42+
43+
> [!NOTE]
44+
> When using trust store certificates with mTLS or SASL_SSL security protocols, you must have Azure Key vault and managed identity configured for your Azure Stream Analytics job.
45+
>
46+
47+
Azure Stream Analytics integrates seamlessly with Azure Key vault to access stored secrets needed for authentication and encryption when using mTLS or SASL_SSL security protocols. Your Azure Stream Analytics job connects to Azure Key vault using managed identity to ensure a secure connection and avoid the exfiltration of secrets.
48+
49+
You can store the certificates as Key vault certificates or Key vault secrets. Private keys are in PEM format.
50+
51+
### VNET integration
52+
When configuring your Azure Stream Analytics job to connect to your Kafka clusters, depending on your configuration, you may have to configure your job to access your Kafka clusters, which are behind a firewall or inside a virtual network. You can visit the Azure Stream Analytics VNET documentation to learn more about configuring private endpoints to access resources inside a virtual network or behind a firewall.
53+
54+
55+
### Configuration
56+
The following table lists the property names and their description for creating a Kafka Input:
57+
58+
| Property name | Description |
59+
|------------------------------|-------------------------------------------------------------------------------------------------------------------------|
60+
| Input/Output Alias | A friendly name used in queries to reference your input or output |
61+
| Bootstrap server addresses | A list of host/port pairs to establish the connection to the Kafka cluster. |
62+
| Kafka topic | A unit of your Kafka cluster you want to write events to. |
63+
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
64+
| Event Serialization format | The serialization format (JSON, CSV, Avro, Parquet, Protobuf) of the incoming data stream. |
65+
66+
67+
68+
### Limitations
69+
* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with at least six (6) streaming units.
70+
* When using mTLS or SASL_SSL with Azure Key vault, you must convert your Java Key Store to PEM format.
71+
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.
72+
73+
> [!NOTE]
74+
> For direct help with using the Azure Stream Analytics Kafka adapter, please reach out to [[email protected]](mailto:[email protected]).
75+
>
76+
77+
78+
## Next steps
79+
> [!div class="nextstepaction"]
80+
> [Quickstart: Create a Stream Analytics job by using the Azure portal](stream-analytics-quick-create-portal.md)
81+
82+
<!--Link references-->
83+
[stream.analytics.developer.guide]: ../stream-analytics-developer-guide.md
84+
[stream.analytics.scale.jobs]: stream-analytics-scale-jobs.md
85+
[stream.analytics.introduction]: stream-analytics-introduction.md
86+
[stream.analytics.get.started]: stream-analytics-real-time-fraud-detection.md
87+
[stream.analytics.query.language.reference]: /stream-analytics-query/stream-analytics-query-language-reference

0 commit comments

Comments
 (0)