Skip to content

Commit e87f83d

Browse files
enkrumahenkrumah
committed
added kafka input and output doc
1 parent 86effcd commit e87f83d

File tree

3 files changed

+173
-0
lines changed

3 files changed

+173
-0
lines changed

‎articles/stream-analytics/TOC.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,8 @@
101101
href: stream-analytics-define-inputs.md
102102
- name: Reference data inputs
103103
href: stream-analytics-use-reference-data.md
104+
- name: Kafka input
105+
href: stream-analytics-define-kafka-input.md
104106
- name: Validate input schema
105107
href: input-validation.md
106108
- name: Read inputs in any format using custom deserializers
@@ -150,6 +152,8 @@
150152
href: azure-functions-output.md
151153
- name: Azure Data Lake Storage Gen 1
152154
href: azure-data-lake-storage-gen1-output.md
155+
- name: Kafka Output
156+
href: kafka-output.md
153157
- name: Extend queries with functions
154158
items:
155159
- name: Functions overview

‎articles/stream-analytics/kafka-output.md

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
---
2+
title: Stream data from Azure Stream Analytics into Kafka
3+
description: Learn about setting up Azure Stream Analytics as a producer to kafka
4+
author: enkrumah
5+
ms.author: ebnkruma
6+
ms.service: stream-analytics
7+
ms.topic: conceptual
8+
ms.date: 09/25/2023
9+
---
10+
11+
# Kafka Output from Azure Stream Analytics (Preview)
12+
13+
Azure Stream Analytics allows you to connect directly to kafka clusters as a producer to output data.The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The Kafka Adapters are backward compatible and support all Kafka versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint depending on the configurations. The configuration relies on existing Kafka configuration conventions.
14+
15+
### Kafka Event Compression
16+
17+
Supported compression types are: None, Gzip, Snappy, LZ4 and Zstd compression.
18+
19+
## Authentication and Encryption
20+
21+
You can use four types of security protocols to connect to your Kafka clusters:
22+
| Property name | Description |
23+
|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
24+
| mTLS | encryption and authentication |
25+
| SASL_SSL | It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and SSL (Secure Sockets Layer) - to ensure both authentication and encryption are in place for data transmission. |
26+
| Kafka topic | A unit of your Kafka cluster you want to write events to. |
27+
| SASL_PLAINTEXT | standard authentication with username and password without encryption |
28+
| None | The serialization format (JSON, CSV, Avro, Parquet) of the incoming data stream. |
29+
30+
31+
> [!IMPORTANT]
32+
> Confluent Cloud supports authenticating using API Keys, OAuth, or SAML single sign-on (SSO). Azure Stream Analytics do not currently support these authentication options. There will be a update in the near future to supporting authenticating to confluent cloud using API Keys.
33+
>
34+
35+
36+
### Key Vault Integration
37+
38+
> [!NOTE]
39+
> When using truststore certificates with mTLS or SASL_SSL security protocols, you must have Azure KeyVault and managed identity configured for your Azure Stream Analytics job.
40+
>
41+
Azure Stream Analytics integrates seamlessly with Azure Key Vault to access stored secrets needed for authentication and encryption when using mTLS or SASL_SSL security protocols. Your Azure Stream Analytics job connects to Azure Key Vault using managed identity to ensure secure connection and avoid exfiltration of secrets.
42+
43+
You can store the certificates as Key Vault certificate or Key Vault secret. Private keys are in PEM format.
44+
45+
### Key Vault Integration
46+
When configuring your Azure Stream Analytics job to connect to your Kafka clusters, depending on your configuration, you may have to configure your job to be able to access your Kafka clusters which are behind a firewall or inside a virtual network. Please visit the Azure Stream Analytics VNET documentation to learn more about configuring private endpoints to access resources which are inside a virtual network or behind a firewall.
47+
48+
49+
### Configuration
50+
The following table lists the property names and their description for creating a Kafka Output:
51+
52+
| Property name | Description |
53+
|------------------------------|-------------------------------------------------------------------------------------------------------------------------|
54+
| Input/Output Alias | A friendly name used in queries to reference your input or output |
55+
| Bootstrap server addresses | A list of host/port pairs to use for establishing the connection to the Kafka cluster. |
56+
| Kafka topic | A unit of your Kafka cluster you want to write events to. |
57+
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports: mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
58+
| Event Serialization format | The serialization format (JSON, CSV, Avro) of the outgoing data stream. |
59+
| Partition key | Azure Stream Analytics assigns partitions using round partitioning. |
60+
| Kafka event compression type | The compression type used for outgoing data stream, such as Gzip, Snappy, Lz4, Zstd or None. | |
61+
62+
### Limitations
63+
* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with a minimum of six (6) streaming units.
64+
* When using mTLS or SASL_SSL with Azure Key Vault, you must convert your Java Key Store to PEM format.
65+
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.
66+
67+
> [!NOTE]
68+
> For direct on using the Azure Stream Analytics kafka adapter, please reach out to [[email protected]](mailto:[email protected]).
69+
>
70+
71+
## Next steps
72+
> [!div class="nextstepaction"]
73+
> [Quickstart: Create a Stream Analytics job by using the Azure portal](stream-analytics-quick-create-portal.md)
74+
75+
<!--Link references-->
76+
[stream.analytics.developer.guide]: ../stream-analytics-developer-guide.md
77+
[stream.analytics.scale.jobs]: stream-analytics-scale-jobs.md
78+
[stream.analytics.introduction]: stream-analytics-introduction.md
79+
[stream.analytics.get.started]: stream-analytics-real-time-fraud-detection.md
80+
[stream.analytics.query.language.reference]: /stream-analytics-query/stream-analytics-query-language-reference

‎articles/stream-analytics/stream-analytics-define-kafka-input.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
title: Stream data from Kafka into Azure Stream Analytics
3+
description: Learn about setting up Azure Stream Analytics as a consumer from kafka
4+
author: enkrumah
5+
ms.author: ebnkruma
6+
ms.service: stream-analytics
7+
ms.topic: conceptual
8+
ms.date: 09/25/2023
9+
---
10+
11+
# Stream data from Kafka into Azure Stream Analytics (Preview)
12+
13+
Kafka is a distributed streaming platform used to publish and subscribe to streams of records. Kafka is designed to allow your apps to process records as they occur. It is an open-source system developed by the Apache Software Foundation written in Java and Scala.
14+
15+
The following are the major use cases:
16+
* Messaging
17+
* Website Activity Tracking
18+
* Metrics
19+
* Log Aggregation
20+
* Stream Processing
21+
22+
Azure Stream Analytics allows you to connect directly to kafka clusters to ingest data.The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The Kafka Adapters are backward compatible and support all Kafka versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint depending on the configurations. The configuration relies on existing Kafka configuration conventions.
23+
24+
### Compression
25+
26+
Supported compression types are: None, Gzip, Snappy, LZ4 and Zstd compression.
27+
28+
## Authentication and Encryption
29+
30+
You can use four types of security protocols to connect to your Kafka clusters:
31+
| Property name | Description |
32+
|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
33+
| mTLS | encryption and authentication |
34+
| SASL_SSL | It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and SSL (Secure Sockets Layer) - to ensure both authentication and encryption are in place for data transmission. |
35+
| Kafka topic | A unit of your Kafka cluster you want to write events to. |
36+
| SASL_PLAINTEXT | standard authentication with username and password without encryption |
37+
| None | The serialization format (JSON, CSV, Avro, Parquet) of the incoming data stream. |
38+
39+
40+
> [!IMPORTANT]
41+
> Confluent Cloud supports authenticating using API Keys, OAuth, or SAML single sign-on (SSO). Azure Stream Analytics do not currently support these authentication options. There will be a update in the near future to supporting authenticating to confluent cloud using API Keys.
42+
>
43+
44+
45+
### Key Vault Integration
46+
47+
> [!NOTE]
48+
> When using truststore certificates with mTLS or SASL_SSL security protocols, you must have Azure KeyVault and managed identity configured for your Azure Stream Analytics job.
49+
>
50+
51+
Azure Stream Analytics integrates seamlessly with Azure Key Vault to access stored secrets needed for authentication and encryption when using mTLS or SASL_SSL security protocols. Your Azure Stream Analytics job connects to Azure Key Vault using managed identity to ensure secure connection and avoid exfiltration of secrets.
52+
53+
You can store the certificates as Key Vault certificate or Key Vault secret. Private keys are in PEM format.
54+
55+
### Key Vault Integration
56+
When configuring your Azure Stream Analytics job to connect to your Kafka clusters, depending on your configuration, you may have to configure your job to be able to access your Kafka clusters which are behind a firewall or inside a virtual network. Please visit the Azure Stream Analytics VNET documentation to learn more about configuring private endpoints to access resources which are inside a virtual network or behind a firewall.
57+
58+
59+
### Configuration
60+
The following table lists the property names and their description for creating a Kafka Input:
61+
62+
| Property name | Description |
63+
|------------------------------|-------------------------------------------------------------------------------------------------------------------------|
64+
| Input/Output Alias | A friendly name used in queries to reference your input or output |
65+
| Bootstrap server addresses | A list of host/port pairs to use for establishing the connection to the Kafka cluster. |
66+
| Kafka topic | A unit of your Kafka cluster you want to write events to. |
67+
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports: mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
68+
| Event Serialization format | The serialization format (JSON, CSV, Avro, Parquet, Protobuf) of the incoming data stream. |
69+
70+
### Limitations
71+
* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with a minimum of six (6) streaming units.
72+
* When using mTLS or SASL_SSL with Azure Key Vault, you must convert your Java Key Store to PEM format.
73+
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.
74+
75+
> [!NOTE]
76+
> For direct on using the Azure Stream Analytics kafka adapter, please reach out to [[email protected]](mailto:[email protected]).
77+
>
78+
79+
80+
## Next steps
81+
> [!div class="nextstepaction"]
82+
> [Quickstart: Create a Stream Analytics job by using the Azure portal](stream-analytics-quick-create-portal.md)
83+
84+
<!--Link references-->
85+
[stream.analytics.developer.guide]: ../stream-analytics-developer-guide.md
86+
[stream.analytics.scale.jobs]: stream-analytics-scale-jobs.md
87+
[stream.analytics.introduction]: stream-analytics-introduction.md
88+
[stream.analytics.get.started]: stream-analytics-real-time-fraud-detection.md
89+
[stream.analytics.query.language.reference]: /stream-analytics-query/stream-analytics-query-language-reference

0 commit comments

Comments
 (0)