You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Connect Stream Analytics jobs to resources in an Azure Virtual Network (VNET)
3
-
description: This article describes how to connect an Azure Stream Analytics job with resources that are in a VNET.
2
+
title: Connect Stream Analytics jobs to resources in an Azure Virtual Network
3
+
description: This article describes how to connect an Azure Stream Analytics job with resources that are in a virtual network.
4
4
author: ahartoon
5
5
ms.author: anboisve
6
6
ms.service: azure-stream-analytics
7
7
ms.topic: conceptual
8
-
ms.date: 08/08/2023
8
+
ms.date: 11/18/2024
9
9
ms.custom:
10
10
---
11
-
# Connect Stream Analytics jobs to resources in an Azure Virtual Network (VNET)
11
+
# Connect Stream Analytics jobs to resources in an Azure Virtual Network
12
12
13
-
Your Stream Analytics jobs make outbound connections to your input and output Azure resources to process data in real time and produce results. These input and output resources (for example, Azure Event Hubs and Azure SQL Database) could be behind an Azure firewall or in an Azure Virtual Network (VNET). Stream Analytics service operates from networks that can't be directly included in your network rules.
13
+
Your Stream Analytics jobs make outbound connections to your input and output Azure resources to process data in real time and produce results. These input and output resources (for example, Azure Event Hubs and Azure SQL Database) could be behind an Azure firewall or in an Azure Virtual Network. Stream Analytics service operates from networks that can't be directly included in your network rules.
14
14
15
15
However, there are several ways to securely connect your Stream Analytics jobs to your input and output resources in such scenarios.
16
-
*[Run your Azure Stream Analytics job in an Azure Virtual Network (Public preview)](../stream-analytics/run-job-in-virtual-network.md)
16
+
*[Run your Azure Stream Analytics job in an Azure Virtual Network](../stream-analytics/run-job-in-virtual-network.md)
17
17
* Use private endpoints in Stream Analytics clusters.
18
18
* Use Managed Identity authentication mode coupled with 'Allow trusted services' networking setting.
19
19
20
-
Your Stream Analytics job does not accept any inbound connection.
20
+
Your Stream Analytics job doesn't accept any inbound connection.
21
21
22
-
## Run your Azure Stream Analytics job in an Azure Virtual Network (Public preview)
23
-
Virtual network (VNET) support enables you to lock down access to Azure Stream Analytics to your virtual network infrastructure. This capability provides you with the benefits of network isolation and can be accomplished by [deploying a containerized instance of your ASA job inside your Virtual Network](../virtual-network/virtual-network-for-azure-services.md). Your VNET injected ASA job can then privately access your resources within the virtual network via:
22
+
## Run your Azure Stream Analytics job in an Azure Virtual Network
23
+
Virtual network support enables you to lock down access to Azure Stream Analytics to your virtual network infrastructure. This capability provides you with the benefits of network isolation and can be accomplished by [deploying a containerized instance of your Azure Stream Analytics job inside your Virtual Network](../virtual-network/virtual-network-for-azure-services.md). Your virtual network injected job can then privately access your resources within the virtual network via:
24
24
25
-
-[Private endpoints](../private-link/private-endpoint-overview.md), which connect your VNet injected ASA job to your data sources over private links powered by Azure Private Link.
26
-
-[Service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md), which connect your data sources to your VNet injected ASA job.
25
+
-[Private endpoints](../private-link/private-endpoint-overview.md), which connect your virtual network injected ASA job to your data sources over private links powered by Azure Private Link.
26
+
-[Service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md), which connect your data sources to your virtual network injected ASA job.
27
27
-[Service tags](../virtual-network/service-tags-overview.md), which allow or deny traffic to Azure Stream Analytics.
28
28
29
-
Currently, VNET integration is only available in **select regions**. Visit [this](../stream-analytics/run-job-in-virtual-network.md) page for most recent list of VNET enabled regions and how to request it in your region.
29
+
Currently, virtual network integration is only available in **select regions**. Visit [this](../stream-analytics/run-job-in-virtual-network.md) page for most recent list of virtual network enabled regions and how to request it in your region.
30
30
31
31
## Private endpoints in Stream Analytics clusters.
32
32
[Stream Analytics clusters](./cluster-overview.md) is a single tenant dedicated compute cluster where you can run your Stream Analytics jobs. You can create managed private endpoints in your Stream Analytics cluster, which allows any jobs running on your cluster to make a secure outbound connection to your input and output resources.
33
33
34
-
The creation of private endpoints in your Stream Analytics cluster is a [two step operation](./private-endpoints.md). This option is best suited for medium to large streaming workloads as the minimum size of a Stream Analytics cluster is 12 SU V2 or 36 SU V1s (SUs can be shared by different jobs in various subscriptions or environments like development, test, and production). See [Azure Stream Analytics cluster](../stream-analytics/cluster-overview.md) for more information.
34
+
The creation of private endpoints in your Stream Analytics cluster is a [two step operation](./private-endpoints.md). This option is best suited for medium to large streaming workloads as the minimum size of a Stream Analytics cluster is 12 SU V2 or 36 SU V1s (SUs can be shared by different jobs in various subscriptions or environments like development, test, and production). For more information, see [Azure Stream Analytics cluster](../stream-analytics/cluster-overview.md).
35
35
36
36
## Managed identity authentication with 'Allow trusted services' configuration
37
37
Some Azure services provide **Allow trusted Microsoft services** networking setting, which when enabled, allows your Stream Analytics jobs to securely connect to your resource using strong authentication. This option allows you to connect your jobs to your input and output resources without requiring a Stream Analytics cluster and private endpoints. Configuring your job to use this technique is a 2-step operation:
38
38
* Use Managed Identity authentication mode when configuring input or output in your Stream Analytics job.
39
39
* Grant your specific Stream Analytics jobs explicit access to your target resources by assigning an Azure role to the job's system-assigned managed identity.
40
40
41
-
Enabling **Allow trusted Microsoft services**does not grant blanket access to any job. This gives you full control of which specific Stream Analytics jobs can access your resources securely.
41
+
Enabling **Allow trusted Microsoft services**doesn't grant blanket access to any job. It gives you full control of which specific Stream Analytics jobs can access your resources securely.
42
42
43
43
Your jobs can connect to the following Azure services using this technique:
44
44
1.[Blob Storage or Azure Data Lake Storage Gen2](./blob-output-managed-identity.md) - can be your job's storage account, streaming input or output.
45
45
2.[Azure Event Hubs](./event-hubs-managed-identity.md) - can be your job's streaming input or output.
46
46
47
-
If your jobs need to connect to other input or output types, you could write from Stream Analytics to Event Hubs output first and then to any destination of your choice using Azure Functions. If you want to directly write from Stream Analytics to other output types secured in a VNet or firewall, then the only option is to use private endpoints in Stream Analytics clusters.
47
+
If your jobs need to connect to other input or output types, you could write from Stream Analytics to Event Hubs output first and then to any destination of your choice using Azure Functions. If you want to directly write from Stream Analytics to other output types secured in a virtual network or firewall, then the only option is to use private endpoints in Stream Analytics clusters.
48
48
49
49
## Next steps
50
50
51
51
*[Create and remove Private Endpoints in Stream Analytics clusters](./private-endpoints.md)
52
-
*[Connect to Event Hubs in a VNet using Managed Identity authentication](./event-hubs-managed-identity.md)
53
-
*[Connect to Blob storage and ADLS Gen2 in a VNet using Managed Identity authentication](./blob-output-managed-identity.md)
52
+
*[Connect to Event Hubs in a virtual network using Managed Identity authentication](./event-hubs-managed-identity.md)
53
+
*[Connect to Blob storage and ADLS Gen2 in a virtual network using Managed Identity authentication](./blob-output-managed-identity.md)
Copy file name to clipboardExpand all lines: articles/stream-analytics/kafka-output.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,20 +5,20 @@ author: AliciaLiMicrosoft
5
5
ms.author: ali
6
6
ms.service: azure-stream-analytics
7
7
ms.topic: conceptual
8
-
ms.date: 02/20/2024
8
+
ms.date: 11/18/2024
9
9
---
10
10
11
-
# Kafka output from Azure Stream Analytics (Preview)
11
+
# Kafka output from Azure Stream Analytics
12
12
13
-
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka output is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
13
+
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The Kafka output is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a virtual network and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
14
14
Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
15
15
16
16
## Steps
17
17
This article shows how to set up Kafka as an output from Azure Stream Analytics. There are six steps:
18
18
19
19
1. Create an Azure Stream Analytics job.
20
-
2. Configure your Azure Stream Analytics job to use managed identity if you are using mTLS or SASL_SSl security protocols.
21
-
3. Configure Azure Key vault if you are using mTLS or SASL_SSl security protocols.
20
+
2. Configure your Azure Stream Analytics job to use managed identity if you're using mTLS or SASL_SSl security protocols.
21
+
3. Configure Azure Key vault if you're using mTLS or SASL_SSl security protocols.
22
22
4. Upload certificates as secrets into Azure Key vault.
23
23
5. Grant Azure Stream Analytics permissions to access the uploaded certificate.
24
24
6. Configure Kafka output in your Azure Stream Analytics job.
@@ -35,7 +35,7 @@ The following table lists the property names and their description for creating
35
35
| Output Alias | A friendly name used in queries to reference your output |
36
36
| Bootstrap server addresses | A list of host/port pairs to establish the connection to the Kafka cluster. |
37
37
| Kafka topic | A named, ordered, and partitioned stream of data that allows for the publish-subscribe and event-driven processing of messages. |
38
-
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
38
+
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT, or None. |
39
39
| Event Serialization format | The serialization format (JSON, CSV, Avro) of the outgoing data stream. |
| Kafka event compression type | The compression type used for outgoing data streams, such as Gzip, Snappy, Lz4, Zstd, or None. |
@@ -54,7 +54,7 @@ You can use four types of security protocols to connect to your Kafka clusters:
54
54
|Property name |Description |
55
55
|----------|-----------|
56
56
|mTLS |Encryption and authentication. Supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 security mechanisms. |
57
-
|SASL_SSL |It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and SSL (Secure Sockets Layer) - to ensure both authentication and encryption are in place for data transmission. The SASL_SSL protocol supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 security mechanisms. |
57
+
|SASL_SSL |It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and Secure Sockets Layer (SSL) - to ensure both authentication and encryption are in place for data transmission. The SASL_SSL protocol supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 security mechanisms. |
58
58
|SASL_PLAINTEXT |standard authentication with username and password without encryption |
59
59
|None | No authentication and encryption. |
60
60
@@ -63,7 +63,7 @@ You can use four types of security protocols to connect to your Kafka clusters:
63
63
> Confluent Cloud supports authentication using API Keys, OAuth, or SAML single sign-on (SSO). Azure Stream Analytics doesn't support OAuth or SAML single sign-on (SSO) authentication.
64
64
> You can connect to the confluent cloud using an API Key with topic-level access via the SASL_SSL security protocol.
65
65
66
-
For a step-by-step tutorial on connecting to confluent cloud kakfa, visit the documentation:
66
+
For a step-by-step tutorial on connecting to confluent cloud Kafka, visit the documentation:
67
67
68
68
* Confluent cloud kafka input: [Stream data from confluent cloud Kafka with Azure Stream Analytics](confluent-kafka-input.md)
69
69
* Confluent cloud kafka output: [Stream data from Azure Stream Analytics into confluent cloud](confluent-kafka-output.md)
@@ -139,7 +139,7 @@ You can configure your ASA job to use managed identity by navigating to the **Ma
139
139
140
140
:::image type="content" source="./media/common/stream-analytics-enable-managed-identity-new.png" alt-text="Screenshot showing how to configure managed identity for an ASA job." lightbox="./media/common/stream-analytics-enable-managed-identity-new.png" :::
141
141
142
-
1. Click on the **managed identity tab** under **configure**.
142
+
1. On the left menu, select **managed identity** tab under **configure**.
143
143
2. Select **Switch Identity** and select the identity to use with the job: system-assigned identity or user-assigned identity.
144
144
3. For user-assigned identity, select the subscription where your user-assigned identity is located and select the name of your identity.
145
145
4. Review and **save**.
@@ -162,17 +162,17 @@ Use the following steps to grant special permissions to your stream analytics jo
162
162
| Members |\<Name of your Stream Analytics job> or \<name of user-assigned identity> |
163
163
164
164
165
-
### VNET integration
165
+
### Virtual network integration
166
166
167
-
If your Kafka is inside a virtual network (VNET) or behind a firewall, you must configure your Azure Stream Analytics job to access your Kafka topic.
167
+
If your Kafka is inside a virtual network or behind a firewall, you must configure your Azure Stream Analytics job to access your Kafka topic.
168
168
Visit the [Run your Azure Stream Analytics job in an Azure Virtual Network documentation](../stream-analytics/run-job-in-virtual-network.md) for more information.
169
169
170
170
171
171
### Limitations
172
-
* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with at least six (6) streaming units or one (1) V2 streaming unit.
172
+
* When configuring your Azure Stream Analytics jobs to use virtual network/SWIFT, your job must be configured with at least six (6) streaming units or one (1) V2 streaming unit.
173
173
* When using mTLS or SASL_SSL with Azure Key vault, you must convert your Java Key Store to PEM format.
174
-
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.
175
-
* Azure Stream Analytics does not support authentication to confluent cloud using OAuth or SAML single sign-on (SSO). You must use API Key via the SASL_SSL protocol
174
+
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to be version 0.10.
175
+
* Azure Stream Analytics doesn't support authentication to confluent cloud using OAuth or SAML single sign-on (SSO). You must use API Key via the SASL_SSL protocol
176
176
177
177
> [!NOTE]
178
178
> For direct help with using the Azure Stream Analytics Kafka output, please reach out to [[email protected]](mailto:[email protected]).
Copy file name to clipboardExpand all lines: articles/stream-analytics/run-job-in-virtual-network.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,10 +5,11 @@ author: ahartoon
5
5
ms.author: anboisve
6
6
ms.service: azure-stream-analytics
7
7
ms.topic: how-to
8
-
ms.date: 10/08/2024
8
+
ms.date: 11/18/2024
9
+
ms.custom: references_regions
9
10
---
10
11
11
-
# Run your Azure Stream Analytics job in an Azure Virtual Network (Public preview)
12
+
# Run your Azure Stream Analytics job in an Azure Virtual Network
12
13
This article describes how to run your Azure Stream Analytics (ASA) job in an Azure virtual network.
13
14
14
15
## Overview
@@ -19,7 +20,7 @@ Virtual network support enables you to lock down access to Azure Stream Analytic
19
20
-[Service tags](../virtual-network/service-tags-overview.md), which allow or deny traffic to Azure Stream Analytics.
20
21
21
22
## Availability
22
-
Currently, this capability is only available in select regions: **West US**, **Central Canada**, **East US**, **East US 2**, **Central US**, **West Europe**, and **North Europe**.
23
+
Currently, this capability is only available in select **regions**: East US, East US 2, West US, Central US, North-Central US, Central Canada, West Europe, North Europe, Southeast Asia, Brazil South, Japan East, UK South, Central India, Australia East, and France Central.
23
24
If you're interested in enabling virtual network integration in your region, **fill out this [form](https://forms.office.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbRzFwASREnlZFvs9gztPNuTdUMU5INk5VT05ETkRBTTdSMk9BQ0w3OEZDQi4u)**.
24
25
25
26
## Requirements for virtual network integration support
0 commit comments