Skip to content

Commit 9d384bb

Browse files
authored
Merge pull request #290253 from MicrosoftDocs/release-ignite-2024-stream-analytics
[Ignite 2024 ship room] Azure Stream Analytics - 317531
2 parents c7fe19a + dd06f8a commit 9d384bb

File tree

6 files changed

+52
-51
lines changed

6 files changed

+52
-51
lines changed
Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,53 @@
11
---
2-
title: Connect Stream Analytics jobs to resources in an Azure Virtual Network (VNET)
3-
description: This article describes how to connect an Azure Stream Analytics job with resources that are in a VNET.
2+
title: Connect Stream Analytics jobs to resources in an Azure Virtual Network
3+
description: This article describes how to connect an Azure Stream Analytics job with resources that are in a virtual network.
44
author: ahartoon
55
ms.author: anboisve
66
ms.service: azure-stream-analytics
77
ms.topic: conceptual
8-
ms.date: 08/08/2023
8+
ms.date: 11/18/2024
99
ms.custom:
1010
---
11-
# Connect Stream Analytics jobs to resources in an Azure Virtual Network (VNET)
11+
# Connect Stream Analytics jobs to resources in an Azure Virtual Network
1212

13-
Your Stream Analytics jobs make outbound connections to your input and output Azure resources to process data in real time and produce results. These input and output resources (for example, Azure Event Hubs and Azure SQL Database) could be behind an Azure firewall or in an Azure Virtual Network (VNET). Stream Analytics service operates from networks that can't be directly included in your network rules.
13+
Your Stream Analytics jobs make outbound connections to your input and output Azure resources to process data in real time and produce results. These input and output resources (for example, Azure Event Hubs and Azure SQL Database) could be behind an Azure firewall or in an Azure Virtual Network. Stream Analytics service operates from networks that can't be directly included in your network rules.
1414

1515
However, there are several ways to securely connect your Stream Analytics jobs to your input and output resources in such scenarios.
16-
* [Run your Azure Stream Analytics job in an Azure Virtual Network (Public preview)](../stream-analytics/run-job-in-virtual-network.md)
16+
* [Run your Azure Stream Analytics job in an Azure Virtual Network](../stream-analytics/run-job-in-virtual-network.md)
1717
* Use private endpoints in Stream Analytics clusters.
1818
* Use Managed Identity authentication mode coupled with 'Allow trusted services' networking setting.
1919

20-
Your Stream Analytics job does not accept any inbound connection.
20+
Your Stream Analytics job doesn't accept any inbound connection.
2121

22-
## Run your Azure Stream Analytics job in an Azure Virtual Network (Public preview)
23-
Virtual network (VNET) support enables you to lock down access to Azure Stream Analytics to your virtual network infrastructure. This capability provides you with the benefits of network isolation and can be accomplished by [deploying a containerized instance of your ASA job inside your Virtual Network](../virtual-network/virtual-network-for-azure-services.md). Your VNET injected ASA job can then privately access your resources within the virtual network via:
22+
## Run your Azure Stream Analytics job in an Azure Virtual Network
23+
Virtual network support enables you to lock down access to Azure Stream Analytics to your virtual network infrastructure. This capability provides you with the benefits of network isolation and can be accomplished by [deploying a containerized instance of your Azure Stream Analytics job inside your Virtual Network](../virtual-network/virtual-network-for-azure-services.md). Your virtual network injected job can then privately access your resources within the virtual network via:
2424

25-
- [Private endpoints](../private-link/private-endpoint-overview.md), which connect your VNet injected ASA job to your data sources over private links powered by Azure Private Link.
26-
- [Service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md), which connect your data sources to your VNet injected ASA job.
25+
- [Private endpoints](../private-link/private-endpoint-overview.md), which connect your virtual network injected ASA job to your data sources over private links powered by Azure Private Link.
26+
- [Service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md), which connect your data sources to your virtual network injected ASA job.
2727
- [Service tags](../virtual-network/service-tags-overview.md), which allow or deny traffic to Azure Stream Analytics.
2828

29-
Currently, VNET integration is only available in **select regions**. Visit [this](../stream-analytics/run-job-in-virtual-network.md) page for most recent list of VNET enabled regions and how to request it in your region.
29+
Currently, virtual network integration is only available in **select regions**. Visit [this](../stream-analytics/run-job-in-virtual-network.md) page for most recent list of virtual network enabled regions and how to request it in your region.
3030

3131
## Private endpoints in Stream Analytics clusters.
3232
[Stream Analytics clusters](./cluster-overview.md) is a single tenant dedicated compute cluster where you can run your Stream Analytics jobs. You can create managed private endpoints in your Stream Analytics cluster, which allows any jobs running on your cluster to make a secure outbound connection to your input and output resources.
3333

34-
The creation of private endpoints in your Stream Analytics cluster is a [two step operation](./private-endpoints.md). This option is best suited for medium to large streaming workloads as the minimum size of a Stream Analytics cluster is 12 SU V2 or 36 SU V1s (SUs can be shared by different jobs in various subscriptions or environments like development, test, and production). See [Azure Stream Analytics cluster](../stream-analytics/cluster-overview.md) for more information.
34+
The creation of private endpoints in your Stream Analytics cluster is a [two step operation](./private-endpoints.md). This option is best suited for medium to large streaming workloads as the minimum size of a Stream Analytics cluster is 12 SU V2 or 36 SU V1s (SUs can be shared by different jobs in various subscriptions or environments like development, test, and production). For more information, see [Azure Stream Analytics cluster](../stream-analytics/cluster-overview.md).
3535

3636
## Managed identity authentication with 'Allow trusted services' configuration
3737
Some Azure services provide **Allow trusted Microsoft services** networking setting, which when enabled, allows your Stream Analytics jobs to securely connect to your resource using strong authentication. This option allows you to connect your jobs to your input and output resources without requiring a Stream Analytics cluster and private endpoints. Configuring your job to use this technique is a 2-step operation:
3838
* Use Managed Identity authentication mode when configuring input or output in your Stream Analytics job.
3939
* Grant your specific Stream Analytics jobs explicit access to your target resources by assigning an Azure role to the job's system-assigned managed identity.
4040

41-
Enabling **Allow trusted Microsoft services** does not grant blanket access to any job. This gives you full control of which specific Stream Analytics jobs can access your resources securely.
41+
Enabling **Allow trusted Microsoft services** doesn't grant blanket access to any job. It gives you full control of which specific Stream Analytics jobs can access your resources securely.
4242

4343
Your jobs can connect to the following Azure services using this technique:
4444
1. [Blob Storage or Azure Data Lake Storage Gen2](./blob-output-managed-identity.md) - can be your job's storage account, streaming input or output.
4545
2. [Azure Event Hubs](./event-hubs-managed-identity.md) - can be your job's streaming input or output.
4646

47-
If your jobs need to connect to other input or output types, you could write from Stream Analytics to Event Hubs output first and then to any destination of your choice using Azure Functions. If you want to directly write from Stream Analytics to other output types secured in a VNet or firewall, then the only option is to use private endpoints in Stream Analytics clusters.
47+
If your jobs need to connect to other input or output types, you could write from Stream Analytics to Event Hubs output first and then to any destination of your choice using Azure Functions. If you want to directly write from Stream Analytics to other output types secured in a virtual network or firewall, then the only option is to use private endpoints in Stream Analytics clusters.
4848

4949
## Next steps
5050

5151
* [Create and remove Private Endpoints in Stream Analytics clusters](./private-endpoints.md)
52-
* [Connect to Event Hubs in a VNet using Managed Identity authentication](./event-hubs-managed-identity.md)
53-
* [Connect to Blob storage and ADLS Gen2 in a VNet using Managed Identity authentication](./blob-output-managed-identity.md)
52+
* [Connect to Event Hubs in a virtual network using Managed Identity authentication](./event-hubs-managed-identity.md)
53+
* [Connect to Blob storage and ADLS Gen2 in a virtual network using Managed Identity authentication](./blob-output-managed-identity.md)

articles/stream-analytics/kafka-output.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,20 +5,20 @@ author: AliciaLiMicrosoft
55
ms.author: ali
66
ms.service: azure-stream-analytics
77
ms.topic: conceptual
8-
ms.date: 02/20/2024
8+
ms.date: 11/18/2024
99
---
1010

11-
# Kafka output from Azure Stream Analytics (Preview)
11+
# Kafka output from Azure Stream Analytics
1212

13-
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka output is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
13+
Azure Stream Analytics allows you to connect directly to Kafka clusters as a producer to output data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The Kafka output is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a virtual network and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions.
1414
Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.
1515

1616
## Steps
1717
This article shows how to set up Kafka as an output from Azure Stream Analytics. There are six steps:
1818

1919
1. Create an Azure Stream Analytics job.
20-
2. Configure your Azure Stream Analytics job to use managed identity if you are using mTLS or SASL_SSl security protocols.
21-
3. Configure Azure Key vault if you are using mTLS or SASL_SSl security protocols.
20+
2. Configure your Azure Stream Analytics job to use managed identity if you're using mTLS or SASL_SSl security protocols.
21+
3. Configure Azure Key vault if you're using mTLS or SASL_SSl security protocols.
2222
4. Upload certificates as secrets into Azure Key vault.
2323
5. Grant Azure Stream Analytics permissions to access the uploaded certificate.
2424
6. Configure Kafka output in your Azure Stream Analytics job.
@@ -35,7 +35,7 @@ The following table lists the property names and their description for creating
3535
| Output Alias | A friendly name used in queries to reference your output |
3636
| Bootstrap server addresses | A list of host/port pairs to establish the connection to the Kafka cluster. |
3737
| Kafka topic | A named, ordered, and partitioned stream of data that allows for the publish-subscribe and event-driven processing of messages. |
38-
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT or None. |
38+
| Security Protocol | How you want to connect to your Kafka cluster. Azure Stream Analytics supports mTLS, SASL_SSL, SASL_PLAINTEXT, or None. |
3939
| Event Serialization format | The serialization format (JSON, CSV, Avro) of the outgoing data stream. |
4040
| Partition key | Azure Stream Analytics assigns partitions using round partitioning. |
4141
| Kafka event compression type | The compression type used for outgoing data streams, such as Gzip, Snappy, Lz4, Zstd, or None. |
@@ -54,7 +54,7 @@ You can use four types of security protocols to connect to your Kafka clusters:
5454
|Property name |Description |
5555
|----------|-----------|
5656
|mTLS |Encryption and authentication. Supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 security mechanisms. |
57-
|SASL_SSL |It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and SSL (Secure Sockets Layer) - to ensure both authentication and encryption are in place for data transmission. The SASL_SSL protocol supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 security mechanisms. |
57+
|SASL_SSL |It combines two different security mechanisms - SASL (Simple Authentication and Security Layer) and Secure Sockets Layer (SSL) - to ensure both authentication and encryption are in place for data transmission. The SASL_SSL protocol supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 security mechanisms. |
5858
|SASL_PLAINTEXT |standard authentication with username and password without encryption |
5959
|None | No authentication and encryption. |
6060

@@ -63,7 +63,7 @@ You can use four types of security protocols to connect to your Kafka clusters:
6363
> Confluent Cloud supports authentication using API Keys, OAuth, or SAML single sign-on (SSO). Azure Stream Analytics doesn't support OAuth or SAML single sign-on (SSO) authentication.
6464
> You can connect to the confluent cloud using an API Key with topic-level access via the SASL_SSL security protocol.
6565
66-
For a step-by-step tutorial on connecting to confluent cloud kakfa, visit the documentation:
66+
For a step-by-step tutorial on connecting to confluent cloud Kafka, visit the documentation:
6767

6868
* Confluent cloud kafka input: [Stream data from confluent cloud Kafka with Azure Stream Analytics](confluent-kafka-input.md)
6969
* Confluent cloud kafka output: [Stream data from Azure Stream Analytics into confluent cloud](confluent-kafka-output.md)
@@ -139,7 +139,7 @@ You can configure your ASA job to use managed identity by navigating to the **Ma
139139

140140
:::image type="content" source="./media/common/stream-analytics-enable-managed-identity-new.png" alt-text="Screenshot showing how to configure managed identity for an ASA job." lightbox="./media/common/stream-analytics-enable-managed-identity-new.png" :::
141141

142-
1. Click on the **managed identity tab** under **configure**.
142+
1. On the left menu, select **managed identity** tab under **configure**.
143143
2. Select **Switch Identity** and select the identity to use with the job: system-assigned identity or user-assigned identity.
144144
3. For user-assigned identity, select the subscription where your user-assigned identity is located and select the name of your identity.
145145
4. Review and **save**.
@@ -162,17 +162,17 @@ Use the following steps to grant special permissions to your stream analytics jo
162162
| Members | \<Name of your Stream Analytics job> or \<name of user-assigned identity> |
163163

164164

165-
### VNET integration
165+
### Virtual network integration
166166

167-
If your Kafka is inside a virtual network (VNET) or behind a firewall, you must configure your Azure Stream Analytics job to access your Kafka topic.
167+
If your Kafka is inside a virtual network or behind a firewall, you must configure your Azure Stream Analytics job to access your Kafka topic.
168168
Visit the [Run your Azure Stream Analytics job in an Azure Virtual Network documentation](../stream-analytics/run-job-in-virtual-network.md) for more information.
169169

170170

171171
### Limitations
172-
* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with at least six (6) streaming units or one (1) V2 streaming unit.
172+
* When configuring your Azure Stream Analytics jobs to use virtual network/SWIFT, your job must be configured with at least six (6) streaming units or one (1) V2 streaming unit.
173173
* When using mTLS or SASL_SSL with Azure Key vault, you must convert your Java Key Store to PEM format.
174-
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.
175-
* Azure Stream Analytics does not support authentication to confluent cloud using OAuth or SAML single sign-on (SSO). You must use API Key via the SASL_SSL protocol
174+
* The minimum version of Kafka you can configure Azure Stream Analytics to connect to be version 0.10.
175+
* Azure Stream Analytics doesn't support authentication to confluent cloud using OAuth or SAML single sign-on (SSO). You must use API Key via the SASL_SSL protocol
176176

177177
> [!NOTE]
178178
> For direct help with using the Azure Stream Analytics Kafka output, please reach out to [[email protected]](mailto:[email protected]).
-5.29 KB
Loading

articles/stream-analytics/run-job-in-virtual-network.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,11 @@ author: ahartoon
55
ms.author: anboisve
66
ms.service: azure-stream-analytics
77
ms.topic: how-to
8-
ms.date: 10/08/2024
8+
ms.date: 11/18/2024
9+
ms.custom: references_regions
910
---
1011

11-
# Run your Azure Stream Analytics job in an Azure Virtual Network (Public preview)
12+
# Run your Azure Stream Analytics job in an Azure Virtual Network
1213
This article describes how to run your Azure Stream Analytics (ASA) job in an Azure virtual network.
1314

1415
## Overview
@@ -19,7 +20,7 @@ Virtual network support enables you to lock down access to Azure Stream Analytic
1920
- [Service tags](../virtual-network/service-tags-overview.md), which allow or deny traffic to Azure Stream Analytics.
2021

2122
## Availability
22-
Currently, this capability is only available in select regions: **West US**, **Central Canada**, **East US**, **East US 2**, **Central US**, **West Europe**, and **North Europe**.
23+
Currently, this capability is only available in select **regions**: East US, East US 2, West US, Central US, North-Central US, Central Canada, West Europe, North Europe, Southeast Asia, Brazil South, Japan East, UK South, Central India, Australia East, and France Central.
2324
If you're interested in enabling virtual network integration in your region, **fill out this [form](https://forms.office.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbRzFwASREnlZFvs9gztPNuTdUMU5INk5VT05ETkRBTTdSMk9BQ0w3OEZDQi4u)**.
2425

2526
## Requirements for virtual network integration support

0 commit comments

Comments
 (0)