You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/how-to-private-endpoint-integration-synapse.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@ ms.date: 11/16/2022
18
18
In this article, learn how to securely integrate with Azure Machine Learning from Azure Synapse. This integration enables you to use Azure Machine Learning from notebooks in your Azure Synapse workspace. Communication between the two workspaces is secured using an Azure Virtual Network.
19
19
20
20
> [!TIP]
21
-
> You can also perform integration in the opposite direction, using Azure Synapse spark pool from Azure Machine Learning. For more information, see [Link Azure Synapse and Azure Machine Learning](how-to-link-synapse-ml-workspaces.md).
21
+
> You can also perform integration in the opposite direction, using Azure Synapse spark pool from Azure Machine Learning. For more information, see [Link Azure Synapse and Azure Machine Learning](v1/how-to-link-synapse-ml-workspaces.md).
22
22
23
23
## Prerequisites
24
24
@@ -141,4 +141,4 @@ To verify that the integration between Azure Synapse and Azure Machine Learning
141
141
## Next steps
142
142
143
143
* [Quickstart: Create a new Azure Machine Learning linked service in Synapse](../synapse-analytics/machine-learning/quickstart-integrate-azure-machine-learning.md).
144
-
* [Link Azure Synapse Analytics and Azure Machine Learning workspaces](how-to-link-synapse-ml-workspaces.md).
144
+
* [Link Azure Synapse Analytics and Azure Machine Learning workspaces](v1/how-to-link-synapse-ml-workspaces.md).
Copy file name to clipboardExpand all lines: articles/machine-learning/v1/how-to-attach-compute-targets.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -130,7 +130,7 @@ Azure Machine Learning also supports attaching an Azure Virtual Machine. The VM
130
130
131
131
## <aid="synapse"></a>Apache Spark pools
132
132
133
-
The Azure Synapse Analytics integration with Azure Machine Learning (preview) allows you to attach an Apache Spark pool backed by Azure Synapse for interactive data exploration and preparation. With this integration, you can have a dedicated compute for data wrangling at scale. For more information, see [How to attach Apache Spark pools powered by Azure Synapse Analytics](../how-to-link-synapse-ml-workspaces.md#attach-synapse-spark-pool-as-a-compute).
133
+
The Azure Synapse Analytics integration with Azure Machine Learning (preview) allows you to attach an Apache Spark pool backed by Azure Synapse for interactive data exploration and preparation. With this integration, you can have a dedicated compute for data wrangling at scale. For more information, see [How to attach Apache Spark pools powered by Azure Synapse Analytics](how-to-link-synapse-ml-workspaces.md#attach-synapse-spark-pool-as-a-compute).
#Customer intent: As a data scientist, I want to prepare my data at scale, and to train my machine learning models from a single notebook using Azure Machine Learning.
15
15
---
16
16
17
-
# Data wrangling with Apache Spark pools (preview)
17
+
# Data wrangling with Apache Spark pools (deprecated)
> The Azure Synapse Analytics integration with Azure Machine Learning available in Python SDK v1 is deprecated. Users can continue using Synapse workspace registered with Azure Machine Learning as a linked service. However, a new Synapse workspace can no longer be registered with Azure Machine Learning as a linked service. We recommend using Managed (Automatic) Synapse compute and attached Synapse Spark pools available in CLI v2 and Python SDK v2. Please see [https://aka.ms/aml-spark](https://aka.ms/aml-spark) for more details.
21
23
22
24
In this article, you learn how to perform data wrangling tasks interactively within a dedicated Synapse session, powered by [Azure Synapse Analytics](../../synapse-analytics/overview-what-is.md), in a Jupyter notebook using the [Azure Machine Learning Python SDK](/python/api/overview/azure/ml/).
23
25
24
26
If you prefer to use Azure Machine Learning pipelines, see [How to use Apache Spark (powered by Azure Synapse Analytics) in your machine learning pipeline (preview)](how-to-use-synapsesparkstep.md).
25
27
26
28
For guidance on how to use Azure Synapse Analytics with a Synapse workspace, see the [Azure Synapse Analytics get started series](../../synapse-analytics/get-started.md).
27
29
28
-
>[!IMPORTANT]
29
-
> The Azure Machine Learning and Azure Synapse Analytics integration is in preview. The capabilities presented in this article employ the `azureml-synapse` package which contains [experimental](/python/api/overview/azure/ml/#stable-vs-experimental) preview features that may change at any time.
30
-
31
30
## Azure Machine Learning and Azure Synapse Analytics integration
32
31
33
32
The Azure Synapse Analytics integration with Azure Machine Learning (preview) allows you to attach an Apache Spark pool backed by Azure Synapse for interactive data exploration and preparation. With this integration, you can have a dedicated compute for data wrangling at scale, all within the same Python notebook you use for training your machine learning models.
@@ -50,9 +49,9 @@ The Azure Synapse Analytics integration with Azure Machine Learning (preview) al
50
49
pip install azureml-synapse
51
50
```
52
51
53
-
* Link your Azure Machine Learning workspace and Azure Synapse Analytics workspace with the [Azure Machine Learning Python SDK](../how-to-link-synapse-ml-workspaces.md#link-sdk) or via the [Azure Machine Learning studio](../how-to-link-synapse-ml-workspaces.md#link-studio)
52
+
* Link your Azure Machine Learning workspace and Azure Synapse Analytics workspace with the [Azure Machine Learning Python SDK](how-to-link-synapse-ml-workspaces.md#link-sdk) or via the [Azure Machine Learning studio](how-to-link-synapse-ml-workspaces.md#link-studio)
54
53
55
-
*[Attach a Synapse Spark pool](../how-to-link-synapse-ml-workspaces.md#attach-synapse-spark-pool-as-a-compute) as a compute target.
54
+
*[Attach a Synapse Spark pool](how-to-link-synapse-ml-workspaces.md#attach-synapse-spark-pool-as-a-compute) as a compute target.
56
55
57
56
## Launch Synapse Spark pool for data wrangling tasks
58
57
@@ -251,7 +250,7 @@ input1 = train_ds.as_mount()
251
250
252
251
## Use a `ScriptRunConfig` to submit an experiment run to a Synapse Spark pool
253
252
254
-
If you're ready to automate and productionize your data wrangling tasks, you can submit an experiment run to [an attached Synapse Spark pool](../how-to-link-synapse-ml-workspaces.md#attach-a-pool-with-the-python-sdk) with the [ScriptRunConfig](/python/api/azureml-core/azureml.core.scriptrunconfig) object.
253
+
If you're ready to automate and productionize your data wrangling tasks, you can submit an experiment run to [an attached Synapse Spark pool](how-to-link-synapse-ml-workspaces.md#attach-a-pool-with-the-python-sdk) with the [ScriptRunConfig](/python/api/azureml-core/azureml.core.scriptrunconfig) object.
255
254
256
255
Similarly, if you have an Azure Machine Learning pipeline, you can use the [SynapseSparkStep to specify your Synapse Spark pool as the compute target](how-to-use-synapsesparkstep.md) for the data preparation step in your pipeline.
Copy file name to clipboardExpand all lines: articles/machine-learning/v1/how-to-link-synapse-ml-workspaces.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
title: Create a linked service with Synapse and Azure Machine Learning workspaces (preview)
2
+
title: Create a linked service with Synapse and Azure Machine Learning workspaces (deprecated)
3
3
titleSuffix: Azure Machine Learning
4
4
description: Learn how to link Azure Synapse and Azure Machine Learning workspaces, and attach Apache Spark pools for a unified data wrangling experience.
#Customer intent: As a workspace administrator, I want to link Azure Synapse workspaces and Azure Machine Learning workspaces and attach Apache Spark pools for a unified data wrangling experience.
15
15
---
16
16
17
-
# Link Azure Synapse Analytics and Azure Machine Learning workspaces and attach Apache Spark pools(preview)
17
+
# Link Azure Synapse Analytics and Azure Machine Learning workspaces and attach Apache Spark pools(deprecated)
In this article, you learn how to create a linked service that links your [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md) workspace and [Azure Machine Learning workspace](concept-workspace.md).
21
+
> [!WARNING]
22
+
> The Azure Synapse Analytics integration with Azure Machine Learning available in Python SDK v1 is deprecated. Users can continue using Synapse workspace registered with Azure Machine Learning as a linked service. However, a new Synapse workspace can no longer be registered with Azure Machine Learning as a linked service. We recommend using Managed (Automatic) Synapse compute and attached Synapse Spark pools available in CLI v2 and Python SDK v2. Please see [https://aka.ms/aml-spark](https://aka.ms/aml-spark) for more details.
23
+
24
+
In this article, you learn how to create a linked service that links your [Azure Synapse Analytics](../../synapse-analytics/overview-what-is.md) workspace and [Azure Machine Learning workspace](../concept-workspace.md).
22
25
23
26
With your Azure Machine Learning workspace linked with your Azure Synapse workspace, you can attach an Apache Spark pool, powered by Azure Synapse Analytics, as a dedicated compute for data wrangling at scale or conduct model training all from the same Python notebook.
24
27
25
28
You can link your ML workspace and Synapse workspace via the [Python SDK](#link-sdk) or the [Azure Machine Learning studio](#link-studio).
26
29
27
30
You can also link workspaces and attach a Synapse Spark pool with a single [Azure Resource Manager (ARM) template](https://github.com/Azure/azure-quickstart-templates/blob/master/quickstarts/microsoft.machinelearningservices/machine-learning-linkedservice-create/azuredeploy.json).
28
31
29
-
>[!IMPORTANT]
30
-
> The Azure Machine Learning and Azure Synapse integration is in public preview. The functionalities presented from the `azureml-synapse` package are [experimental](/python/api/overview/azure/ml/#stable-vs-experimental) preview features, and may change at any time.
31
-
32
32
## Prerequisites
33
33
34
-
*[Create an Azure Machine Learning workspace](quickstart-create-resources.md).
34
+
*[Create an Azure Machine Learning workspace](../quickstart-create-resources.md).
35
35
36
-
*[Create a Synapse workspace in Azure portal](../synapse-analytics/quickstart-create-workspace.md).
36
+
*[Create a Synapse workspace in Azure portal](../../synapse-analytics/quickstart-create-workspace.md).
37
37
38
-
*[Create Apache Spark pool using Azure portal, web tools, or Synapse Studio](../synapse-analytics/quickstart-create-apache-spark-pool-studio.md)
38
+
*[Create Apache Spark pool using Azure portal, web tools, or Synapse Studio](../../synapse-analytics/quickstart-create-apache-spark-pool-studio.md)
39
39
40
40
* Install the [Azure Machine Learning Python SDK](/python/api/overview/azure/ml/intro)
> A managed identity, `system_assigned_identity_principal_id`, is created for each linked service. This managed identity must be granted the **Synapse Apache Spark Administrator** role of the Synapse workspace before you start your Synapse session. [Assign the Synapse Apache Spark Administrator role to the managed identity in the Synapse Studio](../synapse-analytics/security/how-to-manage-synapse-rbac-role-assignments.md).
77
+
> A managed identity, `system_assigned_identity_principal_id`, is created for each linked service. This managed identity must be granted the **Synapse Apache Spark Administrator** role of the Synapse workspace before you start your Synapse session. [Assign the Synapse Apache Spark Administrator role to the managed identity in the Synapse Studio](../../synapse-analytics/security/how-to-manage-synapse-rbac-role-assignments.md).
78
78
>
79
79
> To find the `system_assigned_identity_principal_id` of a specific linked service, use `LinkedService.get('<your-mlworkspace-name>', '<linked-service-name>')`.
80
80
@@ -145,7 +145,7 @@ Follow these steps:
145
145
1. Select **Attach**.
146
146
1. Select your Apache Spark pool from the list and provide a name.
147
147
1. This list identifies the available Synapse Spark pools that can be attached to your compute.
148
-
1. To create a new Synapse Spark pool, see [Create Apache Spark pool with the Synapse Studio](../synapse-analytics/quickstart-create-apache-spark-pool-portal.md)
148
+
1. To create a new Synapse Spark pool, see [Create Apache Spark pool with the Synapse Studio](../../synapse-analytics/quickstart-create-apache-spark-pool-portal.md)
149
149
1. Select **Attach selected**.
150
150
151
151
### Attach a pool with the Python SDK
@@ -188,7 +188,7 @@ ws.compute_targets['Synapse Spark pool alias']
188
188
189
189
## Next steps
190
190
191
-
*[How to data wrangle with Azure Synapse (preview)](v1/how-to-data-prep-synapse-spark-pool.md).
192
-
*[How to use Apache Spark in your machine learning pipeline with Azure Synapse (preview)](v1/how-to-use-synapsesparkstep.md)
193
-
*[Train a model](v1/how-to-set-up-training-targets.md).
194
-
*[How to securely integrate Azure Synapse and Azure Machine Learning workspaces](how-to-private-endpoint-integration-synapse.md).
191
+
*[How to data wrangle with Azure Synapse (preview)](how-to-data-prep-synapse-spark-pool.md).
192
+
*[How to use Apache Spark in your machine learning pipeline with Azure Synapse (preview)](how-to-use-synapsesparkstep.md)
193
+
*[Train a model](how-to-set-up-training-targets.md).
194
+
*[How to securely integrate Azure Synapse and Azure Machine Learning workspaces](../how-to-private-endpoint-integration-synapse.md).
#Customer intent: As a user of both Azure Machine Learning pipelines and Azure Synapse Analytics, I'd like to use Apache Spark for the data preparation of my pipeline
14
15
---
15
16
16
-
# How to use Apache Spark (powered by Azure Synapse Analytics) in your machine learning pipeline (preview)
17
+
# How to use Apache Spark (powered by Azure Synapse Analytics) in your machine learning pipeline (deprecated)
> The Azure Synapse Analytics integration with Azure Machine Learning available in Python SDK v1 is deprecated. Users can continue using Synapse workspace registered with Azure Machine Learning as a linked service. However, a new Synapse workspace can no longer be registered with Azure Machine Learning as a linked service. We recommend using Managed (Automatic) Synapse compute and attached Synapse Spark pools available in CLI v2 and Python SDK v2. Please see [https://aka.ms/aml-spark](https://aka.ms/aml-spark) for more details.
23
24
24
25
In this article, you'll learn how to use Apache Spark pools powered by Azure Synapse Analytics as the compute target for a data preparation step in an Azure Machine Learning pipeline. You'll learn how a single pipeline can use compute resources suited for the specific step, such as data preparation or training. You'll see how data is prepared for the Spark step and how it's passed to the next step.
25
26
26
-
27
-
28
27
## Prerequisites
29
28
30
29
* Create an [Azure Machine Learning workspace](../quickstart-create-resources.md) to hold all your pipeline resources.
@@ -35,10 +34,10 @@ In this article, you'll learn how to use Apache Spark pools powered by Azure Syn
35
34
36
35
## Link your Azure Machine Learning workspace and Azure Synapse Analytics workspace
37
36
38
-
You create and administer your Apache Spark pools in an Azure Synapse Analytics workspace. To integrate an Apache Spark pool with an Azure Machine Learning workspace, you must [link to the Azure Synapse Analytics workspace](../how-to-link-synapse-ml-workspaces.md).
37
+
You create and administer your Apache Spark pools in an Azure Synapse Analytics workspace. To integrate an Apache Spark pool with an Azure Machine Learning workspace, you must [link to the Azure Synapse Analytics workspace](how-to-link-synapse-ml-workspaces.md).
39
38
40
39
Once your Azure Machine Learning workspace and your Azure Synapse Analytics workspaces are linked, you can attach an Apache Spark pool via
* Azure Resource Manager (ARM) template (see this [Example ARM template](https://github.com/Azure/azure-quickstart-templates/blob/master/quickstarts/microsoft.machinelearningservices/machine-learning-linkedservice-create/azuredeploy.json)).
44
43
* You can use the command line to follow the ARM template, add the linked service, and attach the Apache Spark pool with the following code:
Copy file name to clipboardExpand all lines: articles/sentinel/notebooks-with-synapse-export-data.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -126,5 +126,5 @@ For more information, see:
126
126
-[Blog post: Export Historical Log Data from Microsoft Sentinel](https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/export-historical-log-data-from-microsoft-sentinel/ba-p/3413418)
127
127
-[Use Jupyter notebooks to hunt for security threats](notebooks.md)
128
128
-[Tutorial: Get started with Jupyter notebooks and MSTICPy in Microsoft Sentinel](notebook-get-started.md)
129
-
-[Link Azure Synapse Analytics and Azure Machine Learning workspaces and attach Apache Spark pools(preview)](../machine-learning/how-to-link-synapse-ml-workspaces.md)
129
+
-[Link Azure Synapse Analytics and Azure Machine Learning workspaces and attach Apache Spark pools(preview)](../machine-learning/v1/how-to-link-synapse-ml-workspaces.md)
130
130
-[Create your first Microsoft Sentinel notebook](https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/creating-your-first-microsoft-sentinel-notebook/ba-p/2977745) (Blog series)
0 commit comments