Merge pull request #520 from fbsolo-ms1/freshness-update-for-YP

prmerger-automator[bot] · web-flow · commit 5f5b4250bf36 · 2024-09-26T22:44:39.000Z
Freshness update for interactive-data-wrangling-with-apache-spark-azure-ml.md . . .
diff --git a/articles/machine-learning/interactive-data-wrangling-with-apache-spark-azure-ml.md b/articles/machine-learning/interactive-data-wrangling-with-apache-spark-azure-ml.md
@@ -8,81 +8,84 @@ ms.reviewer: yogipandey
 ms.service: azure-machine-learning
 ms.subservice: mldata
 ms.topic: how-to
-ms.date: 10/05/2023
-ms.custom: template-how-to 
+ms.date: 09/26/2024
+ms.custom: template-how-to
 ---
 
 # Interactive Data Wrangling with Apache Spark in Azure Machine Learning
 
-Data wrangling becomes one of the most important steps in machine learning projects. The Azure Machine Learning integration, with Azure Synapse Analytics, provides access to an Apache Spark pool - backed by Azure Synapse - for interactive data wrangling using Azure Machine Learning Notebooks.
+Data wrangling becomes one of the most important aspects of machine learning projects. The integration of Azure Machine Learning integration with Azure Synapse Analytics provides access to an Apache Spark pool - backed by Azure Synapse - for interactive data wrangling that uses Azure Machine Learning Notebooks.
 
-In this article, you'll learn how to perform data wrangling using
+In this article, you learn how to handle data wrangling using
 
 - Serverless Spark compute
 - Attached Synapse Spark pool
 
 ## Prerequisites
 - An Azure subscription; if you don't have an Azure subscription, [create a free account](https://azure.microsoft.com/free) before you begin.
-- An Azure Machine Learning workspace. See [Create workspace resources](./quickstart-create-resources.md).
-- An Azure Data Lake Storage (ADLS) Gen 2 storage account. See [Create an Azure Data Lake Storage (ADLS) Gen 2 storage account](/azure/storage/blobs/create-data-lake-storage-account).
-- (Optional): An Azure Key Vault. See [Create an Azure Key Vault](/azure/key-vault/general/quick-create-portal).
-- (Optional): A Service Principal. See [Create a Service Principal](/azure/active-directory/develop/howto-create-service-principal-portal).
-- [(Optional): An attached Synapse Spark pool in the Azure Machine Learning workspace](./how-to-manage-synapse-spark-pool.md).
+- An Azure Machine Learning workspace. Visit [Create workspace resources](./quickstart-create-resources.md) for more information.
+- An Azure Data Lake Storage (ADLS) Gen 2 storage account. Visit [Create an Azure Data Lake Storage (ADLS) Gen 2 storage account](/azure/storage/blobs/create-data-lake-storage-account) for more information.
+- (Optional): An Azure Key Vault. Visit [Create an Azure Key Vault](/azure/key-vault/general/quick-create-portal) for more information.
+- (Optional): A Service Principal. Visit [Create a Service Principal](/azure/active-directory/develop/howto-create-service-principal-portal) for more information.
+- (Optional): [An attached Synapse Spark pool in the Azure Machine Learning workspace](./how-to-manage-synapse-spark-pool.md).
 
 Before you start your data wrangling tasks, learn about the process of storing secrets
 
 - Azure Blob storage account access key
 - Shared Access Signature (SAS) token
 - Azure Data Lake Storage (ADLS) Gen 2 service principal information
 
-in the Azure Key Vault. You also need to know how to handle role assignments in the Azure storage accounts. The following sections review these concepts. Then, we'll explore the details of interactive data wrangling using the Spark pools in Azure Machine Learning Notebooks.
+in the Azure Key Vault. You also need to know how to handle role assignments in the Azure storage accounts. The following sections in this document describe these concepts. Then, we explore the details of interactive data wrangling, using the Spark pools in Azure Machine Learning Notebooks.
 
 > [!TIP]
-> To learn about Azure storage account role assignment configuration, or if you access data in your storage accounts using user identity passthrough, see [Add role assignments in Azure storage accounts](./apache-spark-environment-configuration.md#add-role-assignments-in-azure-storage-accounts).
+> To learn about Azure storage account role assignment configuration, or if you access data in your storage accounts using user identity passthrough, visit [Add role assignments in Azure storage accounts](./apache-spark-environment-configuration.md#add-role-assignments-in-azure-storage-accounts) for more information.
 
 ## Interactive Data Wrangling with Apache Spark
 
-Azure Machine Learning offers serverless Spark compute, and [attached Synapse Spark pool](./how-to-manage-synapse-spark-pool.md), for interactive data wrangling with Apache Spark in Azure Machine Learning Notebooks. The serverless Spark compute doesn't require creation of resources in the Azure Synapse workspace. Instead, a fully managed serverless Spark compute becomes directly available in the Azure Machine Learning Notebooks. Using a serverless Spark compute is the easiest approach to access a Spark cluster in Azure Machine Learning.
+For interactive data wrangling with Apache Spark in Azure Machine Learning Notebooks, Azure Machine Learning offers serverless Spark compute and [attached Synapse Spark pool](./how-to-manage-synapse-spark-pool.md). The serverless Spark compute doesn't require creation of resources in the Azure Synapse workspace. Instead, a fully managed serverless Spark compute becomes directly available in the Azure Machine Learning Notebooks. Use of a serverless Spark compute is the easiest way to access a Spark cluster in Azure Machine Learning.
 
 ### Serverless Spark compute in Azure Machine Learning Notebooks
 
 A serverless Spark compute is available in Azure Machine Learning Notebooks by default. To access it in a notebook, select **Serverless Spark Compute** under **Azure Machine Learning Serverless Spark** from the **Compute** selection menu.
 
-The Notebooks UI also provides options for Spark session configuration, for the serverless Spark compute. To configure a Spark session:
+The Notebooks UI also provides options for Spark session configuration for the serverless Spark compute. To configure a Spark session:
 
 1. Select **Configure session** at the top of the screen.
-2. Select **Apache Spark version** from the dropdown menu.
+1. Select **Apache Spark version** from the dropdown menu.
     > [!IMPORTANT]
     > Azure Synapse Runtime for Apache Spark: Announcements
     > * Azure Synapse Runtime for Apache Spark 3.2:
     >   * EOLA Announcement Date: July 8, 2023
     >   * End of Support Date: July 8, 2024. After this date, the runtime will be disabled.
-    > * For continued support and optimal performance, we advise that you migrate to Apache Spark 3.3.
-3. Select **Instance type** from the dropdown menu. The following instance types are currently supported:
+    > * Apache Spark 3.3:
+    >   * EOLA Announcement Date: July 12, 2024
+    >   * End of Support Date: March 31, 2025. After this date, the runtime will be disabled.
+    > * For continued support and optimal performance, we advise migration to **Apache Spark 3.4**
+1. Select **Instance type** from the dropdown menu. These types are currently supported:
     - `Standard_E4s_v3`
     - `Standard_E8s_v3`
     - `Standard_E16s_v3`
     - `Standard_E32s_v3`
     - `Standard_E64s_v3`
-4. Input a Spark **Session timeout** value, in minutes.
-5. Select whether to **Dynamically allocate executors**
-6. Select the number of **Executors** for the Spark session.
-7. Select **Executor size** from the dropdown menu.
-8. Select **Driver size** from the dropdown menu.
-9. To use a Conda file to configure a Spark session, check the **Upload conda file** checkbox. Then, select **Browse**, and choose the Conda file with the Spark session configuration you want.
-10. Add **Configuration settings** properties, input values in the **Property** and **Value** textboxes, and select **Add**.
-11. Select **Apply**.
-12. Select **Stop session** in the **Configure new session?** pop-up.
+1. Input a Spark **Session timeout** value, in minutes.
+1. Select whether or not you want to **Dynamically allocate executors**
+1. Select the number of **Executors** for the Spark session.
+1. Select **Executor size** from the dropdown menu.
+1. Select **Driver size** from the dropdown menu.
+1. To use a Conda file to configure a Spark session, check the **Upload conda file** checkbox. Then, select **Browse**, and choose the Conda file with the Spark session configuration you want.
+1. Add **Configuration settings** properties, input values in the **Property** and **Value** textboxes, and select **Add**.
+1. Select **Apply**.
+1. In the **Configure new session?** pop-up, select **Stop session**.
 
 The session configuration changes persist and become available to another notebook session that is started using the serverless Spark compute.
 
 > [!TIP]
 >
-> If you use session-level Conda packages, you can [improve](./apache-spark-azure-ml-concepts.md#improving-session-cold-start-time-while-using-session-level-conda-packages) the Spark session *cold start* time if you set the configuration variable `spark.hadoop.aml.enable_cache` to true. A session cold start with session level Conda packages typically takes 10 to 15 minutes when the session starts for the first time. However, subsequent session cold starts with the configuration variable set to true typically take three to five minutes.
+> If you use session-level Conda packages, you can [improve](./apache-spark-azure-ml-concepts.md#improving-session-cold-start-time-while-using-session-level-conda-packages) the Spark session *cold start* time if you set the configuration variable `spark.hadoop.aml.enable_cache` to **true**. A session cold start with session level Conda packages typically takes 10 to 15 minutes when the session starts for the first time. However, subsequent session cold starts with the configuration variable set to true typically take three to five minutes.
 
 ### Import and wrangle data from Azure Data Lake Storage (ADLS) Gen 2
 
-You can access and wrangle data stored in Azure Data Lake Storage (ADLS) Gen 2 storage accounts with `abfss://` data URIs following one of the two data access mechanisms:
+You can access and wrangle data stored in Azure Data Lake Storage (ADLS) Gen 2 storage accounts with `abfss://` data URIs. To do this, you must follow one of the two data access mechanisms:
 
 - User identity passthrough
 - Service principal-based data access
@@ -127,9 +130,9 @@ To start interactive data wrangling with the user identity passthrough:
 To wrangle data by access through a service principal:
 
 1. Verify that the service principal has **Contributor** and **Storage Blob Data Contributor** [role assignments](./apache-spark-environment-configuration.md#add-role-assignments-in-azure-storage-accounts) in the Azure Data Lake Storage (ADLS) Gen 2 storage account.
-2. [Create Azure Key Vault secrets](./apache-spark-environment-configuration.md#store-azure-storage-account-credentials-as-secrets-in-azure-key-vault) for the service principal tenant ID, client ID and client secret values.
-3. Select **Serverless Spark compute** under **Azure Machine Learning Serverless Spark** from the **Compute** selection menu, or select an attached Synapse Spark pool under **Synapse Spark pools** from the **Compute** selection menu.
-4. To set the service principal tenant ID, client ID and client secret in the configuration, and execute the following code sample.
+1. [Create Azure Key Vault secrets](./apache-spark-environment-configuration.md#store-azure-storage-account-credentials-as-secrets-in-azure-key-vault) for the service principal tenant ID, client ID and client secret values.
+1. In the **Compute** selection menu, select **Serverless Spark compute** under **Azure Machine Learning Serverless Spark**. You can also select an attached Synapse Spark pool under **Synapse Spark pools** from the **Compute** selection menu.
+1. Set the service principal tenant ID, client ID and client secret values in the configuration, and execute the following code sample.
      - The `get_secret()` call in the code depends on name of the Azure Key Vault, and the names of the Azure Key Vault secrets created for the service principal tenant ID, client ID and client secret. Set these corresponding property name/values in the configuration:
        - Client ID property: `fs.azure.account.oauth2.client.id.<STORAGE_ACCOUNT_NAME>.dfs.core.windows.net`
        - Client secret property: `fs.azure.account.oauth2.client.secret.<STORAGE_ACCOUNT_NAME>.dfs.core.windows.net`
@@ -169,18 +172,18 @@ To wrangle data by access through a service principal:
         )
         ```
 
-5. Import and wrangle data using data URI in format `abfss://<FILE_SYSTEM_NAME>@<STORAGE_ACCOUNT_NAME>.dfs.core.windows.net/<PATH_TO_DATA>` as shown in the code sample, using the Titanic data.
+1. Using the Titanic data, import and the wrangle data using the data URI in the `abfss://<FILE_SYSTEM_NAME>@<STORAGE_ACCOUNT_NAME>.dfs.core.windows.net/<PATH_TO_DATA>` format, as shown in the code sample.
 
 ### Import and wrangle data from Azure Blob storage
 
 You can access Azure Blob storage data with either the storage account access key or a shared access signature (SAS) token. You should [store these credentials in the Azure Key Vault as a secret](./apache-spark-environment-configuration.md#store-azure-storage-account-credentials-as-secrets-in-azure-key-vault), and set them as properties in the session configuration.
 
 To start interactive data wrangling:
 1. At the Azure Machine Learning studio left panel, select **Notebooks**.
-1. Select **Serverless Spark compute** under **Azure Machine Learning Serverless Spark** from the **Compute** selection menu, or select an attached Synapse Spark pool under **Synapse Spark pools** from the **Compute** selection menu.
+1. In the **Compute** selection menu, select **Serverless Spark compute** under **Azure Machine Learning Serverless Spark**. You can also select an attached Synapse Spark pool under **Synapse Spark pools** from the **Compute** selection menu.
 1. To configure the storage account access key or a shared access signature (SAS) token for data access in Azure Machine Learning Notebooks:
 
-     - For the access key, set property `fs.azure.account.key.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net` as shown in this code snippet:
+     - For the access key, set the `fs.azure.account.key.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net` property, as shown in this code snippet:
 
         ```python
         from pyspark.sql import SparkSession
@@ -192,7 +195,7 @@ To start interactive data wrangling:
             "fs.azure.account.key.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net", access_key
         )
         ```
-     - For the SAS token, set property `fs.azure.sas.<BLOB_CONTAINER_NAME>.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net` as shown in this code snippet:
+     - For the SAS token, set the `fs.azure.sas.<BLOB_CONTAINER_NAME>.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net` property, as shown in this code snippet:
    
         ```python
         from pyspark.sql import SparkSession
@@ -206,7 +209,7 @@ To start interactive data wrangling:
         )
         ```
         > [!NOTE]
-        > The `get_secret()` calls in the above code snippets require the name of the Azure Key Vault, and the names of the secrets created for the Azure Blob storage account access key or SAS token
+        > The `get_secret()` calls in the earlier code snippets require the name of the Azure Key Vault, and the names of the secrets created for the Azure Blob storage account access key or SAS token.
 
 2. Execute the data wrangling code in the same notebook. Format the data URI as `wasbs://<BLOB_CONTAINER_NAME>@<STORAGE_ACCOUNT_NAME>.blob.core.windows.net/<PATH_TO_DATA>`, similar to what this code snippet shows:
 
@@ -239,7 +242,7 @@ To start interactive data wrangling:
 To access data from [Azure Machine Learning Datastore](how-to-datastore.md), define a path to data on the datastore with [URI format](how-to-create-data-assets.md?tabs=cli#create-data-assets) `azureml://datastores/<DATASTORE_NAME>/paths/<PATH_TO_DATA>`. To wrangle data from an Azure Machine Learning Datastore in a Notebooks session interactively:
 
 1. Select **Serverless Spark compute** under **Azure Machine Learning Serverless Spark** from the **Compute** selection menu, or select an attached Synapse Spark pool under **Synapse Spark pools** from the **Compute** selection menu.
-2. This code sample shows how to read and wrangle Titanic data from an Azure Machine Learning Datastore, using `azureml://` datastore URI, `pyspark.pandas` and `pyspark.ml.feature.Imputer`.
+1. This code sample shows how to read and wrangle Titanic data from an Azure Machine Learning Datastore, using `azureml://` datastore URI, `pyspark.pandas`, and `pyspark.ml.feature.Imputer`.
 
     ```python
     import pyspark.pandas as pd
@@ -271,7 +274,7 @@ The Azure Machine Learning datastores can access data using Azure storage accoun
 - SAS token
 - service principal
 
-or provide credential-less data access. Depending on the datastore type and the underlying Azure storage account type, select an appropriate authentication mechanism to ensure data access. This table summarizes the authentication mechanisms to access data in the Azure Machine Learning datastores:
+or they use credential-less data access. Depending on the datastore type and the underlying Azure storage account type, select an appropriate authentication mechanism to ensure data access. This table summarizes the authentication mechanisms to access data in the Azure Machine Learning datastores:
 
 |Storage account type|Credential-less data access|Data access mechanism|Role assignments|
 | ------------------------ | ------------------------ | ------------------------ | ------------------------ |
@@ -288,7 +291,7 @@ The default file share is mounted to both serverless Spark compute and attached
 
 :::image type="content" source="media/interactive-data-wrangling-with-apache-spark-azure-ml/default-file-share.png" alt-text="Screenshot showing use of a file share.":::
 
-In Azure Machine Learning studio, files in the default file share are shown in the directory tree under the **Files** tab. Notebook code can directly access files stored in this file share with `file://` protocol, along with the absolute path of the file, without more configurations. This code snippet shows how to access a file stored on the default file share:
+In Azure Machine Learning studio, files in the default file share are shown in the directory tree under the **Files** tab. Notebook code can directly access files stored in this file share with the `file://` protocol, along with the absolute path of the file, without more configurations. This code snippet shows how to access a file stored on the default file share:
 
 ```python
 import os
