Merge pull request #203596 from jingyanjingyan/notebook

ttorble · web-flow · commit e447be7ef876 · 2022-07-01T11:54:59.000+01:00
Notebook
diff --git a/articles/synapse-analytics/spark/microsoft-spark-utilities.md b/articles/synapse-analytics/spark/microsoft-spark-utilities.md
@@ -1,14 +1,14 @@
 ---
 title: Introduction to Microsoft Spark utilities
 description: "Tutorial: MSSparkutils in Azure Synapse Analytics notebooks"
-author: ruixinxu 
-services: synapse-analytics 
-ms.service: synapse-analytics 
+author: ruixinxu
+services: synapse-analytics
+ms.service: synapse-analytics
 ms.topic: reference
 ms.subservice: spark
 ms.date: 09/10/2020
 ms.author: ruxu
-ms.reviewer: 
+ms.reviewer:
 zone_pivot_groups: programming-languages-spark-all-minus-sql
 ms.custom: subject-rbac-steps
 ---
@@ -19,9 +19,9 @@ Microsoft Spark Utilities (MSSparkUtils) is a builtin package to help you easily
 
 ## Pre-requisites
 
-### Configure access to Azure Data Lake Storage Gen2 
+### Configure access to Azure Data Lake Storage Gen2
 
-Synapse notebooks use Azure Active Directory (Azure AD) pass-through to access the ADLS Gen2 accounts. You need to be a **Storage Blob Data Contributor** to access the ADLS Gen2 account (or folder). 
+Synapse notebooks use Azure Active Directory (Azure AD) pass-through to access the ADLS Gen2 accounts. You need to be a **Storage Blob Data Contributor** to access the ADLS Gen2 account (or folder).
 
 Synapse pipelines use workspace's Managed Service Identity (MSI) to access the storage accounts. To use MSSparkUtils in your pipeline activities, your workspace identity needs to be **Storage Blob Data Contributor** to access the ADLS Gen2 account (or folder).
 
@@ -30,7 +30,7 @@ Follow these steps to make sure your Azure AD and workspace MSI have access to t
 1. Select the **Access control (IAM)** from the left panel.
 1. Select **Add** > **Add role assignment** to open the Add role assignment page.
 1. Assign the following role. For detailed steps, see [Assign Azure roles using the Azure portal](../../role-based-access-control/role-assignments-portal.md).
-    
+
     | Setting | Value |
     | --- | --- |
     | Role | Storage Blob Data Contributor |
@@ -41,14 +41,14 @@ Follow these steps to make sure your Azure AD and workspace MSI have access to t
     > The managed identity name is also the workspace name.
 
     ![Add role assignment page in Azure portal.](../../../includes/role-based-access-control/media/add-role-assignment-page.png)
- 
+
 1. Select **Save**.
 
 You can access data on ADLS Gen2 with Synapse Spark via the following URL:
 
 `abfss://<container_name>@<storage_account_name>.dfs.core.windows.net/<path>`
 
-### Configure access to Azure Blob Storage  
+### Configure access to Azure Blob Storage
 
 Synapse use [**Shared access signature (SAS)**](../../storage/common/storage-sas-overview.md) to access Azure Blob Storage. To avoid exposing SAS keys in the code, we recommend creating a new linked service in Synapse workspace to the Azure Blob Storage account you want to access.
 
@@ -60,7 +60,7 @@ Follow these steps to add a new linked service for an Azure Blob Storage account
 4. Select **Continue**.
 5. Select the Azure Blob Storage Account to access and configure the linked service name. Suggest using **Account key** for the **Authentication method**.
 6. Select **Test connection** to validate the settings are correct.
-7. Select **Create** first and click **Publish all** to save your changes. 
+7. Select **Create** first and click **Publish all** to save your changes.
 
 You can access data on Azure Blob Storage with Synapse Spark via following URL:
 
@@ -126,29 +126,29 @@ Console.WriteLine(wasbs_path);
 
 ```
 
-::: zone-end 
- 
+::: zone-end
+
 ###  Configure access to Azure Key Vault
 
-You can add an Azure Key Vault as a linked service to manage your credentials in Synapse. 
+You can add an Azure Key Vault as a linked service to manage your credentials in Synapse.
 Follow these steps to add an Azure Key Vault as a Synapse linked service:
 1. Open the [Azure Synapse Studio](https://web.azuresynapse.net/).
 2. Select **Manage** from the left panel and select **Linked services** under the **External connections**.
 3. Search **Azure Key Vault** in the **New linked Service** panel on the right.
 4. Select the Azure Key Vault Account to access and configure the linked service name.
 5. Select **Test connection** to validate the settings are correct.
-6. Select **Create** first and click **Publish all** to save your change. 
+6. Select **Create** first and click **Publish all** to save your change.
 
 Synapse notebooks use Azure active directory(Azure AD) pass-through to access Azure Key Vault. Synapse pipelines use workspace identity(MSI) to access Azure Key Vault. To make sure your code work both in notebook and in Synapse pipeline, we recommend granting secret access permission for both your Azure AD account and workspace identity.
 
 Follow these steps to grant secret access to your workspace identity:
-1. Open the [Azure portal](https://portal.azure.com/) and the Azure Key Vault you want to access. 
+1. Open the [Azure portal](https://portal.azure.com/) and the Azure Key Vault you want to access.
 2. Select the **Access policies** from the left panel.
-3. Select **Add Access Policy**: 
+3. Select **Add Access Policy**:
     - Choose **Key, Secret, & Certificate Management** as config template.
-    - Select **your Azure AD account** and **your workspace identity** (same as your workspace name) in the select principal or make sure it is already assigned. 
+    - Select **your Azure AD account** and **your workspace identity** (same as your workspace name) in the select principal or make sure it is already assigned.
 4. Select **Select** and **Add**.
-5. Select the **Save** button to commit changes.  
+5. Select the **Save** button to commit changes.
 
 ## File system utilities
 
@@ -429,29 +429,29 @@ Removes a file or a directory.
 :::zone pivot = "programming-language-python"
 
 ```python
-mssparkutils.fs.rm('file path', True) # Set the last parameter as True to remove all files and directories recursively 
+mssparkutils.fs.rm('file path', True) # Set the last parameter as True to remove all files and directories recursively
 ```
 ::: zone-end
 
 :::zone pivot = "programming-language-scala"
 
 ```scala
-mssparkutils.fs.rm("file path", true) // Set the last parameter as True to remove all files and directories recursively 
+mssparkutils.fs.rm("file path", true) // Set the last parameter as True to remove all files and directories recursively
 ```
 
 ::: zone-end
 
 :::zone pivot = "programming-language-csharp"
 
 ```csharp
-FS.Rm("file path", true) // Set the last parameter as True to remove all files and directories recursively 
+FS.Rm("file path", true) // Set the last parameter as True to remove all files and directories recursively
 ```
 
 ::: zone-end
 
 
 
-## Notebook utilities 
+## Notebook utilities
 
 :::zone pivot = "programming-language-csharp"
 
@@ -461,7 +461,7 @@ Not supported.
 
 :::zone pivot = "programming-language-python"
 
-You can use the MSSparkUtils Notebook Utilities to run a notebook or exit a notebook with a value. 
+You can use the MSSparkUtils Notebook Utilities to run a notebook or exit a notebook with a value.
 Run the following command to get an overview of the available methods:
 
 ```python
@@ -478,7 +478,7 @@ run(path: String, timeoutSeconds: int, arguments: Map): String -> This method ru
 ```
 
 ### Reference a notebook
-Reference a notebook and returns its exit value. You can run nesting function calls in a notebook interactively or in a pipeline. The notebook being referenced will run on the Spark pool of which notebook calls this function.  
+Reference a notebook and returns its exit value. You can run nesting function calls in a notebook interactively or in a pipeline. The notebook being referenced will run on the Spark pool of which notebook calls this function.
 
 ```python
 
@@ -497,13 +497,13 @@ After the run finished, you will see a snapshot link named '**View notebook run:
 ![Screenshot of a snap link python](./media/microsoft-spark-utilities/spark-utilities-run-notebook-snap-link-sample-python.png)
 
 ### Exit a notebook
-Exits a notebook with a value. You can run nesting function calls in a notebook interactively or in a pipeline. 
+Exits a notebook with a value. You can run nesting function calls in a notebook interactively or in a pipeline.
 
 - When you call an `exit()` function a notebook interactively, Azure Synapse will throw an exception, skip running subsequence cells, and keep Spark session alive.
 
-- When you orchestrate a notebook that calls an `exit()` function in a Synapse pipeline, Azure Synapse will return an exit value, complete the pipeline run, and stop the Spark session.  
+- When you orchestrate a notebook that calls an `exit()` function in a Synapse pipeline, Azure Synapse will return an exit value, complete the pipeline run, and stop the Spark session.
 
-- When you call an `exit()` function in a notebook being referenced, Azure Synapse will stop the further execution in the notebook being referenced, and continue to run next cells in the notebook that call the `run()` function. For example: Notebook1 has three cells and calls an `exit()` function in the second cell. Notebook2 has five cells and calls `run(notebook1)` in the third cell. When you run Notebook2, Notebook1 will be stopped at the second cell when hitting the `exit()` function. Notebook2 will continue to run its fourth cell and fifth cell. 
+- When you call an `exit()` function in a notebook being referenced, Azure Synapse will stop the further execution in the notebook being referenced, and continue to run next cells in the notebook that call the `run()` function. For example: Notebook1 has three cells and calls an `exit()` function in the second cell. Notebook2 has five cells and calls `run(notebook1)` in the third cell. When you run Notebook2, Notebook1 will be stopped at the second cell when hitting the `exit()` function. Notebook2 will continue to run its fourth cell and fifth cell.
 
 
 ```python
@@ -512,9 +512,9 @@ mssparkutils.notebook.exit("value string")
 
 For example:
 
-**Sample1** notebook locates under **folder/** with following two cells: 
+**Sample1** notebook locates under **folder/** with following two cells:
 - cell 1 defines an **input** parameter with default value set to 10.
-- cell 2 exits the notebook with **input** as exit value. 
+- cell 2 exits the notebook with **input** as exit value.
 
 ![Screenshot of a sample notebook](./media/microsoft-spark-utilities/spark-utilities-run-notebook-sample.png)
 
@@ -548,7 +548,7 @@ Sample1 run success with input is 20
 
 :::zone pivot = "programming-language-scala"
 
-You can use the MSSparkUtils Notebook Utilities to run a notebook or exit a notebook with a value. 
+You can use the MSSparkUtils Notebook Utilities to run a notebook or exit a notebook with a value.
 Run the following command to get an overview of the available methods:
 
 ```scala
@@ -565,7 +565,7 @@ run(path: String, timeoutSeconds: int, arguments: Map): String -> This method ru
 ```
 
 ### Reference a notebook
-Reference a notebook and returns its exit value. You can run nesting function calls in a notebook interactively or in a pipeline. The notebook being referenced will run on the Spark pool of which notebook calls this function.  
+Reference a notebook and returns its exit value. You can run nesting function calls in a notebook interactively or in a pipeline. The notebook being referenced will run on the Spark pool of which notebook calls this function.
 
 ```scala
 
@@ -585,13 +585,13 @@ After the run finished, you will see a snapshot link named '**View notebook run:
 
 
 ### Exit a notebook
-Exits a notebook with a value. You can run nesting function calls in a notebook interactively or in a pipeline. 
+Exits a notebook with a value. You can run nesting function calls in a notebook interactively or in a pipeline.
 
 - When you call an `exit()` function a notebook interactively, Azure Synapse will throw an exception, skip running subsequence cells, and keep Spark session alive.
 
-- When you orchestrate a notebook that calls an `exit()` function in a Synapse pipeline, Azure Synapse will return an exit value, complete the pipeline run, and stop the Spark session.  
+- When you orchestrate a notebook that calls an `exit()` function in a Synapse pipeline, Azure Synapse will return an exit value, complete the pipeline run, and stop the Spark session.
 
-- When you call an `exit()` function in a notebook being referenced, Azure Synapse will stop the further execution in the notebook being referenced, and continue to run next cells in the notebook that call the `run()` function. For example: Notebook1 has three cells and calls an `exit()` function in the second cell. Notebook2 has five cells and calls `run(notebook1)` in the third cell. When you run Notebook2, Notebook1 will be stopped at the second cell when hitting the `exit()` function. Notebook2 will continue to run its fourth cell and fifth cell. 
+- When you call an `exit()` function in a notebook being referenced, Azure Synapse will stop the further execution in the notebook being referenced, and continue to run next cells in the notebook that call the `run()` function. For example: Notebook1 has three cells and calls an `exit()` function in the second cell. Notebook2 has five cells and calls `run(notebook1)` in the third cell. When you run Notebook2, Notebook1 will be stopped at the second cell when hitting the `exit()` function. Notebook2 will continue to run its fourth cell and fifth cell.
 
 
 ```python
@@ -600,9 +600,9 @@ mssparkutils.notebook.exit("value string")
 
 For example:
 
-**Sample1** notebook locates under **mssparkutils/folder/** with following two cells: 
+**Sample1** notebook locates under **mssparkutils/folder/** with following two cells:
 - cell 1 defines an **input** parameter with default value set to 10.
-- cell 2 exits the notebook with **input** as exit value. 
+- cell 2 exits the notebook with **input** as exit value.
 
 ![Screenshot of a sample notebook](./media/microsoft-spark-utilities/spark-utilities-run-notebook-sample.png)
 
@@ -640,7 +640,7 @@ Sample1 run success with input is 20
 
 ## Credentials utilities
 
-You can use the MSSparkUtils Credentials Utilities to get the access tokens of linked services and manage secrets in Azure Key Vault. 
+You can use the MSSparkUtils Credentials Utilities to get the access tokens of linked services and manage secrets in Azure Key Vault.
 
 Run the following command to get an overview of the available methods:
 
@@ -681,7 +681,7 @@ putSecret(akvName, secretName, secretValue): puts AKV secret for a given akvName
 
 ### Get token
 
-Returns Azure AD token for a given audience, name (optional). The table below list all the available audience types: 
+Returns Azure AD token for a given audience, name (optional). The table below list all the available audience types:
 
 |Audience Type|Audience key|
 |--|--|
@@ -748,7 +748,7 @@ Credentials.IsValidToken("your token")
 
 ### Get connection string or credentials for linked service
 
-Returns connection string or credentials for linked service. 
+Returns connection string or credentials for linked service.
 
 :::zone pivot = "programming-language-python"
 
@@ -804,7 +804,7 @@ Credentials.GetSecret("azure key vault name","secret name","linked service name"
 
 ### Get secret using user credentials
 
-Returns Azure Key Vault secret for a given Azure Key Vault name, secret name, and linked service name using user credentials. 
+Returns Azure Key Vault secret for a given Azure Key Vault name, secret name, and linked service name using user credentials.
 
 :::zone pivot = "programming-language-python"
 
@@ -873,7 +873,7 @@ Puts Azure Key Vault secret for a given Azure Key Vault name, secret name, and l
 
 ### Put secret using user credentials
 
-Puts Azure Key Vault secret for a given Azure Key Vault name, secret name, and linked service name using user credentials. 
+Puts Azure Key Vault secret for a given Azure Key Vault name, secret name, and linked service name using user credentials.
 
 ```python
 mssparkutils.credentials.putSecret('azure key vault name','secret name','secret value')
@@ -884,7 +884,7 @@ mssparkutils.credentials.putSecret('azure key vault name','secret name','secret
 
 ### Put secret using user credentials
 
-Puts Azure Key Vault secret for a given Azure Key Vault name, secret name, and linked service name using user credentials. 
+Puts Azure Key Vault secret for a given Azure Key Vault name, secret name, and linked service name using user credentials.
 
 ```scala
 mssparkutils.credentials.putSecret("azure key vault name","secret name","secret value")
@@ -901,7 +901,7 @@ mssparkutils.credentials.putSecret("azure key vault name","secret name","secret
 ::: zone-end -->
 
 
-## Environment utilities 
+## Environment utilities
 
 Run following commands to get an overview of the available methods:
 
@@ -1125,11 +1125,36 @@ mssparkutils.runtime.context
 ```
 ::: zone-end
 
+## Session management
+
+### Stop an interactive session
+
+Instead of manually click stop button, sometimes it's more convenient to stop an interactive session by calling an API in the code. For such cases, we provide an API `mssparkutils.session.stop()` to support stopping the interactive session via code, it's available for Scala and Python.
+
+:::zone pivot = "programming-language-python"
+
+```python
+mssparkutils.session.stop()
+```
+::: zone-end
+
+:::zone pivot = "programming-language-scala"
+
+```scala
+mssparkutils.session.stop()
+```
+::: zone-end
+
+`mssparkutils.session.stop()` API will stop the current interactive session asynchronously in the background, it stops the Spark session and release resources occupied by the session so they are available to other sessions in the same pool.
+
+> [!NOTE]
+> We don't recommend call language built-in APIs like `sys.exit` in Scala or `sys.exit()` in Python in your code, because such APIs just
+> kill the interpreter process, leaving Spark session alive and resources not released.
+
 ## Next steps
 
 - [Check out Synapse sample notebooks](https://github.com/Azure-Samples/Synapse/tree/master/Notebooks)
 - [Quickstart: Create an Apache Spark pool in Azure Synapse Analytics using web tools](../quickstart-apache-spark-notebook.md)
 - [What is Apache Spark in Azure Synapse Analytics](apache-spark-overview.md)
 - [Azure Synapse Analytics](../index.yml)
 - [How to use file mount/unmount API in Synapse](./synapse-file-mount-api.md)
-
diff --git a/articles/zone-pivot-groups.yml b/articles/zone-pivot-groups.yml
@@ -715,12 +715,12 @@ groups:
   title: Programming languages
   prompt: Choose a language
   pivots:
-  - id: programming-language-csharp
-    title: C#
-  - id: programming-language-scala
-    title: Scala
   - id: programming-language-python
     title: Python
+  - id: programming-language-scala
+    title: Scala
+  - id: programming-language-csharp
+    title: C#
 - id: aml-control-methods
   # Owner: gopalv
   title: Azure Machine Learning control plane
