Merge pull request #204217 from shuaijunye/UpdateForLMDocs

Documentation updates

Author: ShannonLeavitt

articles/machine-learning/how-to-data-prep-synapse-spark-pool.md

@@ -96,7 +96,7 @@ env.register(workspace=ws)
 To begin data preparation with the Apache Spark pool and your custom environment, specify the Apache Spark pool name and which environment to use during the Apache Spark session. Furthermore, you can provide your subscription ID, the machine learning workspace resource group, and the name of the machine learning workspace.
 
 >[!IMPORTANT]
-> Make sure to [Allow session level packages](../synapse-analytics/spark/apache-spark-manage-python-packages.md#session-scoped-packages) is enabled in the linked Synapse workspace.
+> Make sure to [Allow session level packages](../synapse-analytics/spark/apache-spark-manage-session-packages.md#session-scoped-python-packages) is enabled in the linked Synapse workspace.
 >
 >![enable session level packages](media/how-to-data-prep-synapse-spark-pool/enable-session-level-package.png)
 
@@ -301,7 +301,7 @@ script_run_config = ScriptRunConfig(source_directory = './code',
                                     run_config = run_config)
 ```
 
-For more infomation about  `run_config.spark.configuration` and general Spark configuration, see [SparkConfiguration Class](/python/api/azureml-core/azureml.core.runconfig.sparkconfiguration) and [Apache Spark's configuration documentation](https://spark.apache.org/docs/latest/configuration.html).
+For more information about  `run_config.spark.configuration` and general Spark configuration, see [SparkConfiguration Class](/python/api/azureml-core/azureml.core.runconfig.sparkconfiguration) and [Apache Spark's configuration documentation](https://spark.apache.org/docs/latest/configuration.html).
 
 Once your `ScriptRunConfig` object is set up, you can submit the run.
 

articles/synapse-analytics/.openpublishing.redirection.synapse-analytics.json

@@ -694,6 +694,16 @@
             "source_path_from_root": "/articles/sql-data-warehouse/what-is-a-data-warehouse-unit-dwu-cdwu.md",
             "redirect_url": "/azure/synapse-analytics/sql-data-warehouse/what-is-a-data-warehouse-unit-dwu-cdwu",
             "redirect_document_id": true
+          },
+          {
+            "source_path_from_root": "/articles/synapse-analytics/spark/apache-spark-manage-python-packages.md",
+            "redirect_url": "/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries",
+            "redirect_document_id": true
+          },
+          {
+            "source_path_from_root": "/articles/synapse-analytics/spark/apache-spark-manage-scala-packages.md",
+            "redirect_url": "/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries",
+            "redirect_document_id": false
           }
     ]
 }

articles/synapse-analytics/spark/apache-spark-azure-portal-add-libraries.md

@@ -4,9 +4,8 @@ description: Learn how to add and manage libraries used by Apache Spark in Azure
 author: shuaijunye
 ms.service: synapse-analytics
 ms.topic: how-to
-ms.date: 06/08/2022
+ms.date: 07/07/2022
 ms.author: shuaijunye
-ms.reviewer: sngun 
 ms.subservice: spark
 ms.custom: kr2b-contr-experiment
 ---
@@ -23,23 +22,29 @@ You might need to update your serverless Apache Spark pool environment for vario
 - Your team has built a custom package that you need available in your Apache Spark pool.
 
 To make third party or locally built code available to your applications, install a library onto one of your serverless Apache Spark pools or notebook session.
+
+> [!IMPORTANT]
+>
+> - There are three levels of package installing on Synapse Analytics -- default level, Spark pool level and session level.
+> - Apache Spark in Azure Synapse Analytics has a full Anaconda install plus extra libraries served as the default level installation which is fully managed by Synapse. The Spark pool level packages can be used by all running Artifacts, e.g., Notebook and Spark job definition attaching the corresponding Spark pool. The session level installation will create an environment for the specific Notebook session, the change of session level libraries will not be persisted between sessions.
+> - You can upload custom libraries and a specific version of an open-source library that you would like to use in your Azure Synapse Analytics Workspace. The workspace packages can be installed in your Spark pools.
+> - To be noted, the pool level library management can take certain amount of time depending on the size of packages and the complexity of required dependencies. The session level installation is suggested with experimental and quick iterative scenarios.
   
 ## Default Installation
 
-Apache Spark in Azure Synapse Analytics has a full Anaconda install plus extra libraries. The full libraries list can be found at [Apache Spark version support](apache-spark-version-support.md).
+Default packages include a full Anaconda install plus extra commonly used libraries. The full libraries list can be found at [Apache Spark version support](apache-spark-version-support.md).
 
 When a Spark instance starts, these libraries are included automatically. More packages can be added at the Spark pool level or session level.
 
 ## Workspace packages
 
 When your team develops custom applications or models, you might develop various code artifacts like *.whl* or *.jar* files to package your code.
 
-In Synapse, workspace packages can be custom or private *.whl* or *.jar* files. You can upload these packages to your workspace and later assign them to a specific Spark pool. Once assigned, these workspace packages are installed automatically on all Spark pool sessions.
+In Synapse, workspace packages can be custom or private *.whl* or *.jar* files. You can upload these packages to your workspace and later assign them to a specific serverless Apache Spark pool. Once assigned, these workspace packages are installed automatically on all Spark pool sessions.
 
-To learn more about how to manage workspace libraries, see the following articles:
+To learn more about how to manage workspace libraries, see the following article:
 
-- [Python workspace packages: ](./apache-spark-manage-python-packages.md#install-wheel-files) Upload Python *.whl* files as a workspace package and later add these packages to specific serverless Apache Spark pools.
-- [Scala/Java workspace packages: ](./apache-spark-manage-scala-packages.md#workspace-packages) Upload Scala and Java *.jar* files as a workspace package and later add these packages to specific serverless Apache Spark pools.
+- [Manage workspace packages](./apache-spark-manage-workspace-packages.md)
 
 ## Pool packages
 
@@ -49,7 +54,7 @@ Using the Azure Synapse Analytics pool management capabilities, you can configur
 
 Currently, pool management is only supported for Python. For Python, Synapse Spark pools use Conda to install and manage Python package dependencies. When specifying your pool-level libraries, you can now provide a *requirements.txt* or an *environment.yml* file. This environment configuration file is used every time a Spark instance is created from that Spark pool.
 
-To learn more about these capabilities, see [Python pool management](./apache-spark-manage-python-packages.md#pool-libraries).
+To learn more about these capabilities, see [Manage Spark pool packages](./apache-spark-manage-pool-packages.md).
 
 > [!IMPORTANT]
 >
@@ -65,9 +70,18 @@ Session-scoped packages allow users to define package dependencies at the start
 
 To learn more about how to manage session-scoped packages, see the following articles:
 
-- [Python session packages: ](./apache-spark-manage-python-packages.md) At the start of a session, provide a Conda *environment.yml* to install more Python packages from popular repositories. 
-- [Scala/Java session packages: ](./apache-spark-manage-scala-packages.md) At the start of your session, provide a list of *.jar* files to install using `%%configure`.
+- [Python session packages: ](./apache-spark-manage-session-packages.md#session-scoped-python-packages) At the start of a session, provide a Conda *environment.yml* to install more Python packages from popular repositories. 
+- [Scala/Java session packages: ](./apache-spark-manage-session-packages.md#session-scoped-java-or-scala-packages) At the start of your session, provide a list of *.jar* files to install using `%%configure`.
 
-## Next steps
+## Manage your packages outside Synapse Analytics UI
 
+If your team want to manage the libraries without visiting the package management UIs, you have the options to manage the workspace packages and pool level package updates through Azure PowerShell cmdlets or REST APIs for Synapse Analytics.
+
+To learn more about Azure PowerShell cmdlets and package management REST APIs, see the following articles:
+
+- Azure PowerShell cmdlets for Synapse Analytics: [Manage your Spark pool libraries through Azure PowerShell cmdlets](apache-spark-manage-packages-outside-ui.md#manage-packages-through-azure-powershell-cmdlets)
+- Package management REST APIs: [Manage your Spark pool libraries through REST APIs](apache-spark-manage-packages-outside-ui.md#manage-packages-through-rest-apis)
+
+## Next steps
 - View the default libraries: [Apache Spark version support](apache-spark-version-support.md)
+- Troubleshoot library installation errors: [Troubleshoot library errors](apache-spark-troubleshoot-library-errors.md)

articles/synapse-analytics/spark/apache-spark-custom-conda-channel.md

@@ -1,12 +1,11 @@
 ---
 title: Create custom Conda channel for package management
 description: Learn how to create a custom Conda channel for package management
-author: midesa
+author: shuaijunye
 ms.service: synapse-analytics
 ms.topic: conceptual
-ms.date: 08/11/2021
-ms.author: midesa
-ms.reviewer: sngun 
+ms.date: 07/07/2022
+ms.author: shuaijunye
 ms.subservice: spark
 ---
 
@@ -125,9 +124,8 @@ conda env create --file sample.yml
 source activate env 
 conda list 
 ```
-Now that you've verified your custom channel, you can use the [Python pool management](./apache-spark-manage-python-packages.md) process to update the libraries on your Apache Spark pool.
+Now that you've verified your custom channel, you can use the [Python pool management](./apache-spark-manage-pool-packages.md#manage-packages-from-synapse-studio-or-azure-portal) process to update the libraries on your Apache Spark pool.
 
 ## Next steps
 - View the default libraries: [Apache Spark version support](apache-spark-version-support.md)
-- Manage Python packages: [Python package management](./apache-spark-manage-python-packages.md)
-
+- Manage Session level Python packages: [Python package management on Notebook Session](./apache-spark-manage-session-packages.md#session-scoped-python-packages)

articles/synapse-analytics/spark/apache-spark-manage-packages-outside-UI.md

@@ -0,0 +1,96 @@
+---
+title: Manage packages outside Synapse Analytics Studio UIs
+description: Learn how to manage packages using Azure PowerShell cmdlets or REST APIs
+author: shuaijunye
+ms.service: synapse-analytics
+ms.topic: conceptual
+ms.date: 07/07/2022
+ms.author: shuaijunye
+ms.subservice: spark
+---
+
+# Manage packages outside Synapse Analytics Studio UIs 
+
+You may want to manage your libraries for your serverless Apache Spark pools without going into the Synapse Analytics UI pages. For example, you may find that:
+
+- you develop a custom package and want to upload it to your workspace and use it in your Spark pool. And you want to finish the steps on your local tools without visiting the package management UIs.
+- you are updating your packages through the CI/CD process
+
+In this article, we'll provide a general guide to help you managing libraries through Azure PowerShell cmdlets or REST APIs.
+
+## Manage packages through Azure PowerShell cmdlets
+
+### Add new libraries
+1. [New-AzSynapseWorkspacePackage](https://docs.microsoft.com/powershell/module/az.synapse/new-azsynapseworkspacepackage) command can be used to **upload new libraries to workspace**.
+
+    ```powershell
+    New-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace -Package ".\ContosoPackage.whl"
+    ```
+
+2. The combination of [New-AzSynapseWorkspacePackage](https://docs.microsoft.com/powershell/module/az.synapse/new-azsynapseworkspacepackage) and [Update-AzSynapseSparkPool](https://docs.microsoft.com/powershell/module/az.synapse/update-azsynapsesparkpool) commands can be used to **upload new libraries to workspace** and **attach the library to a Spark pool**.
+
+    ```powershell
+    $package = New-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace -Package ".\ContosoPackage.whl"
+    Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Add -Package $package
+    ```
+
+3. If you want to attach an **existing workspace library** to your Spark pool, please refer to the command combination of [Get-AzSynapseWorkspacePackage](https://docs.microsoft.com/powershell/module/az.synapse/get-azsynapseworkspacepackage) and [Update-AzSynapseSparkPool](https://docs.microsoft.com/powershell/module/az.synapse/update-azsynapsesparkpool).
+
+    ```powershell
+    $packages = Get-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace
+    Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Add -Package $packages
+    ```
+
+### Remove libraries
+1. In order to **remove a installed package** from your Spark pool, please refer to the command combination of [Get-AzSynapseWorkspacePackage](https://docs.microsoft.com/powershell/module/az.synapse/get-azsynapseworkspacepackage) and [Update-AzSynapseSparkPool](https://docs.microsoft.com/powershell/module/az.synapse/update-azsynapsesparkpool).
+
+    ```powershell
+    $package = Get-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace -Name ContosoPackage
+    Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Remove -Package $package
+    ```
+
+2. You can also retrieve a Spark pool and **remove all attached workspace libraries** from the pool by calling [Get-AzSynapseSparkPool](https://docs.microsoft.com/powershell/module/az.synapse/get-azsynapsesparkpool) and [Update-AzSynapseSparkPool](https://docs.microsoft.com/powershell/module/az.synapse/update-azsynapsesparkpool) commands. 
+    ```powershell
+    $pool = Get-AzSynapseSparkPool -ResourceGroupName ContosoResourceGroup -WorkspaceName ContosoWorkspace -Name ContosoSparkPool
+    $pool | Update-AzSynapseSparkPool -PackageAction Remove -Package $pool.WorkspacePackages
+    ```
+
+For more Azure PowerShell cmdlets capabilities, please refer to [Azure PowerShell cmdlets for Azure Synapse Analytics](https://docs.microsoft.com/powershell/module/az.synapse).
+
+
+## Manage packages through REST APIs
+
+### Manage the workspace packages
+With the ability of REST APIs, you can add/delete packages or list all uploaded files of your workspace. See the full supported APIs, please refer to [Overview of workspace library APIs](https://docs.microsoft.com/rest/api/synapse/data-plane/library).
+
+
+### Manage the Spark pool packages
+You can leverage the [Spark pool REST API](https://docs.microsoft.com/rest/api/synapse/big-data-pools/create-or-update) to attach or remove your custom or open source libraries to your Spark pools. 
+
+1. For custom libraries, please specify the list of custom files as the **customLibraries** property in request body. 
+
+    ```json
+    "customLibraries": [
+        {
+            "name": "samplejartestfile.jar",
+            "path": "<workspace-name>/libraries/<jar-name>.jar",
+            "containerName": "prep",
+            "uploadedTimestamp": "1970-01-01T00:00:00Z",
+            "type": "jar"
+        }
+    ]
+    ```
+
+2. You can also update your Spark pool libraries by specifying the **libraryRequirements** property in request body. 
+
+    ```json
+    "libraryRequirements": {
+          "content": "",
+          "filename": "requirements.txt"
+    }
+    ```
+
+## Next steps
+- View the default libraries: [Apache Spark version support](apache-spark-version-support.md)
+- Manage Spark pool level packages through Synapse Studio portal: [Python package management on Notebook Session](./apache-spark-manage-session-packages.md#session-scoped-python-packages)
+