Merge pull request #228175 from shuaijunye/updateForPythonInline

Update the description for Python inline installation

Author: PMEds28

articles/synapse-analytics/spark/apache-spark-azure-portal-add-libraries.md

@@ -4,7 +4,7 @@ description: Learn how to add and manage libraries used by Apache Spark in Azure
 author: shuaijunye
 ms.service: synapse-analytics
 ms.topic: how-to
-ms.date: 11/03/2022
+ms.date: 02/20/2023
 ms.author: shuaijunye
 ms.subservice: spark
 ms.custom: kr2b-contr-experiment
@@ -25,19 +25,22 @@ To make third-party or locally built code available to your applications, instal
 
 ## Overview of package levels
 
-There are three levels of packages installed on Azure Synapse Analytics: 
+There are three levels of packages installed on Azure Synapse Analytics:
 
-- **Default**: Default packages include a full Anaconda installation, plus extra commonly used libraries. For a full list of libraries, see [Apache Spark version support](apache-spark-version-support.md). 
+- **Default**: Default packages include a full Anaconda installation, plus extra commonly used libraries. For a full list of libraries, see [Apache Spark version support](apache-spark-version-support.md).
 
-  When a Spark instance starts, these libraries are included automatically. You can add more packages at the other levels. 
-- **Spark pool**: All running artifacts can use packages at the Spark pool level. For example, you can attach notebook and Spark job definitions to corresponding Spark pools. 
+  When a Spark instance starts, these libraries are included automatically. You can add more packages at the other levels.
+- **Spark pool**: All running artifacts can use packages at the Spark pool level. For example, you can attach notebook and Spark job definitions to corresponding Spark pools.
 
   You can upload custom libraries and a specific version of an open-source library that you want to use in your Azure Synapse Analytics workspace. The workspace packages can be installed in your Spark pools.
 - **Session**: A session-level installation creates an environment for a specific notebook session. The change of session-level libraries isn't persisted between sessions.
 
 > [!NOTE]
-> Pool-level library management can take time, depending on the size of the packages and the complexity of required dependencies. We recommend the session-level installation for experimental and quick iterative scenarios.
-  
+>
+> - Pool-level library management can take time, depending on the size of the packages and the complexity of required dependencies. We recommend the session-level installation for experimental and quick iterative scenarios.
+> - The pool-level library management will produce a stable dependency for running your Notebooks and Spark job definitions. Installing the library to your Spark pool is highly recommended for the pipeline runs.
+> - Session level library management can help you with fast iteration or dealing with the frequent changes of library. However, the stability of session level installation is not promised. Also, in-line commands like %pip and %conda are disabled in pipeline run. Managing library in Notebook session is recommended during the developing phase.
+
 ## Manage workspace packages
 
 When your team develops custom applications or models, you might develop various code artifacts like *.whl*, *.jar*, or *tar.gz* files to package your code.
@@ -52,17 +55,16 @@ In some cases, you might want to standardize the packages that are used on an Ap
 
 By using the pool management capabilities of Azure Synapse Analytics, you can configure the default set of libraries to install on a serverless Apache Spark pool. These libraries are installed on top of the [base runtime](./apache-spark-version-support.md).
 
-Currently, pool management is supported only for Python. For Python, Azure Synapse Spark pools use Conda to install and manage Python package dependencies. 
-
-When you're specifying pool-level libraries, you can now provide a *requirements.txt* or *environment.yml* file. This environment configuration file is used every time a Spark instance is created from that Spark pool.
+For Python libraries, Azure Synapse Spark pools use Conda to install and manage Python package dependencies. You can specify the pool-level Python libraries by providing a *requirements.txt* or *environment.yml* file. This environment configuration file is used every time a Spark instance is created from that Spark pool. You can also attach the workspace packages to your pools.
 
 To learn more about these capabilities, see [Manage Spark pool packages](./apache-spark-manage-pool-packages.md).
 
 > [!IMPORTANT]
+>
 > - If the package that you're installing is large or takes a long time to install, it might affect the Spark instance's startup time.
 > - Altering the PySpark, Python, Scala/Java, .NET, or Spark version is not supported.
 
-## Manage dependencies for DEP-enabled Azure Synapse Spark pools
+### Manage dependencies for DEP-enabled Azure Synapse Spark pools
 
 > [!NOTE]
 > Installing packages from a public repo is not supported within [DEP-enabled workspaces](../security/workspace-data-exfiltration-protection.md). Instead, upload all your dependencies as workspace libraries and install them to your Spark pool.
@@ -81,7 +83,7 @@ If you're having trouble identifying required dependencies, follow these steps:
       source activate synapse-env
    ```
 
-1. Run the following script to identify the required dependencies. 
+2. Run the following script to identify the required dependencies. 
 The script can be used to pass your *requirements.txt* file, which has all the packages and versions that you intend to install in the Spark 3.1 or Spark 3.2 pool. It will print the names of the *new* wheel files/dependencies for your input library requirements. 
 
    ```python
@@ -91,6 +93,7 @@ The script can be used to pass your *requirements.txt* file, which has all the p
       pip install -r <input-user-req.txt> > pip_output.txt
       cat pip_output.txt | grep "Using cached *"
    ```
+
    > [!NOTE]
    > This script will list only the dependencies that are not already present in the Spark pool by default.
 
@@ -102,13 +105,14 @@ Session-scoped packages allow users to define package dependencies at the start
 
 To learn more about how to manage session-scoped packages, see the following articles:
 
-- [Python session packages](./apache-spark-manage-session-packages.md#session-scoped-python-packages): At the start of a session, provide a Conda *environment.yml* file to install more Python packages from popular repositories.
+- [Python session packages](./apache-spark-manage-session-packages.md#session-scoped-python-packages): At the start of a session, provide a Conda *environment.yml* file to install more Python packages from popular repositories. Or you can use %pip and %conda commands to manage libraries in the Notebook code cells.
 
 - [Scala/Java session packages](./apache-spark-manage-session-packages.md#session-scoped-java-or-scala-packages): At the start of your session, provide a list of *.jar* files to install by using `%%configure`.
 
 - [R session packages](./apache-spark-manage-session-packages.md#session-scoped-r-packages-preview): Within your session, you can install packages across all nodes within your Spark pool by using `install.packages` or `devtools`.
 
-## Manage your packages outside the Azure Synapse Analytics UI
+
+## Automate the library management process through Azure PowerShell cmdlets and REST APIs
 
 If your team wants to manage libraries without visiting the package management UIs, you have the option to manage the workspace packages and pool-level package updates through Azure PowerShell cmdlets or REST APIs for Azure Synapse Analytics.
 

articles/synapse-analytics/spark/apache-spark-manage-packages-outside-UI.md

@@ -4,12 +4,12 @@ description: Learn how to manage packages using Azure PowerShell cmdlets or REST
 author: shuaijunye
 ms.service: synapse-analytics
 ms.topic: conceptual
-ms.date: 07/07/2022
+ms.date: 02/23/2023
 ms.author: shuaijunye
 ms.subservice: spark
 ---
 
-# Manage packages outside Synapse Analytics Studio UIs 
+# Automate the library management process through Azure PowerShell cmdlets and REST APIs
 
 You may want to manage your libraries for your serverless Apache Spark pools without going into the Synapse Analytics UI pages. For example, you may find that:
 
@@ -21,6 +21,7 @@ In this article, we'll provide a general guide to help you managing libraries th
 ## Manage packages through Azure PowerShell cmdlets
 
 ### Add new libraries
+
 1. [New-AzSynapseWorkspacePackage](/powershell/module/az.synapse/new-azsynapseworkspacepackage) command can be used to **upload new libraries to workspace**.
 
     ```powershell
@@ -42,29 +43,31 @@ In this article, we'll provide a general guide to help you managing libraries th
     ```
 
 ### Remove libraries
+
 1. In order to **remove a installed package** from your Spark pool, please refer to the command combination of [Get-AzSynapseWorkspacePackage](/powershell/module/az.synapse/get-azsynapseworkspacepackage) and [Update-AzSynapseSparkPool](/powershell/module/az.synapse/update-azsynapsesparkpool).
 
     ```powershell
     $package = Get-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace -Name ContosoPackage
     Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Remove -Package $package
     ```
 
-2. You can also retrieve a Spark pool and **remove all attached workspace libraries** from the pool by calling [Get-AzSynapseSparkPool](/powershell/module/az.synapse/get-azsynapsesparkpool) and [Update-AzSynapseSparkPool](/powershell/module/az.synapse/update-azsynapsesparkpool) commands. 
+2. You can also retrieve a Spark pool and **remove all attached workspace libraries** from the pool by calling [Get-AzSynapseSparkPool](/powershell/module/az.synapse/get-azsynapsesparkpool) and [Update-AzSynapseSparkPool](/powershell/module/az.synapse/update-azsynapsesparkpool) commands.
+
     ```powershell
     $pool = Get-AzSynapseSparkPool -ResourceGroupName ContosoResourceGroup -WorkspaceName ContosoWorkspace -Name ContosoSparkPool
     $pool | Update-AzSynapseSparkPool -PackageAction Remove -Package $pool.WorkspacePackages
     ```
 
 For more Azure PowerShell cmdlets capabilities, please refer to [Azure PowerShell cmdlets for Azure Synapse Analytics](/powershell/module/az.synapse).
 
-
 ## Manage packages through REST APIs
 
 ### Manage the workspace packages
-With the ability of REST APIs, you can add/delete packages or list all uploaded files of your workspace. See the full supported APIs, please refer to [Overview of workspace library APIs](/rest/api/synapse/data-plane/library).
 
+With the ability of REST APIs, you can add/delete packages or list all uploaded files of your workspace. See the full supported APIs, please refer to [Overview of workspace library APIs](/rest/api/synapse/data-plane/library).
 
 ### Manage the Spark pool packages
+
 You can leverage the [Spark pool REST API](/rest/api/synapse/big-data-pools/create-or-update) to attach or remove your custom or open source libraries to your Spark pools. 
 
 1. For custom libraries, please specify the list of custom files as the **customLibraries** property in request body. 
@@ -91,5 +94,6 @@ You can leverage the [Spark pool REST API](/rest/api/synapse/big-data-pools/crea
     ```
 
 ## Next steps
+
 - View the default libraries: [Apache Spark version support](apache-spark-version-support.md)
 - Manage Spark pool level packages through Synapse Studio portal: [Python package management on Notebook Session](./apache-spark-manage-session-packages.md#session-scoped-python-packages)