Merge branch 'release-ignite-arcadia' of https://github.com/MicrosoftDocs/azure-docs-pr into release-ignite-arcadia

Author: julieMSFT

articles/synapse-analytics/security/synapse-workspace-managed-identity.md

@@ -17,7 +17,7 @@ In this article, you'll learn about managed identity in Azure Synapse workspace.
 
 Managed identity for Azure resources is a feature of Azure Active Directory. The feature provides Azure services with an automatically managed identity in Azure AD. You can use the Managed Identity capability to authenticate to any service that support Azure AD authentication.
 
-Managed identities for Azure resources are the new name for the service formerly known as Managed Service Identity (MSI). See [Managed Identities](https://docs.microsoft.com/azure/active-directory/managed-identities-azure-resources/overview) to learn more.
+Managed identities for Azure resources are the new name for the service formerly known as Managed Service Identity (MSI). See [Managed Identities](../../active-directory/managed-identities-azure-resources/overview.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json) to learn more.
 
 ## Azure Synapse workspace managed identity
 

articles/synapse-analytics/spark/apache-spark-development-using-notebooks.md

@@ -387,5 +387,5 @@ Using the following keystroke shortcuts, you can more easily navigate and run co
 
 ## Next steps
 
-- [.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
+- [.NET for Apache Spark documentation](/dotnet/spark?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
 - [Azure Synapse Analytics](https://docs.microsoft.com/azure/synapse-analytics)

articles/synapse-analytics/spark/apache-spark-history-server.md

@@ -233,5 +233,6 @@ Input/output data using Resilient Distributed Datasets (RDDs) does not show in d
 
 ## Next steps
 
-- [.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
-- [Azure Synapse Analytics](../overview-what-is.md)
+- [Azure Synapse Analytics](../overview-what-is.md)
+- [.NET for Apache Spark documentation](/dotnet/spark?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
+

articles/synapse-analytics/spark/apache-spark-job-definitions.md

@@ -18,29 +18,31 @@ This tutorial demonstrates how to use the Azure Synapse Analytics to create Spar
 * View job details after submission.
 
 In this tutorial, you learn how to:
+
 > [!div class="checklist"]
+>
 > * Develop and submit a Spark job definition on a Synapse Spark pool.
 > * View job details after submission.
 
 ## Prerequisites
 
-* An Azure Synapse Analytics workspace. For instructions, see [Create an Azure Synapse Analytics workspace](https://docs.microsoft.com/azure/machine-learning/how-to-manage-workspace#create-a-workspace).
+* An Azure Synapse Analytics workspace. For instructions, see [Create an Azure Synapse Analytics workspace](../../machine-learning/how-to-manage-workspace.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json#create-a-workspace).
 
 ## Get started
 
-Before submitting a Spark job definition, you need to be the Storage Blob Data Owner of the ADLS Gen2 filesystem you want to work with. If you aren't, you need to add the permission manually. 
+Before submitting a Spark job definition, you need to be the Storage Blob Data Owner of the ADLS Gen2 filesystem you want to work with. If you aren't, you need to add the permission manually.
 
 ### Scenario 1: Add permission
 
-1. Open [Microsoft Azure](https://ms.portal.azure.com), then open Storage account.     
+1. Open [Microsoft Azure](https://ms.portal.azure.com), then open Storage account.
 
 2. Click **Containers**, then create a **File system**. This tutorial uses `sparkjob`.
 
     ![Click submit button to submit spark job definition](./media/apache-spark-job-definitions/open-azure-container.png)
 
-    ![The Spark Submission dialog box](./media/apache-spark-job-definitions/create-new-filesystem.png)   
+    ![The Spark Submission dialog box](./media/apache-spark-job-definitions/create-new-filesystem.png)
 
-3. Open `sparkjob`, click **Access Control(IAM)**, then click **Add** and select **Add role assignment**. 
+3. Open `sparkjob`, click **Access Control(IAM)**, then click **Add** and select **Add role assignment**.
 
     ![Click submit button to submit spark job definition](./media/apache-spark-job-definitions/add-role-assignment-01.png)
 
@@ -50,7 +52,6 @@ Before submitting a Spark job definition, you need to be the Storage Blob Data O
 
     ![Click submit button to submit spark job definition](./media/apache-spark-job-definitions/verify-user-role.png)
 
-
 ### Scenario 2: Prepare folder structure
 
 Before submitting a Spark job definition, one job you need to do is uploading files to ADLS Gen2 and preparing folder structure there. We use Storage node in Synapse Studio to store files.
@@ -86,7 +87,7 @@ Before submitting a Spark job definition, one job you need to do is uploading fi
     |Main class name| The fully qualified identifier or the main class that is in the main definition file.|
     |Command-line arguments| Optional arguments to the job.|
     |Reference files| Additional files used for reference in the main definition file. You can select **Upload file** to upload the file to a storage account.|
-    |Spark pool| The job will be submitted to the selected Spark pool.| 
+    |Spark pool| The job will be submitted to the selected Spark pool.|
     |Spark version| Version of Spark that the Spark pool is running.|
     |Executors| Number of executors to be given in the specified Spark pool for the job.|
     |Executor size| Number of cores and memory to be used for executors given in the specified Spark pool for the job.|  
@@ -102,7 +103,7 @@ Before submitting a Spark job definition, one job you need to do is uploading fi
     |Main definition file| The main file used for the job. Select a PY file from your storage. You can select **Upload file** to upload the file to a storage account.|
     |Command-line arguments| Optional arguments to the job.|
     |Reference files| Additional files used for reference in the main definition file. You can select **Upload file** to upload the file to a storage account.|
-    |Spark pool| The job will be submitted to the selected Spark pool.| 
+    |Spark pool| The job will be submitted to the selected Spark pool.|
     |Spark version| Version of Spark that the Spark pool is running.|
     |Executors| Number of executors to be given in the specified Spark pool for the job.|
     |Executor size| Number of cores and memory to be used for executors given in the specified Spark pool for the job.|  
@@ -119,7 +120,7 @@ Before submitting a Spark job definition, one job you need to do is uploading fi
     |Main executable file| The main executable file in the main definition ZIP file.|
     |Command-line arguments| Optional arguments to the job.|
     |Reference files| Additional files needed by the worker nodes for executing the .NET for Spark application that isn't included in the main definition ZIP file(that is, dependent jars, additional user-defined function DLLs, and other config files). You can select **Upload file** to upload the file to a storage account.|
-    |Spark pool| The job will be submitted to the selected Spark pool.| 
+    |Spark pool| The job will be submitted to the selected Spark pool.|
     |Spark version| Version of Spark that the Spark pool is running.|
     |Executors| Number of executors to be given in the specified Spark pool for the job.|
     |Executor size| Number of cores and memory to be used for executors given in the specified Spark pool for the job.|  
@@ -135,18 +136,17 @@ Before submitting a Spark job definition, one job you need to do is uploading fi
 
 After creating a Spark job definition, you can submit it to a Synapse Spark pool. Make sure you've gone through steps in **Get-started** before trying samples in this part.
 
-
-### Scenario 1: Submit Spark job definition 
+### Scenario 1: Submit Spark job definition
 
 1. Open a spark job definition window by clicking it.
 
-      ![Open spark job definition to submit ](./media/apache-spark-job-definitions/open-spark-definition.png)     
+      ![Open spark job definition to submit ](./media/apache-spark-job-definitions/open-spark-definition.png)
 
 2. Click **submit** icon to submit your project to the selected Spark Pool. You can click **Spark monitoring URL** tab to see the LogQuery of the Spark application.
 
     ![Click submit button to submit spark job definition](./media/apache-spark-job-definitions/submit-spark-definition.png)
 
-    ![The Spark Submission dialog box](./media/apache-spark-job-definitions/submit-definition-result.png)   
+    ![The Spark Submission dialog box](./media/apache-spark-job-definitions/submit-definition-result.png)
 
 ### Scenario 2: View Spark job running progress
 
@@ -160,15 +160,13 @@ After creating a Spark job definition, you can submit it to a Synapse Spark pool
 
 ### Scenario 3: Check output file
 
- 1. Click **Data**, then select **Storage accounts**. After a successful run, you can go to the ADLS Gen2 storage and check outputs are generated. 
+ 1. Click **Data**, then select **Storage accounts**. After a successful run, you can go to the ADLS Gen2 storage and check outputs are generated.
 
     ![View output file](./media/apache-spark-job-definitions/view-output-file.png)
 
-
-
 ## Next steps
 
 This tutorial demonstrated how to use the Azure Synapse Analytics to create Spark job definitions, and then submit them to a Synapse Spark pool. Next you can use Azure Synapse Analytics to create Power BI datasets and manage Power BI data. 
 
 - [Connect to data in Power BI Desktop](https://docs.microsoft.com/power-bi/desktop-quickstart-connect-to-data)
-- [Visualize with Power BI](/sql-data-warehouse/sql-data-warehouse-get-started-visualize-with-power-bi)
+- [Visualize with Power BI](../sql-data-warehouse/sql-data-warehouse-get-started-visualize-with-power-bi.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)

articles/synapse-analytics/spark/apache-spark-machine-learning-mllib-notebook.md

@@ -291,9 +291,9 @@ After you have finished running the application, shut down the notebook to relea
 
 ## Next steps
 
-- [.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
+- [.NET for Apache Spark documentation](/dotnet/spark?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
 - [Azure Synapse Analytics](https://docs.microsoft.com/azure/synapse-analytics)
 - [Apache Spark official documentation](https://spark.apache.org/docs/latest/)
 
 >[!NOTE]
-> Some of the official Apache Spark documentation relies on using the Spark console, which is not available on Azure Synapse Spark. Use the [notebook](../spark/apache-spark-notebook-create-spark-use-sql.md) or [IntelliJ](../spark/intellij-tool-synapse.md) experiences instead.
+> Some of the official Apache Spark documentation relies on using the Spark console, which is not available on Azure Synapse Spark. Use the [notebook](../spark/apache-spark-notebook-create-spark-use-sql.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json) or [IntelliJ](../spark/intellij-tool-synapse.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json) experiences instead.

articles/synapse-analytics/spark/apache-spark-notebook-create-spark-use-sql.md

@@ -128,8 +128,8 @@ To ensure the Spark instance is shut down, end any connected sessions(notebooks)
 
 In this quickstart, you learned how to create a Synapse Analytics Apache Spark pool and run a basic Spark SQL query.
 
-- [.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
 - [Azure Synapse Analytics](../overview-what-is.md)
+- [.NET for Apache Spark documentation](/dotnet/spark?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
 - [Apache Spark official documentation](https://spark.apache.org/docs/latest/)
 
 >[!NOTE]

articles/synapse-analytics/spark/apache-spark-what-is-delta-lake.md

@@ -36,5 +36,5 @@ For more information, see [Delta Lake Project](https://lfprojects.org).
 
 ## Next steps
 
-- [.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
+- [.NET for Apache Spark documentation](/dotnet/spark?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
 - [Azure Synapse Analytics](https://docs.microsoft.com/azure/synapse-analytics)

articles/synapse-analytics/spark/intellij-tool-synapse.md

@@ -28,9 +28,9 @@ In this tutorial, you learn how to:
 ## Prerequisites
 
 - [IntelliJ IDEA Community Version](https://www.jetbrains.com/idea/download/download-thanks.html?platform=windows&code=IIC).
-- Azure toolkit plugin 3.27.0-2019.2 – Install from [IntelliJ Plugin repository](https://docs.microsoft.com/java/azure/intellij/azure-toolkit-for-intellij-installation?view=azure-java-stable)
+- Azure toolkit plugin 3.27.0-2019.2 – Install from [IntelliJ Plugin repository](/java/azure/intellij/azure-toolkit-for-intellij-installation?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
 - [JDK (Version 1.8)](https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html).
-- Scala Plugin – Install from [IntelliJ Plugin repository](https://docs.microsoft.com/azure/hdinsight/spark/apache-spark-intellij-tool-plugin#install-scala-plugin-for-intellij-idea).
+- Scala Plugin – Install from [IntelliJ Plugin repository](/hdinsight/spark/apache-spark-intellij-tool-plugin#install-scala-plugin-for-intellij-idea.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json).
 - This prerequisite is only for Windows users.
 
   While you're running the local Spark Scala application on a Windows computer, you might get an exception, as explained in [SPARK-2356](https://issues.apache.org/jira/browse/SPARK-2356). The exception occurs because WinUtils.exe is missing on Windows.
@@ -138,7 +138,7 @@ After creating a Scala application, you can remotely run it.
     |Main class name|The default value is the main class from the selected file. You can change the class by selecting the ellipsis(**...**) and choosing another class.|
     |Job configurations|You can change the default key and values. For more information, see [Apache Livy REST API](https://livy.incubator.apache.org./docs/latest/rest-api.html).|
     |Command line arguments|You can enter arguments separated by space for the main class if needed.|
-    |Referenced Jars and Referenced Files|You can enter the paths for the referenced Jars and files if any. You can also browse files in the Azure virtual file system, which currently only supports ADLS Gen 2 cluster. For more information: [Apache Spark Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-environment) and [How to upload resources to cluster](https://docs.microsoft.com/azure/storage/blobs/storage-quickstart-blobs-storage-explorer).|
+    |Referenced Jars and Referenced Files|You can enter the paths for the referenced Jars and files if any. You can also browse files in the Azure virtual file system, which currently only supports ADLS Gen 2 cluster. For more information: [Apache Spark Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-environment) and [How to upload resources to cluster](../../storage/blobs/storage-quickstart-blobs-storage-explorer.md?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json).|
     |Job Upload Storage|Expand to reveal additional options.|
     |Storage Type|Select **Use Azure Blob to upload** from the drop-down list.|
     |Storage Account|Enter your storage account.|

articles/synapse-analytics/spark/spark-dotnet.md

@@ -32,18 +32,11 @@ Visit the tutorial to learn how to use Azure Synapse Analytics to [create Apache
 
    **On Linux:**
 
-   ```bash
-   cd mySparkApp
-   foo@bar:~/path/to/app$ dotnet publish -c Release -f netcoreapp3.0 -r ubuntu.16.04-x64
-   ```
-
-2. Do the following tasks to zip your published app files so that you can easily upload them to Azure Synapse.
-
-   **On Windows:**
+### .NET for Apache Spark in Azure Synapse Analytics notebooks
 
-   Navigate to *mySparkApp/bin/Release/netcoreapp3.0/ubuntu.16.04-x64*. Then, right-click on **Publish** folder and select **Send to > Compressed (zipped) folder**. Name the new folder **publish.zip**.
+When creating a new notebook, you choose a language kernel that you wish to express your business logic. There is kernel support for several languages, including C#.
 
-   **On Linux, run the following command:**
+To use .NET for Apache Spark in your Azure Synapse Analytics notebook, select **.NET Spark (C#)** as your kernel and attach the notebook to an existing Spark pool.
 
    ```bash
    zip -r publish.zip
@@ -69,14 +62,14 @@ The following features are available when you use .NET for Apache Spark in the A
 * Simple C# statements (such as assignments, printing to console, throwing exceptions, and so on).
 * Multi-line C# code blocks (such as if statements, foreach loops, class definitions, and so on).
 * Access to the standard C# library (such as System, LINQ, Enumerables, and so on).
-* Support for [C# 8.0 language features](https://docs.microsoft.com/dotnet/csharp/whats-new/csharp-8).
+* Support for [C# 8.0 language features](/dotnet/csharp/whats-new/csharp-8?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json).
 * 'spark' as a pre-defined variable to give you access to your Apache Spark session.
 * Support for defining [.NET user-defined functions that can run within Apache Spark](https://github.com/dotnet/spark/blob/master/examples/Microsoft.Spark.CSharp.Examples/Sql).
-*  Support for visualizing output from your Spark jobs using different charts (such as line, bar, or histogram) and layouts (such as single, overlaid, and so on) using the `XPlot.Plotly` library. 
+* Support for visualizing output from your Spark jobs using different charts (such as line, bar, or histogram) and layouts (such as single, overlaid, and so on) using the `XPlot.Plotly` library.
 * Ability to include NuGet packages into your C# notebook.
 
 ## Next steps
 
-* [.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
+* [.NET for Apache Spark documentation](/dotnet/spark?toc=/azure/synapse-analytics/toc.json&bc=/azure/synapse-analytics/breadcrumb/toc.json)
 * [Azure Synapse Analytics](https://docs.microsoft.com/azure/synapse-analytics)
 * [.NET Interactive](https://devblogs.microsoft.com/dotnet/creating-interactive-net-documentation/)

articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-continuous-integration-and-deployment.md

@@ -40,7 +40,7 @@ At this point, you have a simple environment where any check-in to your source c
 
 ## Continuous deployment with the Azure SQL Data Warehouse (or Database) deployment task
 
-1. Add a new task using the [Azure SQL Database deployment task](https://docs.microsoft.com/azure/devops/pipelines/targets/azure-sqldb?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json) and fill in the required fields to connect to your target data warehouse. When this task runs, the DACPAC generated from the previous build process is deployed to the target data warehouse. You can also use the [Azure SQL Data Warehouse deployment task](https://marketplace.visualstudio.com/items?itemName=ms-sql-dw.SQLDWDeployment).
+1. Add a new task using the [Azure SQL Database deployment task](/devops/pipelines/targets/azure-sqldb?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json) and fill in the required fields to connect to your target data warehouse. When this task runs, the DACPAC generated from the previous build process is deployed to the target data warehouse. You can also use the [Azure SQL Data Warehouse deployment task](https://marketplace.visualstudio.com/items?itemName=ms-sql-dw.SQLDWDeployment).
 
       ![Deployment Task](./media/sql-data-warehouse-continuous-integration-and-deployment/4-deployment-task.png "Deployment Task")