Merge branch 'master' of https://github.com/MicrosoftDocs/azure-docs-pr into ds-issue46663

Author: davidsmatlak

articles/hdinsight/TOC.yml

@@ -372,6 +372,8 @@
           href: ./spark/apache-spark-run-machine-learning-automl.md
     - name: Troubleshoot
       items:
+      - name: Can't create Jupyter notebook
+        href: ./spark/troubleshoot-jupyter-notebook-convert.md
       - name: OutOfMemoryError exception
         href: ./spark/apache-spark-troubleshoot-outofmemory.md
       - name: Apache Spark job fails - NoClassDefFoundError
@@ -396,6 +398,8 @@
         href: ./spark/apache-spark-troubleshoot-event-log-requestbodytoolarge.md
       - name: Debug Apache Spark jobs
         href: ./spark/apache-spark-job-debugging.md
+      - name: Debug WASB file operations
+        href: ./spark/troubleshoot-debug-wasb.md
       - name: Use IntelliJ to debug Apache Spark job
         href: ./spark/apache-spark-intellij-tool-debug-remotely-through-ssh.md
       - name: Apache Spark troubleshooting

articles/hdinsight/hdinsight-hadoop-architecture.md

@@ -5,9 +5,9 @@ author: ashishthaps
 ms.author: ashishth
 ms.reviewer: jasonh
 ms.service: hdinsight
-ms.custom: hdinsightactive
 ms.topic: conceptual
-ms.date: 10/28/2019
+ms.custom: hdinsightactive
+ms.date: 02/07/2020
 ---
 
 # Apache Hadoop architecture in HDInsight
@@ -42,6 +42,27 @@ All HDInsight cluster types deploy YARN. The ResourceManager is deployed for hig
 
 ![Apache YARN on Azure HDInsight](./media/hdinsight-hadoop-architecture/apache-yarn-on-hdinsight.png)
 
+## Soft delete
+
+To undelete a file from your Storage Account, see:
+
+### Azure Storage
+
+* [Soft delete for Azure Storage blobs](../storage/blobs/storage-blob-soft-delete.md)
+* [Undelete Blob](https://docs.microsoft.com/rest/api/storageservices/undelete-blob)
+
+### Azure Data Lake Storage Gen 1
+
+[Restore-AzDataLakeStoreDeletedItem](https://docs.microsoft.com/powershell/module/az.datalakestore/restore-azdatalakestoredeleteditem)
+
+### Azure Data Lake Storage Gen 2
+
+[Known issues with Azure Data Lake Storage Gen2](../storage/blobs/data-lake-storage-known-issues.md)
+
+## Trash purging
+
+The `fs.trash.interval` property from **HDFS** > **Advanced core-site** should remain at the default value `0` because you shouldn't store any data on the local file system. This value doesn't affect remote storage accounts(WASB, ADLS GEN1, ABFS)
+
 ## Next steps
 
 * [Use MapReduce in Apache Hadoop on HDInsight](hadoop/hdinsight-use-mapreduce.md)

articles/hdinsight/spark/apache-spark-shell.md

@@ -7,7 +7,7 @@ ms.reviewer: jasonh
 ms.service: hdinsight
 ms.topic: conceptual
 ms.custom: hdinsightactive
-ms.date: 12/12/2019
+ms.date: 02/10/2020
 ---
 
 # Run Apache Spark from the Spark Shell
@@ -22,31 +22,76 @@ An interactive [Apache Spark](https://spark.apache.org/) Shell provides a REPL (
     ssh [email protected]
     ```
 
-1. Spark provides shells for Scala (spark-shell), and Python (pyspark). In your SSH session, enter one of the following commands:
+1. Spark provides shells for Scala (spark-shell), and Python (pyspark). In your SSH session, enter *one* of the following commands:
 
     ```bash
     spark-shell
+
+    # Optional configurations
+    # spark-shell --num-executors 4 --executor-memory 4g --executor-cores 2 --driver-memory 8g --driver-cores 4
+    ```
+
+    ```bash
     pyspark
+
+    # Optional configurations
+    # pyspark --num-executors 4 --executor-memory 4g --executor-cores 2 --driver-memory 8g --driver-cores 4
     ```
 
-    Now you can enter Spark commands in the appropriate language.
+    If you intend to use any optional configuration, ensure you first review [OutOfMemoryError exception for Apache Spark](./apache-spark-troubleshoot-outofmemory.md).
+
+1. A few basic example commands. Choose the relevant language:
 
-1. A few basic example commands:
+    ```spark-shell
+    val textFile = spark.read.textFile("/example/data/fruits.txt")
+    textFile.first()
+    textFile.filter(line => line.contains("apple")).show()
+    ```
+
+    ```pyspark
+    textFile = spark.read.text("/example/data/fruits.txt")
+    textFile.first()
+    textFile.filter(textFile.value.contains("apple")).show()
+    ```
+
+1. Query a CSV file. Note the language below works for `spark-shell` and `pyspark`.
 
     ```scala
-    // Load data
+    spark.read.csv("/HdiSamples/HdiSamples/SensorSampleData/building/building.csv").show()
+    ```
+
+1. Query a CSV file and store results in variable:
+
+    ```spark-shell
     var data = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/HdiSamples/HdiSamples/SensorSampleData/building/building.csv")
+    ```
 
-    // Show data
-    data.show()
+    ```pyspark
+    data = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/HdiSamples/HdiSamples/SensorSampleData/building/building.csv")
+    ```
 
-    // Select certain columns
+1. Display results:
+
+    ```spark-shell
+    data.show()
     data.select($"BuildingID", $"Country").show(10)
+    ```
 
-    // exit shell
+    ```pyspark
+    data.show()
+    data.select("BuildingID", "Country").show(10)
+    ```
+
+1. Exit
+
+    ```spark-shell
     :q
     ```
 
+    ```pyspark
+    exit()
+    ```
+
 ## SparkSession and SparkContext instances
 
 By default when you run the Spark Shell, instances of SparkSession and SparkContext are automatically instantiated for you.

articles/hdinsight/spark/troubleshoot-debug-wasb.md

@@ -0,0 +1,65 @@
+---
+title: Debug WASB file operations in Azure HDInsight
+description: Describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters.
+author: hrasheed-msft
+ms.author: hrasheed
+ms.reviewer: jasonh
+ms.service: hdinsight
+ms.topic: troubleshooting
+ms.date: 02/07/2020
+---
+
+# Debug WASB file operations in Azure HDInsight
+
+There are times when you may want to understand what operations the WASB driver started with Azure Storage. For the client side, the WASB driver produces logs for each file system operation at **DEBUG** level. WASB driver uses log4j to control logging level and the default is **INFO** level. For Azure Storage server-side analytics logs, see [Azure Storage analytics logging](../../storage/common/storage-analytics-logging.md).
+
+A produced log will look similar to:
+
+```log
+18/05/13 04:15:55 DEBUG NativeAzureFileSystem: Moving wasb://[email protected]/user/livy/ulysses.txt/_temporary/0/_temporary/attempt_20180513041552_0000_m_000000_0/part-00000 to wasb://[email protected]/user/livy/ulysses.txt/part-00000
+```
+
+## Turn on WASB debug log for file operations
+
+1. From a web browser, navigate to `https://CLUSTERNAME.azurehdinsight.net`, where `CLUSTERNAME` is the name of your Spark cluster.
+
+1. Navigate to **Spark2** > **Configs** > **advanced spark2-log4j-properties**.
+
+1. Modify `log4j.appender.console.Threshold=INFO` to `log4j.appender.console.Threshold=DEBUG`.
+
+1. Navigate to **Advanced livy2-log4j-properties**.
+
+1. Add the following property:
+
+    ```
+    log4j.logger.org.apache.hadoop.fs.azure.NativeAzureFileSystem=DEBUG
+    ```
+
+1. Save changes.
+
+## Additional logging
+
+The above logs should provide high-level understanding of the file system operations. If the above logs are still not providing useful information, or if you want to investigate blob storage api calls, add `fs.azure.storage.client.logging=true` to the `core-site`. This setting will enable the java sdk logs for wasb storage driver and will print each call to blob storage server. Remove the setting after investigations because it could fill up the disk quickly and could slow down the process.
+
+If the backend is Azure Data Lake based, then use the following log4j setting for the component(for example, spark/tez/hdfs):
+
+```
+log4j.logger.com.microsoft.azure.datalake.store=ALL,adlsFile
+log4j.additivity.com.microsoft.azure.datalake.store=true
+log4j.appender.adlsFile=org.apache.log4j.FileAppender
+log4j.appender.adlsFile.File=/var/log/adl/adl.log
+log4j.appender.adlsFile.layout=org.apache.log4j.PatternLayout
+log4j.appender.adlsFile.layout.ConversionPattern=%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n
+```
+
+Look for the logs in `/var/log/adl/adl.log` for the logs.
+
+## Next steps
+
+If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support:
+
+* Get answers from Azure experts through [Azure Community Support](https://azure.microsoft.com/support/community/).
+
+* Connect with [@AzureSupport](https://twitter.com/azuresupport) - the official Microsoft Azure account for improving customer experience. Connecting the Azure community to the right resources: answers, support, and experts.
+
+* If you need more help, you can submit a support request from the [Azure portal](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade/). Select **Support** from the menu bar or open the **Help + support** hub. For more detailed information, review [How to create an Azure support request](https://docs.microsoft.com/azure/azure-supportability/how-to-create-azure-support-request). Access to Subscription Management and billing support is included with your Microsoft Azure subscription, and Technical Support is provided through one of the [Azure Support Plans](https://azure.microsoft.com/support/plans/).

articles/hdinsight/spark/troubleshoot-jupyter-notebook-convert.md

@@ -0,0 +1,62 @@
+---
+title: Unable to create Jupyter notebook in Azure HDInsight
+description: Describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters.
+author: hrasheed-msft
+ms.author: hrasheed
+ms.reviewer: jasonh
+ms.service: hdinsight
+ms.topic: troubleshooting
+ms.date: 02/11/2020
+---
+
+# Unable to create Jupyter notebook in Azure HDInsight
+
+This article describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters.
+
+## Issue
+
+When starting a Jupyter notebook, you receive an error message that contains:
+
+```error
+Cannot convert notebook to v5 because that version doesn't exist
+```
+
+## Cause
+
+A version mismatch.
+
+## Resolution
+
+1. Use [ssh command](../hdinsight-hadoop-linux-use-ssh-unix.md) to connect to your cluster. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
+
+    ```cmd
+    ssh [email protected]
+    ```
+
+1. Open `_version.py` by executing the following command:
+
+    ```bash
+    sudo nano /usr/bin/anaconda/lib/python2.7/site-packages/nbformat/_version.py
+    ```
+
+1. Change **5** to **4** so the modified line appears as follows:
+
+    ```python
+    version_info = (4, 0, 3)
+    ```
+
+    Save changes by entering **Ctrl + X**, **Y**, **Enter**.
+
+1. From a web browser, navigate to `https://CLUSTERNAME.azurehdinsight.net`, where `CLUSTERNAME` is the name of your cluster.
+
+1. Select **Jupyter** and then restart the service.
+
+## Next steps
+
+If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support:
+
+* Get answers from Azure experts through [Azure Community Support](https://azure.microsoft.com/support/community/).
+
+* Connect with [@AzureSupport](https://twitter.com/azuresupport) - the official Microsoft Azure account for improving customer experience. Connecting the Azure community to the right resources: answers, support, and experts.
+
+* If you need more help, you can submit a support request from the [Azure portal](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade/). Select **Support** from the menu bar or open the **Help + support** hub. For more detailed information, review [How to create an Azure support request](https://docs.microsoft.com/azure/azure-supportability/how-to-create-azure-support-request). Access to Subscription Management and billing support is included with your Microsoft Azure subscription, and Technical Support is provided through one of the [Azure Support Plans](https://azure.microsoft.com/support/plans/).

articles/virtual-machine-scale-sets/use-spot.md

@@ -6,7 +6,7 @@ tags: azure-resource-manager
 ms.service: virtual-machine-scale-sets
 ms.workload: infrastructure-services
 ms.topic: conceptual
-ms.date: 10/23/2019
+ms.date: 02/11/2020
 ms.author: cynthn
 ---
 
@@ -21,7 +21,6 @@ The amount of available capacity can vary based on size, region, time of day, an
 > This preview version is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. 
 > For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
 >
-> For the early part of the public preview, Spot instances will have a fixed price, so there will not be any price-based evictions.
 
 ## Pricing
 

articles/virtual-machines/linux/spot-cli.md

@@ -13,7 +13,7 @@ ms.workload: infrastructure-services
 ms.tgt_pltfrm: na
 ms.devlang: na
 ms.topic: article
-ms.date: 11/20/2019
+ms.date: 02/11/2020
 ms.author: cynthn
 ---
 
@@ -32,7 +32,7 @@ The process to create a VM with Spot using the Azure CLI is the same as detailed
 > This preview version is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. 
 > For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
 >
-> For the early part of the public preview, Spot instances will have a fixed price, so there will not be any price-based evictions.
+
 
 
 ## Install Azure CLI

articles/virtual-machines/linux/spot-template.md

@@ -13,7 +13,7 @@ ms.workload: infrastructure-services
 ms.tgt_pltfrm: na
 ms.devlang: na
 ms.topic: article
-ms.date: 10/14/2019
+ms.date: 02/11/2020
 ms.author: cynthn
 ---
 
@@ -30,7 +30,7 @@ You have option to set a max price you are willing to pay, per hour, for the VM.
 > This preview version is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. 
 > For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
 >
-> For the early part of the public preview, Spot instances will have a fixed price, so there will not be any price-based evictions.
+
 
 
 ## Use a template 
@@ -46,9 +46,6 @@ For Spot template deployments, use`"apiVersion": "2019-03-01"` or later. Add the
 ```
 
 
-> [!IMPORTANT]
-> For the early part of the public preview, you can set a max price, but it will be ignored. Spot VMs will have a fixed price, so there will not be any price-based evictions.
-
 
 Here is a sample template with the added properties for a Spot VM. Replace the resource names with your own and `<password>` with a password for the local administrator account on the VM.
 

articles/virtual-machines/windows/spot-portal.md

@@ -8,7 +8,7 @@ manager: gwallace
 ms.service: virtual-machines-windows
 ms.workload: infrastructure-services
 ms.topic: article
-ms.date: 11/20/2019
+ms.date: 02/11/2020
 ms.author: cynthn
 ---
 
@@ -25,7 +25,7 @@ You have option to set a max price you are willing to pay, per hour, for the VM.
 > This preview version is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. 
 > For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
 >
-> For the early part of the public preview, Spot instances will have a fixed price, so there will not be any price-based evictions.
+
 
 ## Create the VM
 

articles/virtual-machines/windows/spot-powershell.md

@@ -8,7 +8,7 @@ manager: gwallace
 ms.service: virtual-machines-windows
 ms.workload: infrastructure-services
 ms.topic: article
-ms.date: 10/14/2019
+ms.date: 02/11/2020
 ms.author: cynthn
 ---
 
@@ -26,7 +26,7 @@ You have option to set a max price you are willing to pay, per hour, for the VM.
 > This preview version is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. 
 > For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
 >
-> For the early part of the public preview, Spot instances will have a fixed price, so there will not be any price-based evictions.
+
 
 
 ## Create the VM
@@ -35,9 +35,6 @@ Create a spotVM using [New-AzVmConfig](/powershell/module/az.compute/new-azvmcon
 - `-1` so the VM is not evicted based on price.
 - a dollar amount, up to 5 digits. For example, `-MaxPrice .98765` means that the VM will be deallocated once the price for a spotVM goes about $.98765 per hour.
 
-> [!IMPORTANT]
-> For the early part of the public preview, you can set a max price, but it will be ignored. Spot VMs will have a fixed price, so there will not be any price-based evictions.
-
 
 This example creates a spotVM that will not be deallocated based on pricing (only when Azure needs the capacity back).