Merge pull request #94713 from dagiro/storage3

storage3

Author: TedA-M

articles/hdinsight/hdinsight-hadoop-use-data-lake-storage-gen2.md

@@ -7,7 +7,7 @@ ms.reviewer: jasonh
 ms.service: hdinsight
 ms.custom: hdinsightactive
 ms.topic: conceptual
-ms.date: 08/27/2019
+ms.date: 11/04/2019
 ---
 
 # Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters
@@ -29,13 +29,13 @@ To create an HDInsight cluster that uses Data Lake Storage Gen2 for storage, fol
 
 ### Create a user-assigned managed identity
 
-Create a user-assigned managed identity, if you don’t already have one. 
+Create a user-assigned managed identity, if you don’t already have one.
 
 1. Sign in to the [Azure portal](https://portal.azure.com).
 1. In the upper-left click **Create a resource**.
 1. In the search box, type **user assigned** and click **User Assigned Managed Identity**.
 1. Click **Create**.
-1. Enter a name for your managed identity, select the correct subscription, resource group and location.
+1. Enter a name for your managed identity, select the correct subscription, resource group, and location.
 1. Click **Create**.
 
 For more information on how managed identities work in Azure HDInsight, see [Managed identities in Azure HDInsight](hdinsight-managed-identities.md).
@@ -44,7 +44,7 @@ For more information on how managed identities work in Azure HDInsight, see [Man
 
 ### Create a Data Lake Storage Gen2 account
 
-Create an Azure Data Lake Storage Gen2 storage account. 
+Create an Azure Data Lake Storage Gen2 storage account.
 
 1. Sign in to the [Azure portal](https://portal.azure.com).
 1. In the upper-left click **Create a resource**.
@@ -68,28 +68,28 @@ Assign the managed identity to the **Storage Blob Data Owner** role on the stora
 
 1. In the [Azure portal](https://portal.azure.com), go to your storage account.
 1. Select your storage account, then select **Access control (IAM)** to display the access control settings for the account. Select the **Role assignments** tab to see the list of role assignments.
-    
+
     ![Screenshot showing storage access control settings](./media/hdinsight-hadoop-use-data-lake-storage-gen2/portal-access-control.png)
-    
+
 1. Select the **+ Add role assignment** button to add a new role.
 1. In the **Add role assignment** window, select the **Storage Blob Data Owner** role. Then, select the subscription that has the managed identity and storage account. Next, search to locate the user-assigned managed identity that you created previously. Finally, select the managed identity, and it will be listed under **Selected members**.
-    
+
     ![Screenshot showing how to assign an RBAC role](./media/hdinsight-hadoop-use-data-lake-storage-gen2/add-rbac-role3-window.png)
-    
+
 1. Select **Save**. The user-assigned identity that you selected is now listed under the selected role.
 1. After this initial setup is complete, you can create a cluster through the portal. The cluster must be in the same Azure region as the storage account. In the **Storage** section of the cluster creation menu, select the following options:
-        
+
     * For **Primary storage type**, select **Azure Data Lake Storage Gen2**.
     * Under **Select a Storage account**, search for and select the newly created Data Lake Storage Gen2 storage account.
-        
+
         ![Storage settings for using Data Lake Storage Gen2 with Azure HDInsight](./media/hdinsight-hadoop-use-data-lake-storage-gen2/primary-storage-type-adls-gen2.png)
-    
+
     * Under **Identity**, select the correct subscription and the newly created user-assigned managed identity.
 
         ![Identity settings for using Data Lake Storage Gen2 with HDInsight](./media/hdinsight-hadoop-use-data-lake-storage-gen2/managed-identity-cluster-creation.png)
 
 > [!Note]
-> To add a secondary Data Lake Storage Gen2 account, at the storage account level, simply assign the managed identity created earlier to the new Data Lake Storage Gen2 storage account that you wish to add.Please be advised that adding a secondary Data Lake Storage Gen2 account via the "Additional storage accounts" blade on HDInsight is not supported. 
+> To add a secondary Data Lake Storage Gen2 account, at the storage account level, simply assign the managed identity created earlier to the new Data Lake Storage Gen2 storage account that you wish to add.Please be advised that adding a secondary Data Lake Storage Gen2 account via the "Additional storage accounts" blade on HDInsight is not supported.
 
 ## Create a cluster with Data Lake Storage Gen2 through the Azure CLI
 
@@ -108,10 +108,10 @@ The code snippet below does the following initial steps:
 
 1. Logs in to your Azure account.
 1. Sets the active subscription where the create operations will be done.
-1. Creates a new resource group for the new deployment activities. 
+1. Creates a new resource group for the new deployment activities.
 1. Creates a user-assigned managed identity.
 1. Adds an extension to the Azure CLI to use features for Data Lake Storage Gen2.
-1. Creates a new Data Lake Storage Gen2 account by using the `--hierarchical-namespace true` flag. 
+1. Creates a new Data Lake Storage Gen2 account by using the `--hierarchical-namespace true` flag.
 
 ```azurecli
 az login
@@ -166,7 +166,87 @@ The lifecycle of a user-assigned identity is managed separately from the lifecyc
 
 To set permissions for users to query data, use Azure AD security groups as the assigned principal in ACLs. Don't directly assign file-access permissions to individual users or service principals. When you use Azure AD security groups to control the flow of permissions, you can add and remove users or service principals without reapplying ACLs to an entire directory structure. You only have to add or remove the users from the appropriate Azure AD security group. ACLs aren't inherited, so reapplying ACLs requires updating the ACL on every file and subdirectory.
 
+## Access files from the cluster
+
+There are several ways you can access the files in Data Lake Storage Gen2 from an HDInsight cluster.
+
+* **Using the fully qualified name**. With this approach, you provide the full path to the file that you want to access.
+
+    ```
+    abfs://<containername>@<accountname>.dfs.core.windows.net/<file.path>/
+    ```
+
+* **Using the shortened path format**. With this approach, you replace the path up to the cluster root with:
+
+    ```
+    abfs:///<file.path>/
+    ```
+
+* **Using the relative path**. With this approach, you only provide the relative path to the file that you want to access.
+
+    ```
+    /<file.path>/
+    ```
+
+### Data access examples
+
+Examples are based on an [ssh connection](./hdinsight-hadoop-linux-use-ssh-unix.md) to the head node of the cluster. The examples use all three URI schemes. Replace `CONTAINERNAME` and `STORAGEACCOUNT` with the relevant values
+
+#### A few hdfs commands
+
+1. Create a simple file on local storage.
+
+    ```bash
+    touch testFile.txt
+    ```
+
+1. Create directories on cluster storage.
+
+    ```bash
+    hdfs dfs -mkdir abfs://[email protected]/sampledata1/
+    hdfs dfs -mkdir abfs:///sampledata2/
+    hdfs dfs -mkdir /sampledata3/
+    ```
+
+1. Copy data from local storage to cluster storage.
+
+    ```bash
+    hdfs dfs -copyFromLocal testFile.txt  abfs://[email protected]/sampledata1/
+    hdfs dfs -copyFromLocal testFile.txt  abfs:///sampledata2/
+    hdfs dfs -copyFromLocal testFile.txt  /sampledata3/
+    ```
+
+1. List directory contents on cluster storage.
+
+    ```bash
+    hdfs dfs -ls abfs://[email protected]/sampledata1/
+    hdfs dfs -ls abfs:///sampledata2/
+    hdfs dfs -ls /sampledata3/
+    ```
+
+#### Creating a Hive table
+
+Three file locations are shown for illustrative purposes. For actual execution, use only one of the `LOCATION` entries.
+
+```hql
+DROP TABLE myTable;
+CREATE EXTERNAL TABLE myTable (
+    t1 string,
+    t2 string,
+    t3 string,
+    t4 string,
+    t5 string,
+    t6 string,
+    t7 string)
+ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '
+STORED AS TEXTFILE
+LOCATION 'abfs://[email protected]/example/data/';
+LOCATION 'abfs:///example/data/';
+LOCATION '/example/data/';
+```
+
 ## Next steps
 
 * [Azure HDInsight integration with Data Lake Storage Gen2 preview - ACL and security update](https://azure.microsoft.com/blog/azure-hdinsight-integration-with-data-lake-storage-gen-2-preview-acl-and-security-update/)
 * [Introduction to Azure Data Lake Storage Gen2](../storage/blobs/data-lake-storage-introduction.md)
+* [Tutorial: Extract, transform, and load data using Interactive Query in Azure HDInsight](./interactive-query/interactive-query-tutorial-analyze-flight-data.md)