freshness17

Author: dagiro

articles/hdinsight/hdinsight-hadoop-create-linux-clusters-adf.md

@@ -6,16 +6,17 @@ ms.author: hrasheed
 ms.reviewer: jasonh
 ms.service: hdinsight
 ms.topic: tutorial
-ms.date: 04/18/2019
+ms.date: 10/09/2019
 #Customer intent: As a data worker, I need to create a Hadoop cluster and run Hive jobs on demand
 ---
 
 # Tutorial: Create on-demand Apache Hadoop clusters in HDInsight using Azure Data Factory
+
 [!INCLUDE [selector](../../includes/hdinsight-create-linux-cluster-selector.md)]
 
 In this tutorial, you learn how to create a [Apache Hadoop](https://hadoop.apache.org/) cluster, on demand, in Azure HDInsight using Azure Data Factory. You then use data pipelines in Azure Data Factory to run Hive jobs and delete the cluster. By the end of this tutorial, you learn how to operationalize a big data job run where cluster creation, job run, and cluster deletion are performed on a schedule.
 
-This tutorial covers the following tasks: 
+This tutorial covers the following tasks:
 
 > [!div class="checklist"]
 > * Create an Azure storage account
@@ -33,11 +34,11 @@ If you don't have an Azure subscription, [create a free account](https://azure.m
 
 * The PowerShell [Az Module](https://docs.microsoft.com/powershell/azure/overview) installed.
 
-* An Azure Active Directory service principal. Once you have created the service principal, be sure to retrieve the **application ID** and **authentication key** using the instructions in the linked article. You need these values later in this tutorial. Also, make sure the service principal is a member of the *Contributor* role of the subscription or the resource group in which the cluster is created. For instructions to retrieve the required values and assign the right roles, see [Create an Azure Active Directory service principal](../active-directory/develop/howto-create-service-principal-portal.md).
+* An Azure Active Directory service principal. Once you've created the service principal, be sure to retrieve the **application ID** and **authentication key** using the instructions in the linked article. You need these values later in this tutorial. Also, make sure the service principal is a member of the *Contributor* role of the subscription or the resource group in which the cluster is created. For instructions to retrieve the required values and assign the right roles, see [Create an Azure Active Directory service principal](../active-directory/develop/howto-create-service-principal-portal.md).
 
 ## Create preliminary Azure objects
 
-In this section, you create various objects that will be used for the HDInsight cluster you create on-demand. The created storage account will contain the sample [HiveQL](https://cwiki.apache.org/confluence/display/Hive/LanguageManual) script (`partitionweblogs.hql`) that you use to simulate a sample [Apache Hive](https://hive.apache.org/) job that runs on the cluster.
+In this section, you create various objects that will be used for the HDInsight cluster you create on-demand. The created storage account will contain the sample [HiveQL](https://cwiki.apache.org/confluence/display/Hive/LanguageManual) script, `partitionweblogs.hql`, that you use to simulate a sample [Apache Hive](https://hive.apache.org/) job that runs on the cluster.
 
 This section uses an Azure PowerShell script to create the storage account and copy over the required files within the storage account. The Azure PowerShell sample script in this section performs the following tasks:
 
@@ -47,9 +48,6 @@ This section uses an Azure PowerShell script to create the storage account and c
 4. Creates a Blob container in the storage account
 5. Copies the sample HiveQL script (**partitionweblogs.hql**) the Blob container. The script is available at [https://hditutorialdata.blob.core.windows.net/adfhiveactivity/script/partitionweblogs.hql](https://hditutorialdata.blob.core.windows.net/adfhiveactivity/script/partitionweblogs.hql). The sample script is already available in another public Blob container. The PowerShell script below makes a copy of these files into the Azure Storage account it creates.
 
-> [!WARNING]  
-> Storage account kind `BlobStorage` cannot be used for HDInsight clusters.
-
 **To create a storage account and copy the files using Azure PowerShell:**
 
 > [!IMPORTANT]  
@@ -77,6 +75,10 @@ if(-not($sub))
 {
     Connect-AzAccount
 }
+
+# If you have multiple subscriptions, set the one to use
+# Select-AzSubscription -SubscriptionId "<SUBSCRIPTIONID>"
+
 #endregion
 
 ####################################
@@ -123,11 +125,13 @@ Write-Host "`nCopying files ..." -ForegroundColor Green
 
 $blobs = Get-AzStorageBlob `
     -Context $sourceContext `
-    -Container $sourceContainerName
+    -Container $sourceContainerName `
+    -Blob "hivescripts\hivescript.hql"
 
 $blobs|Start-AzStorageBlobCopy `
     -DestContext $destContext `
-    -DestContainer $destContainerName
+    -DestContainer $destContainerName `
+    -DestBlob "hivescripts\partitionweblogs.hql"
 
 Write-Host "`nCopied files ..." -ForegroundColor Green
 Get-AzStorageBlob `
@@ -146,16 +150,16 @@ Write-host "`nScript completed" -ForegroundColor Green
 **To verify the storage account creation**
 
 1. Sign on to the [Azure portal](https://portal.azure.com).
-2. Select **Resource groups** on the left pane.
-3. Select the resource group name you created in your PowerShell script. Use the filter if you have too many resource groups listed.
-4. On the **Resources** tile, you see one resource listed unless you share the resource group with other projects. That resource is the storage account with the name you specified earlier. Select the storage account name.
-5. Select the **Blobs** tiles.
-6. Select the **adfgetstarted** container. You see a folder called **hivescripts**.
-7. Open the folder and make sure it contains the sample script file, **partitionweblogs.hql**.
+1. From the left, navigate to **All services** > **General** > **Resource groups**.
+1. Select the resource group name you created in your PowerShell script. Use the filter if you have too many resource groups listed.
+1. From the **Overview** view, you see one resource listed unless you share the resource group with other projects. That resource is the storage account with the name you specified earlier. Select the storage account name.
+1. Select the **Containers** tile.
+1. Select the **adfgetstarted** container. You see a folder called **hivescripts**.
+1. Open the folder and make sure it contains the sample script file, **partitionweblogs.hql**.
 
 ## Understand the Azure Data Factory activity
 
-[Azure Data Factory](../data-factory/introduction.md) orchestrates and automates the movement and transformation of data. Azure Data Factory can create an HDInsight Hadoop cluster just-in-time to process an input data slice and delete the cluster when the processing is complete. 
+[Azure Data Factory](../data-factory/introduction.md) orchestrates and automates the movement and transformation of data. Azure Data Factory can create an HDInsight Hadoop cluster just-in-time to process an input data slice and delete the cluster when the processing is complete.
 
 In Azure Data Factory, a data factory can have one or more data pipelines. A data pipeline has one or more activities. There are two types of activities:
 
@@ -190,12 +194,13 @@ In this article, you configure the Hive activity to create an on-demand HDInsigh
     |Resource group | Select **Use existing** and then select the resource group you created using the PowerShell script. |
     |Version | Leave at **V2**. |
     |Location | The location is automatically set to the location you specified while creating the resource group earlier. For this tutorial, the location is set to **East US**. |
+    |Enable GIT|Uncheck this box.|
 
     ![Create Azure Data Factory using Azure portal](./media/hdinsight-hadoop-create-linux-clusters-adf/create-data-factory-portal.png "Create Azure Data Factory using Azure portal")
 
 4. Select **Create**. Creating a data factory might take anywhere between 2 to 4 minutes.
 
-5. Once the data factory is created, you will receive a **Deployment succeeded** notification with a **Go to resource** button.  Select **Go to resource** to open the Data Factory default view.
+5. Once the data factory is created, you'll receive a **Deployment succeeded** notification with a **Go to resource** button.  Select **Go to resource** to open the Data Factory default view.
 
 6. Select **Author & Monitor** to launch the Azure Data Factory authoring and monitoring portal.
 
@@ -230,7 +235,7 @@ In this section, you author two linked services within your data factory.
     |Azure subscription |Select your subscription from the drop-down list.|
     |Storage account name |Select the Azure Storage account you created as part of the PowerShell script.|
 
-    Then select **Finish**.
+    Select **Test connection** and if successful, then select **Create**.
 
     ![Provide name for Azure Storage linked service](./media/hdinsight-hadoop-create-linux-clusters-adf/hdinsight-data-factory-storage-linked-service-details.png "Provide name for Azure Storage linked service")
 
@@ -258,13 +263,12 @@ In this section, you author two linked services within your data factory.
     | Cluster name prefix | Provide a value that will be prefixed to all the cluster types that are created by the data factory. |
     |Subscription |Select your subscription from the drop-down list.|
     | Select resource group | Select the resource group you created as part of the PowerShell script you used earlier.|
-    |Select region | Select a region from the drop-down list.|
     | OS type/Cluster SSH user name | Enter an SSH user name, commonly `sshuser`. |
     | OS type/Cluster SSH password | Provide a password for the SSH user |
     | OS type/Cluster user name | Enter a cluster user name, commonly `admin`. |
-    | OS type/Cluster user password | Provide a password for the cluster user. |
+    | OS type/Cluster password | Provide a password for the cluster user. |
 
-    Then select **Finish**.
+    Then select **Create**.
 
     ![Provide values for HDInsight linked service](./media/hdinsight-hadoop-create-linux-clusters-adf/hdinsight-data-factory-linked-service-details.png "Provide values for HDInsight linked service")
 
@@ -278,7 +282,7 @@ In this section, you author two linked services within your data factory.
 
     ![Add activities to Data Factory pipeline](./media/hdinsight-hadoop-create-linux-clusters-adf/hdinsight-data-factory-add-hive-pipeline.png "Add activities to Data Factory pipeline")
 
-3. Make sure you have the Hive activity selected, select the **HDI Cluster** tab, and from the **HDInsight Linked Service** drop-down list, select the linked service you created earlier, **HDinightLinkedService**, for HDInsight.
+3. Make sure you have the Hive activity selected, select the **HDI Cluster** tab, and from the **HDInsight Linked Service** drop-down list, select the linked service you created earlier, **HDInsightLinkedService**, for HDInsight.
 
     ![Provide HDInsight cluster details for the pipeline](./media/hdinsight-hadoop-create-linux-clusters-adf/hdinsight-hive-activity-select-hdinsight-linked-service.png "Provide HDInsight cluster details for the pipeline")
 
@@ -318,7 +322,7 @@ In this section, you author two linked services within your data factory.
 
 1. Select **Refresh** to refresh the status.
 
-1. You can also select the **View Activity Runs** icon to see the activity run associated with the pipeline. In the screenshot below, you see only one activity run since there is only one activity in the pipeline you created. To switch back to the previous view, select **Pipelines** towards the top of the page.
+1. You can also select the **View Activity Runs** icon to see the activity run associated with the pipeline. In the screenshot below, you see only one activity run since there's only one activity in the pipeline you created. To switch back to the previous view, select **Pipelines** towards the top of the page.
 
     ![Monitor the Azure Data Factory pipeline activity](./media/hdinsight-hadoop-create-linux-clusters-adf/hdinsight-data-factory-monitor-pipeline-activity.png "Monitor the Azure Data Factory pipeline activity")
 
@@ -336,7 +340,7 @@ In this section, you author two linked services within your data factory.
 
 ## Clean up resources
 
-With the on-demand HDInsight cluster creation, you do not need to explicitly delete the HDInsight cluster. The cluster is deleted based on the configuration you provided while creating the pipeline. However, even after the cluster is deleted, the storage accounts associated with the cluster continue to exist. This behavior is by design so that you can keep your data intact. However, if you do not want to persist the data, you may delete the storage account you created.
+With the on-demand HDInsight cluster creation, you don't need to explicitly delete the HDInsight cluster. The cluster is deleted based on the configuration you provided while creating the pipeline. However, even after the cluster is deleted, the storage accounts associated with the cluster continue to exist. This behavior is by design so that you can keep your data intact. However, if you don't want to persist the data, you may delete the storage account you created.
 
 Alternatively, you can delete the entire resource group that you created for this tutorial. This deletes the storage account and the Azure Data Factory that you created.