Merge pull request #109358 from dagiro/ETL2

ETL2

Author: v-albemi

articles/hdinsight/hdinsight-sales-insights-etl.md

@@ -7,7 +7,7 @@ ms.reviewer: jasonh
 ms.service: hdinsight
 ms.topic: tutorial
 ms.custom: hdinsightactive
-ms.date: 03/24/2020
+ms.date: 04/15/2020
 ---
 
 # Tutorial: Create an end-to-end data pipeline to derive sales insights in Azure HDInsight
@@ -22,23 +22,28 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
 
 ## Prerequisites
 
-* Azure CLI. See [Install the Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli).
+* Azure CLI - at least version 2.2.0. See [Install the Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli).
+
+* jq, a command-line JSON processor.  See [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/).
 
 * A member of the [Azure built-in role - owner](../role-based-access-control/built-in-roles.md).
 
-* [Power BI Desktop](https://www.microsoft.com/download/details.aspx?id=45331) to visualize business insights at the end of this tutorial.
+* If using PowerShell to trigger the Data Factory pipeline, you'll need the [Az Module](https://docs.microsoft.com/powershell/azure/overview).
+
+* [Power BI Desktop](https://aka.ms/pbiSingleInstaller) to visualize business insights at the end of this tutorial.
 
 ## Create resources
 
 ### Clone the repository with scripts and data
 
-1. Sign in to the [Azure portal](https://portal.azure.com).
+1. Log in to your Azure subscription. If you plan to use Azure Cloud Shell, then select **Try it** in the upper-right corner of the code block. Else, enter the command below:
 
-1. Open Azure Cloud Shell from the top menu bar. Select your subscription for creating a file share if Cloud Shell prompts you.
+    ```azurecli-interactive
+    az login
 
-   ![Open Azure Cloud Shell](./media/hdinsight-sales-insights-etl/hdinsight-sales-insights-etl-click-cloud-shell.png)
-
-1. In the **Select environment** drop-down menu, choose **Bash**.
+    # If you have multiple subscriptions, set the one to use
+    # az account set --subscription "SUBSCRIPTIONID"
+    ```
 
 1. Ensure you're a member of the Azure role [owner](../role-based-access-control/built-in-roles.md). Replace `[email protected]` with your account and then enter the command:
 
@@ -50,29 +55,7 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
 
     If no record is returned, you aren't a member and won't be able to complete this tutorial.
 
-1. List your subscriptions entering the command:
-
-    ```azurecli
-    az account list --output table
-    ```
-
-    Note the ID of the subscription that you'll use for this project.
-
-1. Set the subscription you'll use for this project. Replace `SUBSCRIPTIONID` with the actual value, then enter the command.
-
-    ```azurecli
-    subscriptionID="SUBSCRIPTIONID"
-    az account set --subscription $subscriptionID
-    ```
-
-1. Create a new resource group for the project. Replace `RESOURCEGROUP` with the desired name, then enter the command.
-
-    ```azurecli
-    resourceGroup="RESOURCEGROUP"
-    az group create --name $resourceGroup --location westus
-    ```
-
-1. Download the data and scripts for this tutorial from the [HDInsight sales insights ETL repository](https://github.com/Azure-Samples/hdinsight-sales-insights-etl).  Enter the following command:
+1. Download the data and scripts for this tutorial from the [HDInsight sales insights ETL repository](https://github.com/Azure-Samples/hdinsight-sales-insights-etl). Enter the following command:
 
     ```bash
     git clone https://github.com/Azure-Samples/hdinsight-sales-insights-etl.git
@@ -93,12 +76,20 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
     chmod +x scripts/*.sh
     ````
 
-1. Execute the script. Replace `RESOURCE_GROUP_NAME` and `LOCATION` with the relevant values, then enter the command:
+1. Set variable for resource group. Replace `RESOURCE_GROUP_NAME` with the name of an existing or new resource group, then enter the command:
+
+    ```bash
+    resourceGroup="RESOURCE_GROUP_NAME"
+    ```
+
+1. Execute the script. Replace `LOCATION` with a desired value, then enter the command:
 
     ```bash
-    ./scripts/resources.sh RESOURCE_GROUP_NAME LOCATION
+    ./scripts/resources.sh $resourceGroup LOCATION
     ```
 
+    If you're not sure which region to specify, you can retrieve a list of supported regions for your subscription with the [az account list-locations](https://docs.microsoft.com/cli/azure/account?view=azure-cli-latest#az-account-list-locations) command.
+
     The command will deploy the following resources:
 
     * An Azure Blob storage account. This account will hold the company sales data.
@@ -110,49 +101,26 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
 
 Cluster creation can take around 20 minutes.
 
-The `resources.sh` script contains the following commands. It isn't required for you to run these commands if you already executed the script in the previous step.
-
-* `az group deployment create` - This command uses an Azure Resource Manager template (`resourcestemplate.json`) to create the specified resources with the desired configuration.
-
-    ```azurecli
-    az group deployment create --name ResourcesDeployment \
-        --resource-group $resourceGroup \
-        --template-file resourcestemplate.json \
-        --parameters "@resourceparameters.json"
-    ```
-
-* `az storage blob upload-batch` - This command uploads the sales data .csv files into the newly created Blob storage account by using this command:
-
-    ```azurecli
-    az storage blob upload-batch -d rawdata \
-        --account-name <BLOB STORAGE NAME> -s ./ --pattern *.csv
-    ```
-
-The default password for SSH access to the clusters is `Thisisapassword1`. If you want to change the password, go to the `resourcesparameters.json` file and change the password for the `sparksshPassword`, `sparkClusterLoginPassword`, `llapClusterLoginPassword`, and `llapsshPassword` parameters.
+The default password for SSH access to the clusters is `Thisisapassword1`. If you want to change the password, go to the `./templates/resourcesparameters_remainder.json` file and change the password for the `sparksshPassword`, `sparkClusterLoginPassword`, `llapClusterLoginPassword`, and `llapsshPassword` parameters.
 
 ### Verify deployment and collect resource information
 
-1. If you want to check the status of your deployment, go to the resource group on the Azure portal. Select **Deployments** under **Settings**. Select the name of your deployment, `ResourcesDeployment`. Here you can see the resources that have successfully deployed and the resources that are still in progress.
+1. If you want to check the status of your deployment, go to the resource group on the Azure portal. Under **Settings**, select **Deployments**, then your deployment. Here you can see the resources that have successfully deployed and the resources that are still in progress.
 
 1. To view the names of the clusters, enter the following command:
 
-    ```azurecli
-    sparkCluster=$(az hdinsight list \
-        --resource-group $resourceGroup \
-        --query "[?contains(name,'spark')].{clusterName:name}" -o tsv)
-
-    llapCluster=$(az hdinsight list \
-        --resource-group $resourceGroup \
-        --query "[?contains(name,'llap')].{clusterName:name}" -o tsv)
+    ```bash
+    sparkClusterName=$(cat resourcesoutputs_remainder.json | jq -r '.properties.outputs.sparkClusterName.value')
+    llapClusterName=$(cat resourcesoutputs_remainder.json | jq -r '.properties.outputs.llapClusterName.value')
 
-    echo $sparkCluster
-    echo $llapCluster
+    echo "Spark Cluster" $sparkClusterName
+    echo "LLAP cluster" $llapClusterName
     ```
 
 1. To view the Azure storage account and access key, enter the following command:
 
     ```azurecli
-    blobStorageName=$(cat resourcesoutputs.json | jq -r '.properties.outputs.blobStorageName.value')
+    blobStorageName=$(cat resourcesoutputs_storage.json | jq -r '.properties.outputs.blobStorageName.value')
 
     blobKey=$(az storage account keys list \
         --account-name $blobStorageName \
@@ -166,7 +134,7 @@ The default password for SSH access to the clusters is `Thisisapassword1`. If yo
 1. To view the Data Lake Storage Gen2 account and access key, enter the following command:
 
     ```azurecli
-    ADLSGen2StorageName=$(cat resourcesoutputs.json | jq -r '.properties.outputs.adlsGen2StorageName.value')
+    ADLSGen2StorageName=$(cat resourcesoutputs_storage.json | jq -r '.properties.outputs.adlsGen2StorageName.value')
 
     adlsKey=$(az storage account keys list \
         --account-name $ADLSGen2StorageName \
@@ -186,10 +154,13 @@ This data factory will have one pipeline with two activities:
 * The first activity will copy the data from Azure Blob storage to the Data Lake Storage Gen 2 storage account to mimic data ingestion.
 * The second activity will transform the data in the Spark cluster. The script transforms the data by removing unwanted columns. It also appends a new column that calculates the revenue that a single transaction generates.
 
-To set up your Azure Data Factory pipeline, execute the following command:
+To set up your Azure Data Factory pipeline, execute the  command below.  You should still be at the `hdinsight-sales-insights-etl` directory.
 
 ```bash
-./scripts/adf.sh
+blobStorageName=$(cat resourcesoutputs_storage.json | jq -r '.properties.outputs.blobStorageName.value')
+ADLSGen2StorageName=$(cat resourcesoutputs_storage.json | jq -r '.properties.outputs.adlsGen2StorageName.value')
+
+./scripts/adf.sh $resourceGroup $ADLSGen2StorageName $blobStorageName
 ```
 
 This script does the following things:
@@ -200,35 +171,47 @@ This script does the following things:
 1. Obtains storage keys for the Data Lake Storage Gen2 and Blob storage accounts.
 1. Creates another resource deployment to create an Azure Data Factory pipeline, with its associated linked services and activities. It passes the storage keys as parameters to the template file so that the linked services can access the storage accounts correctly.
 
-The Data Factory pipeline is deployed through the following command:
-
-```azurecli-interactive
-az group deployment create --name ADFDeployment \
-    --resource-group $resourceGroup \
-    --template-file adftemplate.json \
-    --parameters "@adfparameters.json"
-```
-
 ## Run the data pipeline
 
 ### Trigger the Data Factory activities
 
 The first activity in the Data Factory pipeline that you've created moves the data from Blob storage to Data Lake Storage Gen2. The second activity applies the Spark transformations on the data and saves the transformed .csv files to a new location. The entire pipeline might take a few minutes to finish.
 
-To trigger the pipelines, you can either:
+To retrieve the Data Factory name, enter the following command:
 
-* Trigger the Data Factory pipelines in PowerShell. Replace `DataFactoryName` with the actual Data Factory name, then run the following commands:
+```azurecli
+cat resourcesoutputs_adf.json | jq -r '.properties.outputs.factoryName.value'
+```
+
+To trigger the pipeline, you can either:
+
+* Trigger the Data Factory pipeline in PowerShell. Replace `RESOURCEGROUP`, and `DataFactoryName` with the appropriate values, then run the following commands:
 
     ```powershell
-    Invoke-AzDataFactoryV2Pipeline -DataFactory DataFactoryName -PipelineName "CopyPipeline_k8z"
-    Invoke-AzDataFactoryV2Pipeline -DataFactory DataFactoryName -PipelineName "sparkTransformPipeline"
+    # If you have multiple subscriptions, set the one to use
+    # Select-AzSubscription -SubscriptionId "<SUBSCRIPTIONID>"
+
+    $resourceGroup="RESOURCEGROUP"
+    $dataFactory="DataFactoryName"
+
+    $pipeline =Invoke-AzDataFactoryV2Pipeline `
+        -ResourceGroupName $resourceGroup `
+        -DataFactory $dataFactory `
+        -PipelineName "IngestAndTransform"
+
+    Get-AzDataFactoryV2PipelineRun `
+        -ResourceGroupName $resourceGroup  `
+        -DataFactoryName $dataFactory `
+        -PipelineRunId $pipeline
     ```
 
+    Re-execute `Get-AzDataFactoryV2PipelineRun` as needed to monitor progress.
+
     Or
 
-* Open the data factory and select **Author & Monitor**. Trigger the copy pipeline and then the Spark pipeline from the portal. For information on triggering pipelines through the portal, see [Create on-demand Apache Hadoop clusters in HDInsight using Azure Data Factory](hdinsight-hadoop-create-linux-clusters-adf.md#trigger-a-pipeline).
+* Open the data factory and select **Author & Monitor**. Trigger the `IngestAndTransform` pipeline from the portal. For information on triggering pipelines through the portal, see [Create on-demand Apache Hadoop clusters in HDInsight using Azure Data Factory](hdinsight-hadoop-create-linux-clusters-adf.md#trigger-a-pipeline).
 
-To verify that the pipelines have run, you can take either of the following steps:
+To verify that the pipeline has run, you can take either of the following steps:
 
 * Go to the **Monitor** section in your data factory through the portal.
 * In Azure Storage Explorer, go to your Data Lake Storage Gen 2 storage account. Go to the `files` file system, and then go to the `transformed` folder and check its contents to see if the pipeline succeeded.
@@ -237,37 +220,48 @@ For other ways to transform data by using HDInsight, see [this article on using
 
 ### Create a table on the Interactive Query cluster to view data on Power BI
 
-1. Copy the `query.hql` file to the LLAP cluster by using SCP. Replace `LLAPCLUSTERNAME` with the actual name, then enter the command:
+1. Copy the `query.hql` file to the LLAP cluster by using SCP. Enter the command:
 
     ```bash
-    scp scripts/query.hql [email protected]:/home/sshuser/
+    llapClusterName=$(cat resourcesoutputs_remainder.json | jq -r '.properties.outputs.llapClusterName.value')
+    scp scripts/query.hql sshuser@$llapClusterName-ssh.azurehdinsight.net:/home/sshuser/
     ```
 
-2. Use SSH to access the LLAP cluster. Replace `LLAPCLUSTERNAME` with the actual name, then enter the command. If you haven't altered the `resourcesparameters.json` file, the password is `Thisisapassword1`.
+    Reminder: The default password is `Thisisapassword1`.
+
+1. Use SSH to access the LLAP cluster. Enter the command:
 
     ```bash
-    ssh sshuser@LLAPCLUSTERNAME-ssh.azurehdinsight.net
+    ssh sshuser@$llapClusterName-ssh.azurehdinsight.net
     ```
 
-3. Use the following command to run the script:
+1. Use the following command to run the script:
 
     ```bash
     beeline -u 'jdbc:hive2://localhost:10001/;transportMode=http' -f query.hql
     ```
 
-This script will create a managed table on the Interactive Query cluster that you can access from Power BI.
+    This script will create a managed table on the Interactive Query cluster that you can access from Power BI.
 
 ### Create a Power BI dashboard from sales data
 
 1. Open Power BI Desktop.
-1. Select **Get Data**.
-1. Search for **HDInsight Interactive Query cluster**.
-1. Paste the URI for your cluster there. It should be in the format `https://LLAPCLUSTERNAME.azurehdinsight.net`.
 
-   Enter `default` for the database.
-1. Enter the username and password that you use to access the cluster.
+1. From the menu, navigate to **Get data** > **More...** > **Azure** > **HDInsight Interactive Query**.
 
-After the data is loaded, you can experiment with the dashboard that you want to create. See the following links to get started with Power BI dashboards:
+1. Select **Connect**.
+
+1. From the **HDInsight Interactive Query** dialog:
+    1. In the **Server** text box, enter the name of your LLAP cluster in the format of `https://LLAPCLUSTERNAME.azurehdinsight.net`.
+    1. In the **database** text box, enter `default`.
+    1. Select **OK**.
+
+1. From the **AzureHive** dialog:
+    1. In the **User name** text box, enter `admin`.
+    1. In the **Password** text box, enter `Thisisapassword1`.
+    1. Select **Connect**.
+
+1. From **Navigator**, select `sales`, and/or `sales_raw` to preview the data. After the data is loaded, you can experiment with the dashboard that you want to create. See the following links to get started with Power BI dashboards:
 
 * [Introduction to dashboards for Power BI designers](https://docs.microsoft.com/power-bi/service-dashboards)
 * [Tutorial: Get started with the Power BI service](https://docs.microsoft.com/power-bi/service-get-started)
@@ -276,9 +270,18 @@ After the data is loaded, you can experiment with the dashboard that you want to
 
 If you're not going to continue to use this application, delete all resources by using the following command so that you aren't charged for them.
 
-```azurecli-interactive
-az group delete -n $resourceGroup
-```
+1. To remove the resource group, enter the command:
+
+    ```azurecli
+    az group delete -n $resourceGroup
+    ```
+
+1. To remove the service principal, enter the commands:
+
+    ```azurecli
+    servicePrincipal=$(cat serviceprincipal.json | jq -r '.name')
+    az ad sp delete --id $servicePrincipal
+    ```
 
 ## Next steps