Merge pull request #97395 from dagiro/freshness81

freshness81

Author: GitHubber17

articles/hdinsight/hadoop/apache-hadoop-use-sqoop-mac-linux.md

@@ -1,15 +1,13 @@
 ---
 title: Apache Sqoop with Apache Hadoop - Azure HDInsight 
 description: Learn how to use Apache Sqoop to import and export between Apache Hadoop on HDInsight and an Azure SQL Database.
-keywords: hadoop sqoop,sqoop
-
 author: hrasheed-msft
 ms.author: hrasheed
 ms.reviewer: jasonh
 ms.service: hdinsight
-ms.custom: hdinsightactive,hdiseo17may2017
 ms.topic: conceptual
-ms.date: 04/15/2019
+ms.custom: hdinsightactive,hdiseo17may2017
+ms.date: 11/28/2019
 ---
 
 # Use Apache Sqoop to import and export data between Apache Hadoop on HDInsight and SQL Database
@@ -22,58 +20,120 @@ Learn how to use Apache Sqoop to import and export between an Apache Hadoop clus
 
 * Completion of [Set up test environment](./hdinsight-use-sqoop.md#create-cluster-and-sql-database) from [Use Apache Sqoop with Hadoop in HDInsight](./hdinsight-use-sqoop.md).
 
-* A client to query the Azure SQL database. Consider using [SQL Server Management Studio](../../sql-database/sql-database-connect-query-ssms.md) or [Visual Studio Code](../../sql-database/sql-database-connect-query-vscode.md).
-
 * An SSH client. For more information, see [Connect to HDInsight (Apache Hadoop) using SSH](../hdinsight-hadoop-linux-use-ssh-unix.md).
 
-## Sqoop export
+* Familiarity with Sqoop. For more information, see [Sqoop User Guide](https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html).
 
-From Hive to SQL Server.
+## Set up
 
-1. Use SSH to connect to the HDInsight cluster. Replace `CLUSTERNAME` with the name of your cluster, then enter the command:
+1. Use [ssh command](../hdinsight-hadoop-linux-use-ssh-unix.md) to connect to your cluster. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
 
     ```cmd
     ssh [email protected]
     ```
 
-2. Replace `MYSQLSERVER` with the name of your SQL Server. To verify that Sqoop can see your SQL Database, enter the command below in your open SSH connection. Enter the password for the SQL Server login when prompted. This command returns a list of databases.
+1. For ease of use, set variables. Replace `PASSWORD`, `MYSQLSERVER`, and `MYDATABASE` with the relevant values, and then enter the commands below:
 
     ```bash
-    sqoop list-databases --connect jdbc:sqlserver://MYSQLSERVER.database.windows.net:1433 --username sqluser -P
+    export password='PASSWORD'
+    export sqlserver="MYSQLSERVER"
+    export database="MYDATABASE"
+
+
+    export serverConnect="jdbc:sqlserver://$sqlserver.database.windows.net:1433;user=sqluser;password=$password"
+    export serverDbConnect="jdbc:sqlserver://$sqlserver.database.windows.net:1433;user=sqluser;password=$password;database=$database"
     ```
 
-3. Replace `MYSQLSERVER` with the name of your SQL Server, and `MYDATABASE` with the name of your SQL database. To export data from the Hive `hivesampletable` table to the `mobiledata` table in SQL Database, enter the command below in your open SSH connection. Enter the password for the SQL Server login when prompted
+## Sqoop export
+
+From Hive to SQL Server.
+
+1. To verify that Sqoop can see your SQL Database, enter the command below in your open SSH connection. This command returns a list of databases.
+
+    ```bash
+    sqoop list-databases --connect $serverConnect
+    ```
+
+1. Enter the following command to see a list of tables for the specified database:
 
     ```bash
-    sqoop export --connect 'jdbc:sqlserver://MYSQLSERVER.database.windows.net:1433;database=MYDATABASE' --username sqluser -P -table 'mobiledata' --hcatalog-table hivesampletable
+    sqoop list-tables --connect $serverDbConnect
     ```
 
-4. To verify that data was exported, use the following queries from your SQL client to view the exported data:
+1. To export data from the Hive `hivesampletable` table to the `mobiledata` table in SQL Database, enter the command below in your open SSH connection:
 
-    ```sql
-    SELECT COUNT(*) FROM [dbo].[mobiledata] WITH (NOLOCK);
-    SELECT TOP(25) * FROM [dbo].[mobiledata] WITH (NOLOCK);
+    ```bash
+    sqoop export --connect $serverDbConnect \
+    -table mobiledata \
+    --hcatalog-table hivesampletable
+    ```
+
+1. To verify that data was exported, use the following queries from your SSH connection to view the exported data:
+
+    ```bash
+    sqoop eval --connect $serverDbConnect \
+    --query "SELECT COUNT(*) from dbo.mobiledata WITH (NOLOCK)"
+
+
+    sqoop eval --connect $serverDbConnect \
+    --query "SELECT TOP(10) * from dbo.mobiledata WITH (NOLOCK)"
     ```
 
 ## Sqoop import
 
 From SQL Server to Azure storage.
 
-1. Replace `MYSQLSERVER` with the name of your SQL Server, and `MYDATABASE` with the name of your SQL database. Enter the command below in your open SSH connection to import data from the `mobiledata` table in SQL Database, to the `wasb:///tutorials/usesqoop/importeddata` directory on HDInsight. Enter the password for the SQL Server login when prompted. The fields in the data are separated by a tab character, and the lines are terminated by a new-line character.
+1. Enter the command below in your open SSH connection to import data from the `mobiledata` table in SQL Database, to the `wasbs:///tutorials/usesqoop/importeddata` directory on HDInsight. The fields in the data are separated by a tab character, and the lines are terminated by a new-line character.
+
+    ```bash
+    sqoop import --connect $serverDbConnect \
+    --table mobiledata \
+    --target-dir 'wasb:///tutorials/usesqoop/importeddata' \
+    --fields-terminated-by '\t' \
+    --lines-terminated-by '\n' -m 1
+    ```
+
+1. Alternatively, you can also specify a Hive table:
 
     ```bash
-    sqoop import --connect 'jdbc:sqlserver://MYSQLSERVER.database.windows.net:1433;database=MYDATABASE' --username sqluser -P --table 'mobiledata' --target-dir 'wasb:///tutorials/usesqoop/importeddata' --fields-terminated-by '\t' --lines-terminated-by '\n' -m 1
+    sqoop import --connect $serverDbConnect \
+    --table mobiledata \
+    --target-dir 'wasb:///tutorials/usesqoop/importeddata2' \
+    --fields-terminated-by '\t' \
+    --lines-terminated-by '\n' \
+    --create-hive-table \
+    --hive-table mobiledata_imported2 \
+    --hive-import -m 1
     ```
 
-2. Once the import has completed, enter the following command in your open SSH connection to list out the data in the new directory:
+1. Once the import has completed, enter the following command in your open SSH connection to list out the data in the new directory:
 
     ```bash
-    hdfs dfs -text /tutorials/usesqoop/importeddata/part-m-00000
+    hadoop fs -tail /tutorials/usesqoop/importeddata/part-m-00000
     ```
 
+1. Use [beeline](./apache-hadoop-use-hive-beeline.md) to verify that the table has been created in Hive.
+
+    1. Connect
+
+        ```bash
+        beeline -u 'jdbc:hive2://headnodehost:10001/;transportMode=http'
+        ```
+
+    1. Execute each query below one at a time and review the output:
+
+        ```hql
+        show tables;
+        describe mobiledata_imported2;
+        SELECT COUNT(*) FROM mobiledata_imported2;
+        SELECT * FROM mobiledata_imported2 LIMIT 10;
+        ```
+
+    1. Exit beeline with `!exit`.
+
 ## Limitations
 
-* Bulk export - With Linux-based HDInsight, the Sqoop connector used to export data to Microsoft SQL Server or Azure SQL Database does not support bulk inserts.
+* Bulk export - With Linux-based HDInsight, the Sqoop connector used to export data to Microsoft SQL Server or Azure SQL Database doesn't support bulk inserts.
 
 * Batching - With Linux-based HDInsight, When using the `-batch` switch when performing inserts, Sqoop makes multiple inserts instead of batching the insert operations.
 
@@ -91,7 +151,7 @@ From SQL Server to Azure storage.
 
 ## Next steps
 
-Now you have learned how to use Sqoop. To learn more, see:
+Now you've learned how to use Sqoop. To learn more, see:
 
 * [Use Apache Oozie with HDInsight](../hdinsight-use-oozie-linux-mac.md): Use Sqoop action in an Oozie workflow.
 * [Analyze flight delay data using HDInsight](../interactive-query/interactive-query-tutorial-analyze-flight-data.md): Use Interactive Query to analyze flight delay data, and then use Sqoop to export data to an Azure SQL database.