You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-upload-data.md
+14-20Lines changed: 14 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,20 +8,21 @@ ms.author: hrasheed
8
8
ms.service: hdinsight
9
9
ms.custom: hdinsightactive,hdiseo17may2017
10
10
ms.topic: conceptual
11
-
ms.date: 02/08/2019
11
+
ms.date: 05/31/2019
12
12
---
13
+
13
14
# Upload data for Apache Hadoop jobs in HDInsight
14
15
15
-
Azure HDInsight provides a full-featured Hadoop distributed file system (HDFS) over Azure Storage and Azure Data Lake Storage (Gen1 and Gen2). Azure Storage and Data Lake Storage Gen1 and Gen2 are designed as HDFS extensions to provide a seamless experience to customers. They enable the full set of components in the Hadoop ecosystem to operate directly on the data it manages. Azure Storage, Data Lake Storage Gen1, and Gen2 are distinct file systems that are optimized for storage of data and computations on that data. For information about the benefits of using Azure Storage, see [Use Azure Storage with HDInsight][hdinsight-storage], [Use Data Lake Storage Gen1 with HDInsight](hdinsight-hadoop-use-data-lake-store.md), and [Use Data Lake Storage Gen2 with HDInsight](hdinsight-hadoop-use-data-lake-storage-gen2.md).
16
+
Azure HDInsight provides a full-featured Hadoop distributed file system (HDFS) over Azure Storage and Azure Data Lake Storage (Gen1 and Gen2). Azure Storage and Data Lake Storage Gen1 and Gen2 are designed as HDFS extensions to provide a seamless experience to customers. They enable the full set of components in the Hadoop ecosystem to operate directly on the data it manages. Azure Storage, Data Lake Storage Gen1, and Gen2 are distinct file systems that are optimized for storage of data and computations on that data. For information about the benefits of using Azure Storage, see [Use Azure Storage with HDInsight](hdinsight-hadoop-use-blob-storage.md), [Use Data Lake Storage Gen1 with HDInsight](hdinsight-hadoop-use-data-lake-store.md), and [Use Data Lake Storage Gen2 with HDInsight](hdinsight-hadoop-use-data-lake-storage-gen2.md).
16
17
17
18
## Prerequisites
18
19
19
20
Note the following requirements before you begin:
20
21
21
-
* An Azure HDInsight cluster. For instructions, see [Get started with Azure HDInsight][hdinsight-get-started] or [Create HDInsight clusters](hdinsight-hadoop-provision-linux-clusters.md).
22
+
* An Azure HDInsight cluster. For instructions, see [Get started with Azure HDInsight](hadoop/apache-hadoop-linux-tutorial-get-started.md) or [Create HDInsight clusters](hdinsight-hadoop-provision-linux-clusters.md).
22
23
* Knowledge of the following articles:
23
24
24
-
-[Use Azure Storage with HDInsight][hdinsight-storage]
25
+
-[Use Azure Storage with HDInsight](hdinsight-hadoop-use-blob-storage.md)
25
26
-[Use Data Lake Storage Gen1 with HDInsight](hdinsight-hadoop-use-data-lake-store.md)
26
27
-[Use Data Lake Storage Gen2 with HDInsight](hdinsight-hadoop-use-data-lake-storage-gen2.md)
27
28
@@ -58,11 +59,11 @@ For example, `hadoop fs -copyFromLocal data.txt /example/data/data.txt`
58
59
59
60
Because the default file system for HDInsight is in Azure Storage, /example/data.txt is actually in Azure Storage. You can also refer to the file as:
For a list of other Hadoop commands that work with files, see [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html)
68
69
@@ -98,7 +99,7 @@ The Azure Data Factory service is a fully managed service for composing data sto
98
99
### <aid="sqoop"></a>Apache Sqoop
99
100
Sqoop is a tool designed to transfer data between Hadoop and relational databases. You can use it to import data from a relational database management system (RDBMS), such as SQL Server, MySQL, or Oracle into the Hadoop distributed file system (HDFS), transform the data in Hadoop with MapReduce or Hive, and then export the data back into an RDBMS.
100
101
101
-
For more information, see [Use Sqoop with HDInsight][hdinsight-use-sqoop].
102
+
For more information, see [Use Sqoop with HDInsight](hadoop/hdinsight-use-sqoop.md).
102
103
103
104
### Development SDKs
104
105
Azure Storage can also be accessed using an Azure SDK from the following programming languages:
You can also increase the value of `fs.azure.write.request.size` globally by using Apache Ambari. The following steps can be used to change the value in the Ambari Web UI:
148
149
149
-
1. In your browser, go to the Ambari Web UI for your cluster. This is https://CLUSTERNAME.azurehdinsight.net, where **CLUSTERNAME** is the name of your cluster.
150
+
1. In your browser, go to the Ambari Web UI for your cluster. This is `https://CLUSTERNAME.azurehdinsight.net`, where `CLUSTERNAME` is the name of your cluster.
150
151
151
152
When prompted, enter the admin name and password for the cluster.
152
153
2. From the left side of the screen, select **HDFS**, and then select the **Configs** tab.
153
154
3. In the **Filter...** field, enter `fs.azure.write.request.size`. This displays the field and current value in the middle of the page.
154
155
4. Change the value from 262144 (256 KB) to the new value. For example, 4194304 (4 MB).
155
156
156
-

157
+

157
158
158
159
For more information on using Ambari, see [Manage HDInsight clusters using the Apache Ambari Web UI](hdinsight-hadoop-manage-ambari.md).
159
160
160
161
## Next steps
161
162
Now that you understand how to get data into HDInsight, read the following articles to learn how to perform analysis:
162
163
163
-
*[Get started with Azure HDInsight][hdinsight-get-started]
164
-
*[Submit Apache Hadoop jobs programmatically][hdinsight-submit-jobs]
165
-
*[Use Apache Hive with HDInsight][hdinsight-use-hive]
166
-
*[Use Apache Pig with HDInsight][hdinsight-use-pig]
0 commit comments