You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-hadoop-create-linux-clusters-with-secure-transfer-storage.md
+15-46Lines changed: 15 additions & 46 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,47 +7,35 @@ ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
8
ms.topic: conceptual
9
9
ms.custom: hdinsightactive
10
-
ms.date: 01/22/2020
10
+
ms.date: 02/18/2020
11
11
---
12
12
13
-
# Create Apache Hadoop cluster with secure transfer storage accounts in Azure HDInsight
13
+
# Apache Hadoop clusters with secure transfer storage accounts in Azure HDInsight
14
14
15
15
The [Secure transfer required](../storage/common/storage-require-secure-transfer.md) feature enhances the security of your Azure Storage account by enforcing all requests to your account through a secure connection. This feature and the wasbs scheme are only supported by HDInsight cluster version 3.6 or newer.
16
16
17
-
**Enabling secure storage transfer after creating a cluster can result in errors using your storage account and is not recommended. It is better to create a new cluster with the property enabled.**
17
+
> [!IMPORTANT]
18
+
> Enabling secure storage transfer after creating a cluster can result in errors using your storage account and is not recommended. It is better to create a new cluster using a storage account with secure transfer already enabled.
18
19
19
-
## Prerequisites
20
+
## Storage accounts
20
21
21
-
Before you begin this article, you must have:
22
+
### Azure portal
22
23
23
-
* Azure subscription: To create a free one-month trial account, browse to [azure.microsoft.com/free](https://azure.microsoft.com/free).
24
-
* An Azure Storage account with secure transfer enabled. For the instructions, see [Create a storage account](../storage/common/storage-account-create.md) and [Require secure transfer](../storage/common/storage-require-secure-transfer.md).
25
-
* A Blob container on the storage account.
24
+
By default, the secure transfer required property is enabled when you create a storage account in Azure portal.
26
25
27
-
## Create cluster
26
+
To update an existing storage account with Azure portal, see [Require secure transfer with Azure portal](../storage/common/storage-require-secure-transfer.md#require-secure-transfer-for-an-existing-storage-account).
In this section, you create a Hadoop cluster in HDInsight using an [Azure Resource Manager template](../azure-resource-manager/templates/deploy-powershell.md). The template is located in [GitHub](https://azure.microsoft.com/resources/templates/101-hdinsight-linux-with-existing-default-storage-account/). Resource Manager template experience isn't required for following this article. For other cluster creation methods and understanding the properties used in this article, see [Create HDInsight clusters](hdinsight-hadoop-provision-linux-clusters.md).
30
+
For the PowerShell cmdlet [New-AzStorageAccount](https://docs.microsoft.com/powershell/module/az.storage/new-azstorageaccount), ensure parameter `-EnableHttpsTrafficOnly` is set to `1`.
32
31
33
-
1. Click the following image to sign in to Azure and open the Resource Manager template in the Azure portal.
32
+
To update an existing storage account with PowerShell, see [Require secure transfer with PowerShell](../storage/common/storage-require-secure-transfer.md#require-secure-transfer-with-powershell).
34
33
35
-
<ahref="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-hdinsight-linux-with-existing-default-storage-account%2Fazuredeploy.json"target="_blank"><imgsrc="./media/hdinsight-hadoop-create-linux-clusters-with-secure-transfer-storage/hdi-deploy-to-azure1.png"alt="Deploy to Azure button for new cluster"></a>
34
+
### Azure CLI
36
35
37
-
2. Follow the instructions to create the cluster with the following specifications:
36
+
For the Azure CLI command [az storage account create](https://docs.microsoft.com/cli/azure/storage/account?view=azure-cli-latest#az-storage-account-create), ensure parameter `--https-only` is set to `true`.
38
37
39
-
* Specify HDInsight version 3.6. Version 3.6 or newer is required.
40
-
* Specify a secure transfer enabled storage account.
41
-
* Use short name for the storage account.
42
-
* Both the storage account and the blob container must be created beforehand.
43
-
44
-
For the instructions, see [Create cluster](hadoop/apache-hadoop-linux-tutorial-get-started.md#create-cluster).
45
-
46
-
If you use script action to provide your own configuration files, you must use wasbs in the following settings:
47
-
48
-
* fs.defaultFS (core-site)
49
-
* spark.eventLog.dir
50
-
* spark.history.fs.logDirectory
38
+
To update an existing storage account with Azure CLI, see [Require secure transfer with Azure CLI](../storage/common/storage-require-secure-transfer.md#require-secure-transfer-with-azure-cli).
51
39
52
40
## Add additional storage accounts
53
41
@@ -59,25 +47,6 @@ There are several options to add additional secure transfer enabled storage acco
59
47
60
48
## Next steps
61
49
62
-
In this article, you've learned how to create an HDInsight cluster, and enable secure transfer to the storage accounts.
63
-
64
-
To learn more about analyzing data with HDInsight, see the following articles:
65
-
66
-
* To learn more about using [Apache Hive](https://hive.apache.org/) with HDInsight, including how to perform Hive queries from Visual Studio, see [Use Apache Hive with HDInsight](hadoop/hdinsight-use-hive.md).
67
-
* To learn about [Apache Hadoop MapReduce](https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html), a way to write programs that process data on Hadoop, see [Use Apache Hadoop MapReduce with HDInsight](hadoop/hdinsight-use-mapreduce.md).
68
-
* To learn about using the HDInsight Tools for Visual Studio to analyze data on HDInsight, see [Get started using Visual Studio Apache Hadoop tools for HDInsight](hadoop/apache-hadoop-visual-studio-tools-get-started.md).
69
-
70
-
To learn more about how HDInsight stores data or how to get data into HDInsight, see the following articles:
71
-
50
+
* The use of Azure Storage (WASB) instead of [Apache Hadoop HDFS](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html) as the default data store
72
51
* For information on how HDInsight uses Azure Storage, see [Use Azure Storage with HDInsight](hdinsight-hadoop-use-blob-storage.md).
73
52
* For information on how to upload data to HDInsight, see [Upload data to HDInsight](hdinsight-upload-data.md).
74
-
75
-
To learn more about creating or managing an HDInsight cluster, see the following articles:
76
-
77
-
* To learn about managing your Linux-based HDInsight cluster, see [Manage HDInsight clusters using Apache Ambari](hdinsight-hadoop-manage-ambari.md).
78
-
* To learn more about the options you can select when creating an HDInsight cluster, see [Creating HDInsight on Linux using custom options](hdinsight-hadoop-provision-linux-clusters.md).
79
-
* If you're familiar with Linux, and Apache Hadoop, but want to know specifics about Hadoop on the HDInsight, see [Working with HDInsight on Linux](hdinsight-hadoop-linux-information.md). This article provides information such as:
80
-
81
-
* URLs for services hosted on the cluster, such as [Apache Ambari](https://ambari.apache.org/) and [WebHCat](https://cwiki.apache.org/confluence/display/Hive/WebHCat)
82
-
* The location of [Apache Hadoop](https://hadoop.apache.org/) files and examples on the local file system
83
-
* The use of Azure Storage (WASB) instead of [Apache Hadoop HDFS](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html) as the default data store
0 commit comments