You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-hadoop-create-linux-clusters-with-secure-transfer-storage.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,11 +9,13 @@ ms.custom: hdinsightactive
9
9
ms.topic: conceptual
10
10
ms.date: 07/24/2018
11
11
---
12
+
12
13
# Create Apache Hadoop cluster with secure transfer storage accounts in Azure HDInsight
13
14
14
15
The [Secure transfer required](../storage/common/storage-require-secure-transfer.md) feature enhances the security of your Azure Storage account by enforcing all requests to your account through a secure connection. This feature and the wasbs scheme are only supported by HDInsight cluster version 3.6 or newer.
15
16
16
17
## Prerequisites
18
+
17
19
Before you begin this article, you must have:
18
20
19
21
***Azure subscription**: To create a free one-month trial account, browse to [azure.microsoft.com/free](https://azure.microsoft.com/free).
@@ -24,12 +26,11 @@ Before you begin this article, you must have:
In this section, you create a Hadoop cluster in HDInsight using an [Azure Resource Manager template](../azure-resource-manager/resource-group-template-deploy.md). The template is located in [GitHub](https://azure.microsoft.com/resources/templates/101-hdinsight-linux-with-existing-default-storage-account/). Resource Manager template experience is not required for following this article. For other cluster creation methods and understanding the properties used in this article, see [Create HDInsight clusters](hdinsight-hadoop-provision-linux-clusters.md).
29
30
30
31
1. Click the following image to sign in to Azure and open the Resource Manager template in the Azure portal.
31
32
32
-
<ahref="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-hdinsight-linux-with-existing-default-storage-account%2Fazuredeploy.json"target="_blank"><imgsrc="./media/hdinsight-hadoop-create-linux-clusters-with-secure-transfer-storage/hdi-deploy-to-azure1.png"alt="Deploy to Azure"></a>
33
+
<ahref="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-hdinsight-linux-with-existing-default-storage-account%2Fazuredeploy.json"target="_blank"><imgsrc="./media/hdinsight-hadoop-create-linux-clusters-with-secure-transfer-storage/hdi-deploy-to-azure1.png"alt="Deploy to Azure button for new cluster"></a>
33
34
34
35
2. Follow the instructions to create the cluster with the following specifications:
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-hadoop-customize-cluster-bootstrap.md
+3-6Lines changed: 3 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,7 +12,7 @@ ms.date: 04/19/2019
12
12
13
13
# Customize HDInsight clusters using Bootstrap
14
14
15
-
Bootstrap scripts allow you to install and configure components in Azure HDInsight programmatically.
15
+
Bootstrap scripts allow you to install and configure components in Azure HDInsight programmatically.
16
16
17
17
There are three approaches to set configuration file settings as your HDInsight cluster is created:
18
18
@@ -52,7 +52,6 @@ The following PowerShell code customizes an [Apache Hive](https://hive.apache.or
52
52
> [!IMPORTANT]
53
53
> The parameter `Spark2Defaults` may need to be used with [Add-AzHDInsightConfigValue](https://docs.microsoft.com/powershell/module/az.hdinsight/add-azhdinsightconfigvalue). You can pass empty values to the parameter as shown in the code example below.
*[Create Apache Hadoop clusters in HDInsight][hdinsight-provision-cluster] provides instructions on how to create an HDInsight cluster by using other custom options.
132
132
*[Develop Script Action scripts for HDInsight][hdinsight-write-script]
133
133
*[Install and use Apache Spark on HDInsight clusters][hdinsight-install-spark]
@@ -137,8 +137,6 @@ You can use bootstrap in Resource Manager template:
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-hadoop-customize-cluster-linux.md
+22-24Lines changed: 22 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -40,12 +40,12 @@ Get more information on working with access management:
40
40
A script action is Bash script that runs on the nodes in an HDInsight cluster. Characteristics and features of script actions are as follows:
41
41
42
42
* Must be stored on a URI that's accessible from the HDInsight cluster. The following are possible storage locations:
43
-
43
+
44
44
* For regular clusters:
45
-
45
+
46
46
* ADLS Gen1: The service principal HDInsight uses to access Data Lake Storage must have read access to the script. The URI format for scripts stored in Data Lake Storage Gen1 is
* A blob in an Azure Storage account that's either the primary or additional storage account for the HDInsight cluster. HDInsight is granted access to both of these types of storage accounts during cluster creation.
50
50
51
51
> [!IMPORTANT]
@@ -56,9 +56,9 @@ A script action is Bash script that runs on the nodes in an HDInsight cluster. C
56
56
For example URIs, see [Example script action scripts](#example-script-action-scripts).
57
57
58
58
* For clusters with ESP:
59
-
59
+
60
60
* The wasb:// or wasbs:// or http[s]:// URIs are supported.
61
-
61
+
62
62
* Can be restricted to run on only certain node types. Examples are head nodes or worker nodes.
63
63
64
64
* Can be persisted or ad hoc.
@@ -103,15 +103,13 @@ The script runs while HDInsight is being configured. The script runs in parallel
103
103
> [!NOTE]
104
104
> You can perform operations like stopping and starting services, including Apache Hadoop-related services. If you stop services, make sure that the Ambari service and other Hadoop-related services are running before the script finishes. These services are required to successfully determine the health and state of the cluster while it's being created.
105
105
106
-
107
106
During cluster creation, you can use many script actions at once. These scripts are invoked in the order in which they were specified.
108
107
109
108
> [!IMPORTANT]
110
109
> Script actions must finish within 60 minutes, or they time out. During cluster provisioning, the script runs concurrently with other setup and configuration processes. Competition for resources such as CPU time or network bandwidth might cause the script to take longer to finish than it does in your development environment.
111
110
>
112
111
> To minimize the time it takes to run the script, avoid tasks like downloading and compiling applications from the source. Precompile applications and store the binary in Azure Storage.
113
112
114
-
115
113
### Script action on a running cluster
116
114
117
115
A failure in a script run on an already running cluster doesn't automatically cause the cluster to change to a failed state. After a script finishes, the cluster should return to a running state.
@@ -157,13 +155,13 @@ This section explains the different ways you can use script actions when you cre
157
155
158
156
1. Start to create a cluster as described in [Set up clusters in HDInsight with Apache Hadoop, Apache Spark, Apache Kafka, and more](hdinsight-hadoop-provision-linux-clusters.md). During cluster creation, you arrive at a __Cluster summary__ page. From the __Cluster summary__ page, select the __edit__ link for __Advanced settings__.
4. Use the __Select a script__ entry to select a premade script. To use a custom script, select __Custom__. Then provide the __Name__ and __Bash script URI__ for your script.
164
+
1. Use the __Select a script__ entry to select a premade script. To use a custom script, select __Custom__. Then provide the __Name__ and __Bash script URI__ for your script.
167
165
168
166
169
167
@@ -179,13 +177,13 @@ This section explains the different ways you can use script actions when you cre
179
177
180
178
Use the __Persist this script action__ entry to make sure that the script is applied during scaling operations.
181
179
182
-
5. Select __Create__ to save the script. Then you can use __+ Submit new__ to add another script.
180
+
1. Select __Create__ to save the script. Then you can use __+ Submit new__ to add another script.
When you're done adding scripts, select the __Select__ button and then the __Next__ button to return to the __Cluster summary__ section.
187
185
188
-
3. To create the cluster, select __Create__ from the __Cluster summary__ selection.
186
+
1. To create the cluster, select __Create__ from the __Cluster summary__ selection.
189
187
190
188
### Use a script action from Azure Resource Manager templates
191
189
@@ -245,7 +243,7 @@ Go to the [Azure portal](https://portal.azure.com):
245
243
246
244
247
245
248
-
4. Use the __Select a script__ entry to select a premade script. To use a custom script, select __Custom__. Then provide the __Name__ and __Bash script URI__ for your script.
246
+
1. Use the __Select a script__ entry to select a premade script. To use a custom script, select __Custom__. Then provide the __Name__ and __Bash script URI__ for your script.
249
247
250
248
251
249
@@ -261,7 +259,7 @@ Go to the [Azure portal](https://portal.azure.com):
261
259
262
260
Use the __Persist this script action__ entry to make sure the script is applied during scaling operations.
263
261
264
-
5. Finally, select the **Create** button to apply the script to the cluster.
262
+
1. Finally, select the **Create** button to apply the script to the cluster.
265
263
266
264
### Apply a script action to a running cluster from Azure PowerShell
267
265
@@ -342,17 +340,17 @@ For an example of using the .NET SDK to apply scripts to a cluster, see [Apply a
342
340
343
341
1. From the default view, under **Settings**, select **Script actions**.
344
342
345
-
4. A history of scripts for this cluster displays on the script actions section. This information includes a list of persisted scripts. The following screenshot shows that the Solr script has been run on this cluster. The screenshot doesn't show any persisted scripts.
343
+
1. A history of scripts for this cluster displays on the script actions section. This information includes a list of persisted scripts. The following screenshot shows that the Solr script has been run on this cluster. The screenshot doesn't show any persisted scripts.
5. Select a script from the history to display the **Properties** section for this script. From the top of the screen, you can rerun the script or promote it.
347
+
1. Select a script from the history to display the **Properties** section for this script. From the top of the screen, you can rerun the script or promote it.
@@ -411,7 +409,7 @@ Two types of open-source components are available in the HDInsight service:
411
409
> [!WARNING]
412
410
> Components provided with the HDInsight cluster are fully supported. Microsoft Support helps to isolate and resolve issues related to these components.
413
411
>
414
-
> Custom components receive commercially reasonable support to help you further troubleshoot the issue. Microsoft Support might be able to resolve the issue. Or they might ask you to engage available channels for the open-source technologies where deep expertise for that technology is found. Many community sites can be used. Examples are [MSDN forum for HDInsight](https://social.msdn.microsoft.com/Forums/azure/home?forum=hdinsight) and [Stack Overflow](https://stackoverflow.com).
412
+
> Custom components receive commercially reasonable support to help you further troubleshoot the issue. Microsoft Support might be able to resolve the issue. Or they might ask you to engage available channels for the open-source technologies where deep expertise for that technology is found. Many community sites can be used. Examples are [MSDN forum for HDInsight](https://social.msdn.microsoft.com/Forums/azure/home?forum=hdinsight) and [Stack Overflow](https://stackoverflow.com).
415
413
>
416
414
> Apache projects also have project sites on the [Apache website](https://apache.org). An example is [Hadoop](https://hadoop.apache.org/).
417
415
@@ -439,7 +437,7 @@ You can use the Ambari web UI to view information logged by script actions. If t
439
437
440
438
3. Find the entries that have **run\_customscriptaction**in the **Operations** column. These entries are created when the script actions run.
441
439
442
-
To view the **STDOUT** and **STDERR** output, selectthe**run\customscriptaction** entry and drill down through the links. This output is generated when the script runs and might have useful information.
0 commit comments