You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-apps-use-edge-node.md
+60-56Lines changed: 60 additions & 56 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,21 +1,20 @@
1
1
---
2
2
title: Use empty edge nodes on Apache Hadoop clusters in Azure HDInsight
3
3
description: How to add an empty edge node to an HDInsight cluster that can be used as a client, and then test/host your HDInsight applications.
4
-
ms.reviewer: jasonh
5
4
author: hrasheed-msft
6
-
5
+
ms.author: hrasheed
6
+
ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
-
ms.custom: hdinsightactive,hdiseo17may2017
9
8
ms.topic: conceptual
10
-
ms.date: 06/13/2019
11
-
ms.author: hrasheed
9
+
ms.custom: hdinsightactive,hdiseo17may2017
10
+
ms.date: 01/27/2020
12
11
---
13
12
14
13
# Use empty edge nodes on Apache Hadoop clusters in HDInsight
15
14
16
-
Learn how to add an empty edge node to an HDInsight cluster. An empty edge node is a Linux virtual machine with the same client tools installed and configured as in the headnodes, but with no [Apache Hadoop](https://hadoop.apache.org/) services running. You can use the edge node for accessing the cluster, testing your client applications, and hosting your client applications.
15
+
Learn how to add an empty edge node to an HDInsight cluster. An empty edge node is a Linux virtual machine with the same client tools installed and configured as in the headnodes, but with no [Apache Hadoop](https://hadoop.apache.org/) services running. You can use the edge node for accessing the cluster, testing your client applications, and hosting your client applications.
17
16
18
-
You can add an empty edge node to an existing HDInsight cluster, to a new cluster when you create the cluster. Adding an empty edge node is done using Azure Resource Manager template. The following sample demonstrates how it is done using a template:
17
+
You can add an empty edge node to an existing HDInsight cluster, to a new cluster when you create the cluster. Adding an empty edge node is done using Azure Resource Manager template. The following sample demonstrates how it's done using a template:
19
18
20
19
"resources": [
21
20
{
@@ -46,13 +45,13 @@ You can add an empty edge node to an existing HDInsight cluster, to a new cluste
46
45
}
47
46
],
48
47
49
-
As shown in the sample, you can optionally call a [script action](hdinsight-hadoop-customize-cluster-linux.md) to perform additional configuration, such as installing [Apache Hue](hdinsight-hadoop-hue-linux.md) in the edge node. The script action script must be publicly accessible on the web. For example, if the script is stored in Azure storage, use either public containers or public blobs.
48
+
As shown in the sample, you can optionally call a [script action](hdinsight-hadoop-customize-cluster-linux.md) to perform additional configuration, such as installing [Apache Hue](hdinsight-hadoop-hue-linux.md) in the edge node. The script action script must be publicly accessible on the web. For example, if the script is stored in Azure Storage, use either public containers or public blobs.
50
49
51
50
The edge node virtual machine size must meet the HDInsight cluster worker node vm size requirements. For the recommended worker node vm sizes, see [Create Apache Hadoop clusters in HDInsight](hdinsight-hadoop-provision-linux-clusters.md#cluster-types).
52
51
53
-
After you have created an edge node, you can connect to the edge node using SSH, and run client tools to access the Hadoop cluster in HDInsight.
52
+
After you've created an edge node, you can connect to the edge node using SSH, and run client tools to access the Hadoop cluster in HDInsight.
54
53
55
-
> [!WARNING]
54
+
> [!WARNING]
56
55
> Custom components that are installed on the edge node receive commercially reasonable support from Microsoft. This might result in resolving problems you encounter. Or, you may be referred to community resources for further assistance. The following are some of the most active sites for getting help from the community:
57
56
>
58
57
> *[MSDN forum for HDInsight](https://social.msdn.microsoft.com/Forums/azure/home?forum=hdinsight)
@@ -64,56 +63,61 @@ After you have created an edge node, you can connect to the edge node using SSH,
64
63
> Ubuntu images become available for new HDInsight cluster creation within 3 months of being published. As of January 2019, running clusters (including edge nodes) are **not** auto-patched. Customers must use script actions or other mechanisms to patch a running cluster. For more information, see [OS patching for HDInsight](./hdinsight-os-patching.md).
65
64
66
65
## Add an edge node to an existing cluster
67
-
In this section, you use a Resource Manager template to add an edge node to an existing HDInsight cluster. The Resource Manager template can be found in [GitHub](https://azure.microsoft.com/resources/templates/101-hdinsight-linux-add-edge-node/). The Resource Manager template calls a script action located at https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-hdinsight-linux-add-edge-node/scripts/EmptyNodeSetup.sh. The script doesn't perform any actions. It is to demonstrate calling script action from a Resource Manager template.
68
66
69
-
**To add an empty edge node to an existing cluster**
67
+
In this section, you use a Resource Manager template to add an edge node to an existing HDInsight cluster. The Resource Manager template can be found in [GitHub](https://azure.microsoft.com/resources/templates/101-hdinsight-linux-add-edge-node/). The Resource Manager template calls a script action located at https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-hdinsight-linux-add-edge-node/scripts/EmptyNodeSetup.sh. The script doesn't perform any actions. It's to demonstrate calling script action from a Resource Manager template.
68
+
69
+
1. Select the following image to sign in to Azure and open the Azure Resource Manager template in the Azure portal.
70
70
71
-
1. Click the following image to sign in to Azure and open the Azure Resource Manager template in the Azure portal.
72
-
73
71
<ahref="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-hdinsight-linux-add-edge-node%2Fazuredeploy.json"target="_blank"><imgsrc="./media/hdinsight-apps-use-edge-node/hdi-deploy-to-azure1.png"alt="Deploy to Azure button for new cluster"></a>
74
-
3. Configure the following properties:
75
-
76
-
***Subscription**: Select an Azure subscription used for creating the cluster.
77
-
***Resource group**: Select the resource group used for the existing HDInsight cluster.
78
-
***Location**: Select the location of the existing HDInsight cluster.
79
-
***Cluster Name**: Enter the name of an existing HDInsight cluster.
80
-
***Edge Node Size**: Select one of the VM sizes. The vm size must meet the worker node vm size requirements. For the recommended worker node vm sizes, see [Create Apache Hadoop clusters in HDInsight](hdinsight-hadoop-provision-linux-clusters.md#cluster-types).
81
-
***Edge Node Prefix**: The default value is **new**. Using the default value, the edge node name is **new-edgenode**. You can customize the prefix from the portal. You can also customize the full name from the template.
82
-
83
-
4. Check **I agree to the terms and conditions stated above**, and then click **Purchase** to create the edge node.
84
-
85
-
>[!IMPORTANT]
72
+
73
+
1. Configure the following properties:
74
+
75
+
|Property |Description |
76
+
|---|---|
77
+
|Subscription|Select an Azure subscription used for creating the cluster.|
78
+
|Resource group|Select the resource group used for the existing HDInsight cluster.|
79
+
|Location|Select the location of the existing HDInsight cluster.|
80
+
|Cluster Name|Enter the name of an existing HDInsight cluster.|
81
+
82
+
1. Check **I agree to the terms and conditions stated above**, and then select **Purchase** to create the edge node.
83
+
84
+
> [!IMPORTANT]
86
85
> Make sure to select the Azure resource group for the existing HDInsight cluster. Otherwise, you get the error message "Can not perform requested operation on nested resource. Parent resource '<ClusterName>' not found."
87
86
88
87
## Add an edge node when creating a cluster
89
-
In this section, you use a Resource Manager template to create HDInsight cluster with an edge node. The Resource Manager template can be found in the [Azure quickstart templates gallery](https://azure.microsoft.com/documentation/templates/101-hdinsight-linux-with-edge-node/). The Resource Manager template calls a script action located at https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-hdinsight-linux-with-edge-node/scripts/EmptyNodeSetup.sh. The script doesn't perform any actions. It is to demonstrate calling script action from a Resource Manager template.
90
88
91
-
**To create an HDInsight cluster with an edge node**
89
+
In this section, you use a Resource Manager template to create HDInsight cluster with an edge node. The Resource Manager template can be found in the [Azure quickstart templates gallery](https://azure.microsoft.com/documentation/templates/101-hdinsight-linux-with-edge-node/). The Resource Manager template calls a script action located at https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-hdinsight-linux-with-edge-node/scripts/EmptyNodeSetup.sh. The script doesn't perform any actions. It's to demonstrate calling script action from a Resource Manager template.
92
90
93
91
1. Create an HDInsight cluster if you don't have one yet. See [Get started using Hadoop in HDInsight](hadoop/apache-hadoop-linux-tutorial-get-started.md).
94
-
2. Click the following image to sign in to Azure and open the Azure Resource Manager template in the Azure portal.
95
-
92
+
93
+
1. Select the following image to sign in to Azure and open the Azure Resource Manager template in the Azure portal.
94
+
96
95
<ahref="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-hdinsight-linux-with-edge-node%2Fazuredeploy.json"target="_blank"><imgsrc="./media/hdinsight-apps-use-edge-node/hdi-deploy-to-azure1.png"alt="Deploy to Azure button for new cluster"></a>
97
-
3. Configure the following properties:
98
-
99
-
***Subscription**: Select an Azure subscription used for creating the cluster.
100
-
***Resource group**: Create a new resource group used for the cluster.
101
-
***Location**: Select a location for the resource group.
102
-
***Cluster Name**: Enter a name for the new cluster to create.
103
-
***Cluster Login User Name**: Enter the Hadoop HTTP user name. The default name is **admin**.
104
-
***Cluster Login Password**: Enter the Hadoop HTTP user password.
105
-
***Ssh User Name**: Enter the SSH user name. The default name is **sshuser**.
106
-
***Ssh Password**: Enter the SSH user password.
107
-
***Install Script Action**: Keep the default value for going through this article.
108
-
109
-
Some properties have been hardcoded in the template: Cluster type, Cluster worker node count, Edge node size, and Edge node name.
110
-
4. Check **I agree to the terms and conditions stated above**, and then click **Purchase** to create the cluster with the edge node.
96
+
97
+
1. Configure the following properties:
98
+
99
+
|Property |Description |
100
+
|---|---|
101
+
|Subscription|Select an Azure subscription used for creating the cluster.|
102
+
|Resource group|Create a new resource group used for the cluster.|
103
+
|Location|Select a location for the resource group.|
104
+
|Cluster Name|Enter a name for the new cluster to create.|
105
+
|Cluster Login User Name|Enter the Hadoop HTTP user name. The default name is **admin**.|
106
+
|Cluster Login Password|Enter the Hadoop HTTP user password.|
107
+
|Ssh User Name|Enter the SSH user name. The default name is **sshuser**.|
108
+
|Ssh Password|Enter the SSH user password.|
109
+
|Install Script Action|Keep the default value for going through this article.|
110
+
111
+
Some properties have been hardcoded in the template: Cluster type, Cluster worker node count, Edge node size, and Edge node name.
112
+
113
+
1. Check **I agree to the terms and conditions stated above**, and then select **Purchase** to create the cluster with the edge node.
111
114
112
115
## Add multiple edge nodes
113
116
114
117
You can add multiple edge nodes to an HDInsight cluster. The multiple edge nodes configuration can only be done using Azure Resource Manager Templates. See the template sample at the beginning of this article. You need to update the **targetInstanceCount** to reflect the number of edge nodes you would like to create.
115
118
116
119
## Access an edge node
120
+
117
121
The edge node ssh endpoint is <EdgeNodeName>.<ClusterName>-ssh.azurehdinsight.net:22. For example, new-edgenode.myedgenode0914-ssh.azurehdinsight.net:22.
118
122
119
123
The edge node appears as an application on the Azure portal. The portal gives you the information to access the edge node using SSH.
@@ -122,38 +126,38 @@ The edge node appears as an application on the Azure portal. The portal gives y
122
126
123
127
1. Sign on to the [Azure portal](https://portal.azure.com).
124
128
2. Open the HDInsight cluster with an edge node.
125
-
3.Click**Applications**. You shall see the edge node. The default name is **new-edgenode**.
126
-
4.Click the edge node. You shall see the SSH endpoint.
129
+
3.Select**Applications**. You shall see the edge node. The default name is **new-edgenode**.
130
+
4.Select the edge node. You shall see the SSH endpoint.
127
131
128
132
**To use Hive on the edge node**
129
133
130
134
1. Use SSH to connect to the edge node. For information, see [Use SSH with HDInsight](hdinsight-hadoop-linux-use-ssh-unix.md).
131
135
132
-
2. After you have connected to the edge node using SSH, use the following command to open the Hive console:
133
-
136
+
2. After you've connected to the edge node using SSH, use the following command to open the Hive console:
137
+
134
138
hive
139
+
135
140
3. Run the following command to show Hive tables in the cluster:
136
-
141
+
137
142
show tables;
138
143
139
144
## Delete an edge node
140
-
You can delete an edge node from the Azure portal.
141
145
142
-
**To access an edge node**
146
+
You can delete an edge node from the Azure portal.
143
147
144
148
1. Sign on to the [Azure portal](https://portal.azure.com).
145
149
2. Open the HDInsight cluster with an edge node.
146
-
3.Click**Applications**. You shall see a list of edge nodes.
147
-
4. Right-click the edge node you want to delete, and then click**Delete**.
148
-
5.Click**Yes** to confirm.
150
+
3.Select**Applications**. You shall see a list of edge nodes.
151
+
4. Right-click the edge node you want to delete, and then select**Delete**.
152
+
5.Select**Yes** to confirm.
149
153
150
154
## Next steps
151
-
In this article, you have learned how to add an edge node and how to access the edge node. To learn more, see the following articles:
155
+
156
+
In this article, you've learned how to add an edge node and how to access the edge node. To learn more, see the following articles:
152
157
153
158
*[Install HDInsight applications](hdinsight-apps-install-applications.md): Learn how to install an HDInsight application to your clusters.
154
159
*[Install custom HDInsight applications](hdinsight-apps-install-custom-applications.md): learn how to deploy an unpublished HDInsight application to HDInsight.
155
160
*[Publish HDInsight applications](hdinsight-apps-publish-applications.md): Learn how to publish your custom HDInsight applications to Azure Marketplace.
156
161
*[MSDN: Install an HDInsight application](https://msdn.microsoft.com/library/mt706515.aspx): Learn how to define HDInsight applications.
157
162
*[Customize Linux-based HDInsight clusters using Script Action](hdinsight-hadoop-customize-cluster-linux.md): learn how to use Script Action to install additional applications.
158
163
*[Create Linux-based Apache Hadoop clusters in HDInsight using Resource Manager templates](hdinsight-hadoop-create-linux-clusters-arm-templates.md): learn how to call Resource Manager templates to create HDInsight clusters.
0 commit comments