You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The sizing of any cluster to meet job performance and manage costs ahead of time is always tricky, and hard to determine! One of the lucrative benefits of building data lake house over Cloud is its elasticity, which means to use autoscale feature to maximize the utilization of resources at hand. Auto scale with Kubernetes is one key to establishing a cost optimized ecosystem. With varied usage patterns in any enterprise, there could be variations in cluster loads over time that could lead to clusters being under-provisioned (lousy performance) or overprovisioned (unnecessary costs due to idle resources).
14
13
@@ -128,7 +127,7 @@ The default value is set to **180 seconds**
128
127
129
128
:::image type="content" source="./media/hdinsight-on-aks-autoscale-clusters/add-permissions-concept-step-5.png" alt-text="Screenshot showing how to add role assignment." border="true" lightbox="./media/hdinsight-on-aks-autoscale-clusters/add-permissions-concept-step-5.png":::
This article describes how to enable Log Analytics to monitor & collect logs for cluster pool and cluster operations on HDInsight on AKS. You can enable the integration during cluster pool creation or post the creation.
14
12
Once the integration at cluster pool is enabled, it isn't possible to disable the integration. However, you can disable the log analytics for individual clusters, which are part of the same pool.
Cluster pools are a logical grouping of clusters and maintain a set of clusters in the same pool. It helps in building robust interoperability across multiple cluster types and allow enterprises to have the clusters in the same virtual network. One cluster pool corresponds to one cluster in AKS infrastructure.
14
12
15
13
This article describes how to manage a cluster pool.
Clusters are individual compute workloads such as Apache Spark, Apache Flink, and Trino, which can be created rapidly in few minutes with preset configurations and few clicks.
14
12
15
13
This article describes how to manage a cluster using Azure portal.
@@ -38,7 +36,7 @@ You can view the cluster details in the "Overview" blade of your cluster. It pro
38
36
|Resource group| The resource group in which cluster is created.|
39
37
|Cluster pool name| Cluster pool name inside which the cluster is created.|
40
38
|Cluster type| The type of the cluster such as Spark, Trino, or Flink.|
41
-
|HDInsight on AKS version| HDInsight on AKS cluster version. For more information. see [versioning](./versions.md).|
39
+
|HDInsight on AKS version| HDInsight on AKS cluster version. For more information, see [versioning](./versions.md).|
42
40
|Cluster endpoint| The endpoint of the cluster.|
43
41
|Cluster package| Component versions associated with the cluster.|
44
42
|Subscription details| Subscription name and subscription ID.|
Azure HDInsight on AKS provides a mechanism called **Script Actions** that invoke custom scripts to customize the cluster. These scripts are used to install additional components and change configuration settings. Script actions can be provisioned only during cluster creation as of now. Post cluster creation, Script Actions are part of the roadmap.
13
11
This article explains how you can provision script actions when you create an HDInsight on AKS cluster.
14
12
15
13
## Use a script action during cluster creation using Azure portal
16
14
17
-
1. Upload the script action in a `ADLS/WASB` storage(does not have to be the primary cluster storage). In this example we will consider an `ADLS` storage.
15
+
1. Upload the script action in a `ADLS/WASB` storage(doesn't have to be the primary cluster storage). In this example we consider an `ADLS` storage.
18
16
To upload a script into your storage, navigate into the target storage and the container where you want to upload it.
19
17
20
-
:::image type="content" source="./media/manage-script-actions/upload-script-action-1.png" alt-text="Screenshot showing the how to select container." border="true" lightbox="./media/manage-script-actions/upload-script-action-1.png":::
18
+
:::image type="content" source="./media/manage-script-actions/upload-script-action-1.png" alt-text="Screenshot showing how to select container." border="true" lightbox="./media/manage-script-actions/upload-script-action-1.png":::
21
19
22
20
1. To upload a script into your storage, navigate into the target storage and the container. Click on the upload button and select the script from your local drive.
23
-
After the script gets uploaded you should be able to see it in the container(see below image).
21
+
After the script gets uploaded you should be able to see it in the container(see following image).
24
22
25
23
:::image type="content" source="./media/manage-script-actions/upload-script-action-2.png" alt-text="Screenshot showing how to upload the script." border="true" lightbox="./media/manage-script-actions/upload-script-action-2.png":::
26
24
@@ -32,7 +30,7 @@ This article explains how you can provision script actions when you create an HD
32
30
33
31
This action opens the Script Action window. Provide the following details:
34
32
35
-
:::image type="content" source="./media/manage-script-actions/manage-script-action-add-step-2.png" alt-text="Screenshot showing the Add Script action window opens in the Azure portal.":::
33
+
:::image type="content" source="./media/manage-script-actions/manage-script-action-add-step-2.png" alt-text="Screenshot showing how to add Script action window opens in the Azure portal.":::
36
34
37
35
|Property|Description|
38
36
|-|-|
@@ -45,7 +43,7 @@ This article explains how you can provision script actions when you create an HD
45
43
:::image type="content" source="./media/manage-script-actions/manage-script-action-add-node-type-step-3.png" alt-text="Screenshot showing the list of services where to the apply the script actions." border="true" lightbox="./media/manage-script-actions/manage-script-action-add-node-type-step-3.png":::
46
44
47
45
> [!NOTE]
48
-
> * All the Script Actions will be persisted.
46
+
> * All the Script Actions persisted.
49
47
> * Script actions are available only for Apache Spark cluster type.
HDInsight on AKS provides elasticity with options to scale up and scale down the number of cluster nodes. This elasticity works to help increase resource utilization and improve cost efficiency.
When an enterprise wants to use their own virtual network for the cluster deployments, securing the traffic of the virtual network becomes important.
15
14
This article provides the steps to secure outbound traffic from your HDInsight on AKS cluster via Azure Firewall using [Azure CLI](/azure/cloud-shell/quickstart?tabs=azurecli).
HDInsight on AKS relies on AKS outbound dependencies and they're entirely defined with FQDNs, which don't have static addresses behind them. The lack of static IP addresses means one can't use Network Security Groups (NSGs) to lock down the outbound traffic from the cluster using IPs.
14
12
15
13
If you still prefer to use NSG to secure your traffic, then you need to configure the following rules in the NSG to do a coarse-grained control.
![prmerger-automator[bot]](https://avatars.githubusercontent.com/u/22479449?v=4&size=40){kind=avatar}
0 commit comments