|
| 1 | +--- |
| 2 | +title: Availability sets in AKS enabled by Azure Arc |
| 3 | +description: Learn how to enable availability sets in AKS enabled by Arc to improve the availability and distribution of your Kubernetes workloads. |
| 4 | +ms.topic: how-to |
| 5 | +author: sethmanheim |
| 6 | +ms.date: 09/06/2024 |
| 7 | +ms.author: sethm |
| 8 | +ms.reviewer: rbaziwane |
| 9 | +ms.lastreviewed: 08/15/2024 |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +# Availability sets in AKS enabled by Azure Arc |
| 14 | + |
| 15 | +*Availability sets* are logical groups of VMs that have weak anti-affinity relationships with each other, to ensure that they are spread evenly across the available fault domains in a physical cluster. A fault domain in this context is a physical host or a group of physical hosts. By using availability sets, AKS Arc can improve the availability and distribution of your Kubernetes workloads. Availability sets can avoid scenarios in which a single node failure can cause multiple VMs to go down or become unbalanced. |
| 16 | + |
| 17 | +## Overview |
| 18 | + |
| 19 | +If you use AKS on Azure Stack HCI and Windows Server to run Kubernetes workloads on-premises, you might encounter some challenges with the current architecture. For example, you might notice that multiple virtual machines (VMs) within the same node pool can exist on the same physical host, which is not ideal for high availability. Or, you might see that VMs do not rebalance across physical hosts when a host recovers from an issue, resulting in uneven distribution of workloads. These issues can affect the performance and reliability of your applications, causing unnecessary disruption in your business operations. |
| 20 | + |
| 21 | +Availability sets offer several benefits for AKS on Azure Stack HCI and Windows Server users, such as: |
| 22 | + |
| 23 | +- Improves the availability and resilience of your applications by avoiding scenarios in which multiple VMs within the same node pool or control plane go down or become unbalanced due to a single node failure. |
| 24 | +- Optimizes the resource usage and performance of your cluster by ensuring that VMs are evenly distributed across the available nodes and not concentrated on a single node or a subset of nodes. |
| 25 | +- Aligns with the best practices and expectations of your customers and partners who are looking for a reliable and consistent on-premises Kubernetes experience. |
| 26 | + |
| 27 | +## Enable availability sets |
| 28 | + |
| 29 | +With AKS on Azure Stack HCI 23H2, the availability sets feature is enabled by default when you create a new node pool. |
| 30 | + |
| 31 | +With AKS on Azure Stack HCI 22H2, the availability sets feature is disabled by default. To enable it, add the `-enableAvailabilitySet` parameter when you create a workload cluster. For example: |
| 32 | + |
| 33 | +```powershell |
| 34 | +New-AksHciCluster -Name <name> -controlPlaneNodeCount 3 -osType Linux -kubernetesVersion $kubernetesVersion -enableAvailabilitySet |
| 35 | +``` |
| 36 | + |
| 37 | +## How availability sets work in AKS enabled by Azure Arc |
| 38 | + |
| 39 | +When you create a new AKS Arc cluster, AKS Arc automatically creates availability sets, one for the control plane VMs and another one for each of the node pools in the cluster. Each node pool has its own availability set. With this layout, AKS Arc ensures that VMs of the same role (control plane or node pool) are never located on the same physical host and that they're distributed across the available nodes in a cluster. |
| 40 | + |
| 41 | +Once the availability sets are created and the VMs assigned, the system automatically places them on the appropriate physical nodes. If a node fails, the system also automatically fails over the VMs to other nodes and rebalances them when the node recovers. This way, you can achieve high availability and optimal distribution of your Kubernetes workloads without manual intervention. |
| 42 | + |
| 43 | +Consider an AKS on Azure Stack HCI 23H2 cluster with two physical host machines, **Host A** and **Host B**, three control plane VMs, and two worker node VMs, **Nodepool1VM1** and **Nodepool1VM2**. To ensure high availability of your Kubernetes applications, the node pool VMs must never share the same host, unless one of the hosts is temporarily unavailable for planned maintenance or capacity issue, which can cause the VM (virtual machine) to be temporarily placed on an alternative host. |
| 44 | + |
| 45 | +In the following diagram, each color represents an anti-affinity group: |
| 46 | + |
| 47 | +:::image type="content" source="media/availability-sets/anti-affinity-1.png" alt-text="Diagram showing hosts in anti-affinity group."::: |
| 48 | + |
| 49 | +If **Host B** goes down due to a reboot, **Control Plane VM2**, **Control Plane VM3**, and **Nodepool1VM2** fail over to **Host A** as shown in the following figure. Assuming your application is running pods in **NodePoolVM1**, this reboot has no impact on your application: |
| 50 | + |
| 51 | +:::image type="content" source="media/availability-sets/anti-affinity-2.png" alt-text="Diagram showing host B down."::: |
| 52 | + |
| 53 | +In the old architecture, if **Host B** came back online after a reboot, there was no guarantee that the VMs would move back from Host A to Host B (rebalancing), thus forcing the workloads to stay on the same host, and create a single point of failure, as shown in the following diagram: |
| 54 | + |
| 55 | +:::image type="content" source="media/availability-sets/anti-affinity-3.png" alt-text="Diagram showing no rebalancing."::: |
| 56 | + |
| 57 | +Availability sets for AKS Arc can help to rebalance VMs once a host recovers from temporary outage. In this example, **ControlPlaneVM2**, **ControlPlaneVM3**, and **Nodepool1VM2** automatically move to **Host B**, as shown here: |
| 58 | + |
| 59 | +:::image type="content" source="media/availability-sets/anti-affinity-1.png" alt-text="Diagram showing hosts in anti-affinity group."::: |
| 60 | + |
| 61 | +> [!IMPORTANT] |
| 62 | +> Availability sets in AKS Arc are a new feature that's still evolving and improving. We do not yet support manual configuration of the fault domains or availability sets. You can't change the fault domains of an availability set after it's created. VMs are assigned to an availability set at cluster creation, and can't be migrated to a different availability set. |
| 63 | +
|
| 64 | +## Add or delete machines |
| 65 | + |
| 66 | +In a host deletion scenario, the host is no longer considered a part of the cluster. This deletion typically occurs when you replace a machine due to hardware issues, or scale down the HCI cluster for other reasons. During a node outage, the node remains part of the HCI cluster but appears as **Down**. |
| 67 | + |
| 68 | +If a physical machine (fault domain) is permanently deleted from the cluster, the availability set configuration isn't modified to reduce the number of fault domains. In this scenario, the availability set enters an unhealthy state. We recommend that you redeploy your workload clusters so that the availability set is updated with the proper number of fault domains. |
| 69 | + |
| 70 | +When a new physical machine (fault domain) is added to the cluster, the availability set configuration is automatically expanded to include the new machine. However, the existing VMs don't rebalance to apply this new configuration, since they are already assigned to availability sets. We recommend that you redeploy your workload clusters so that the availability set is updated with the proper number of fault domains. |
| 71 | + |
| 72 | +## Next steps |
| 73 | + |
| 74 | +[AKS overview](aks-overview.md) |
0 commit comments