You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/aks/operator-best-practices-cluster-isolation.md
+44-29Lines changed: 44 additions & 29 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,34 +1,46 @@
1
1
---
2
-
title: Best practices for cluster isolation
2
+
title: Cluster isolation best practices for Azure Kubernetes Service (AKS)
3
3
titleSuffix: Azure Kubernetes Service
4
-
description: Learn the cluster operator best practices for isolation in Azure Kubernetes Service (AKS)
4
+
description: Learn the cluster operator best practices for isolation in Azure Kubernetes Service (AKS).
5
5
ms.topic: conceptual
6
-
ms.date: 03/09/2021
6
+
ms.date: 05/25/2023
7
7
8
8
---
9
9
10
10
# Best practices for cluster isolation in Azure Kubernetes Service (AKS)
11
11
12
-
As you manage clusters in Azure Kubernetes Service (AKS), you often need to isolate teams and workloads. AKS provides flexibility in how you can run multi-tenant clusters and isolate resources. To maximize your investment in Kubernetes, first understand and implement AKS multi-tenancy and isolation features.
12
+
As you manage clusters in Azure Kubernetes Service (AKS), you often need to isolate teams and workloads. AKS allows flexibility in how you run multi-tenant clusters and isolate resources. To maximize your investment in Kubernetes, it's important you understand AKS multi-tenancy and isolation features.
13
13
14
14
This best practices article focuses on isolation for cluster operators. In this article, you learn how to:
15
15
16
16
> [!div class="checklist"]
17
-
> * Plan for multi-tenant clusters and separation of resources
18
-
> * Use logical or physical isolation in your AKS clusters
17
+
>
18
+
> * Plan for multi-tenant clusters and separation of resources.
19
+
> * Use logical or physical isolation in your AKS clusters.
19
20
20
21
## Design clusters for multi-tenancy
21
22
22
-
Kubernetes lets you logically isolate teams and workloads in the same cluster. The goal is to provide the least number of privileges, scoped to the resources each team needs. A Kubernetes [Namespace][k8s-namespaces] creates a logical isolation boundary. Additional Kubernetes features and considerations for isolation and multi-tenancy include the following areas:
23
+
Kubernetes lets you logically isolate teams and workloads in the same cluster. The goal is to provide the least number of privileges scoped to the resources each team needs. A Kubernetes [Namespace][k8s-namespaces] creates a logical isolation boundary. Other Kubernetes features and considerations for isolation and multi-tenancy include the following areas:
24
+
25
+
-[Best practices for cluster isolation in Azure Kubernetes Service (AKS)](#best-practices-for-cluster-isolation-in-azure-kubernetes-service-aks)
26
+
-[Design clusters for multi-tenancy](#design-clusters-for-multi-tenancy)
27
+
-[Scheduling](#scheduling)
28
+
-[Networking](#networking)
29
+
-[Authentication and authorization](#authentication-and-authorization)
*Scheduling* uses basic features such as resource quotas and pod disruption budgets. For more information about these features, see [Best practices for basic scheduler features in AKS][aks-best-practices-scheduler].
37
+
*Scheduling* uses basic features like resource quotas and pod disruption budgets. For more information about these features, see [Best practices for basic scheduler features in AKS][aks-best-practices-scheduler].
27
38
28
39
More advanced scheduler features include:
29
-
* Taints and tolerations
30
-
* Node selectors
31
-
* Node and pod affinity or anti-affinity.
40
+
41
+
* Taints and tolerations.
42
+
* Node selectors.
43
+
* Node and pod affinity or anti-affinity.
32
44
33
45
For more information about these features, see [Best practices for advanced scheduler features in AKS][aks-best-practices-advanced-scheduler].
34
46
@@ -39,53 +51,56 @@ For more information about these features, see [Best practices for advanced sche
39
51
### Authentication and authorization
40
52
41
53
*Authentication and authorization* uses:
42
-
* Role-based access control (RBAC)
43
-
* Azure Active Directory (AD) integration
44
-
* Pod identities
45
-
* Secrets in Azure Key Vault
54
+
55
+
* Role-based access control (RBAC).
56
+
* Azure Active Directory (AD) integration.
57
+
* Pod identities.
58
+
* Secrets in Azure Key Vault.
46
59
47
60
For more information about these features, see [Best practices for authentication and authorization in AKS][aks-best-practices-identity].
48
61
49
62
### Containers
63
+
50
64
*Containers* include:
51
-
* The Azure Policy Add-on for AKS to enforce pod security.
52
-
* The use of pod security admission.
53
-
* Scanning both images and the runtime for vulnerabilities.
65
+
66
+
* The Azure Policy add-on for AKS to enforce pod security.
67
+
* Pod security admission.
68
+
* Scanning images and runtime for vulnerabilities.
54
69
* Using App Armor or Seccomp (Secure Computing) to restrict container access to the underlying node.
55
70
56
-
## Logically isolate clusters
71
+
## Logically isolated clusters
57
72
58
73
> **Best practice guidance**
59
74
>
60
75
> Separate teams and projects using *logical isolation*. Minimize the number of physical AKS clusters you deploy to isolate teams or applications.
61
76
62
-
With logical isolation, a single AKS cluster can be used for multiple workloads, teams, or environments. Kubernetes [Namespaces][k8s-namespaces] form the logical isolation boundary for workloads and resources.
77
+
With logical isolation, you can use a single AKS cluster for multiple workloads, teams, or environments. Kubernetes [Namespaces][k8s-namespaces] form the logical isolation boundary for workloads and resources.
63
78
64
79
65
80
66
-
Logical separation of clusters usually provides a higher pod density than physically isolated clusters, with less excess compute capacity sitting idle in the cluster. When combined with the Kubernetes cluster autoscaler, you can scale the number of nodes up or down to meet demands. This best practice approach to autoscaling minimizes costs by running only the number of nodes required.
81
+
Logical separation of clusters usually provides a higher pod density than physically isolated clusters, with less excess compute capacity sitting idle in the cluster. When combined with the Kubernetes cluster autoscaler, you can scale the number of nodes up or down to meet demands. This best practice approach minimizes costs by running only the required number of nodes.
67
82
68
-
Currently, Kubernetes environments aren't completely safe for hostile multi-tenant usage. In a multi-tenant environment, multiple tenants are working on a common, shared infrastructure. If all tenants cannot be trusted, you will need extra planning to prevent tenants from impacting the security and service of others.
83
+
Kubernetes environments aren't entirely safe for hostile multi-tenant usage. In a multi-tenant environment, multiple tenants work on a shared infrastructure. If all tenants can't be trusted, you need extra planning to prevent tenants from impacting the security and service of others.
69
84
70
-
Additional security features, like Kubernetes RBAC for nodes, efficiently block exploits. For true security when running hostile multi-tenant workloads, you should only trust a hypervisor. The security domain for Kubernetes becomes the entire cluster, not an individual node.
85
+
Other security features, like Kubernetes RBAC for nodes, efficiently block exploits. For true security when running hostile multi-tenant workloads, you should only trust a hypervisor. The security domain for Kubernetes becomes the entire cluster and not an individual node.
71
86
72
87
For these types of hostile multi-tenant workloads, you should use physically isolated clusters.
73
88
74
-
## Physically isolate clusters
89
+
## Physically isolated clusters
75
90
76
-
> **Best practice guidance**
91
+
> **Best practice guidance**
77
92
>
78
-
> Minimize the use of physical isolation for each separate team or application deployment. Instead, use *logical* isolation, as discussed in the previous section.
93
+
> Minimize the use of physical isolation for each separate team or application deployment and use *logical* isolation instead.
79
94
80
-
Physically separating AKS clusters is a common approach to cluster isolation. In this isolation model, teams or workloads are assigned their own AKS cluster. While physical isolation might look like the easiest way to isolate workloads or teams, it adds management and financial overhead. Now, you must maintain these multiple clusters and individually provide access and assign permissions. You'll also be billed for each the individual node.
95
+
Physically separating AKS clusters is a common approach to cluster isolation. In this isolation model, teams or workloads are assigned their own AKS cluster. While physical isolation might look like the easiest way to isolate workloads or teams, it adds management and financial overhead. With physically isolated clusters, you must maintain multiple clusters and individually provide access and assign permissions. You're also billed for each individual node.
81
96
82
97
83
98
84
-
Physically separate clusters usually have a low pod density. Since each team or workload has their own AKS cluster, the cluster is often over-provisioned with compute resources. Often, a small number of pods are scheduled on those nodes. Unclaimed node capacity can't be used for applications or services in development by other teams. These excess resources contribute to the additional costs in physically separate clusters.
99
+
Physically isolated clusters usually have a low pod density. Since each team or workload has their own AKS cluster, the cluster is often over-provisioned with compute resources. Often, a few pods are scheduled on those nodes. Unclaimed node capacity can't be used for applications or services in development by other teams. These excess resources contribute to the extra costs in physically isolated clusters.
85
100
86
101
## Next steps
87
102
88
-
This article focused on cluster isolation. For more information about cluster operations in AKS, see the following best practices:
103
+
This article focused on cluster isolation. For more information about cluster operations in AKS, see the following best practice articles:
0 commit comments