Skip to content

Commit 8820b3d

Browse files
American-DipperAmerican-Dipper
authored
Merge pull request #272804 from schaffererin/userstory246322
New Fleet doc: Use taints and tolerations to schedule workloads in Fleet
2 parents 9e32559 + ce5a1e6 commit 8820b3d

File tree

5 files changed

+335
-2
lines changed

5 files changed

+335
-2
lines changed

‎articles/kubernetes-fleet/concepts-fleet.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: "Azure Kubernetes Fleet Manager and member clusters"
33
description: This article provides a conceptual overview of Azure Kubernetes Fleet Manager and member clusters.
4-
ms.date: 04/01/2024
4+
ms.date: 04/23/2024
55
author: shashankbarsin
66
ms.author: shasb
77
ms.service: kubernetes-fleet
@@ -25,10 +25,22 @@ A fleet consists of the following components:
2525

2626
## What are member clusters?
2727

28-
The `MemberCluster` represents a cluster-scoped API established within the hub cluster, serving as a representation of a cluster within the fleet. This API offers a dependable, uniform, and automated approach for multi-cluster applications to identify registered clusters within a fleet. It also facilitates applications in querying a list of clusters managed by the fleet or in observing cluster statuses for subsequent actions. For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/MemberCluster/README.md).
28+
The `MemberCluster` represents a cluster-scoped API established within the hub cluster, serving as a representation of a cluster within the fleet. This API offers a dependable, uniform, and automated approach for multi-cluster applications to identify registered clusters within a fleet. It also facilitates applications in querying a list of clusters managed by the fleet or in observing cluster statuses for subsequent actions.
2929

3030
You can join Azure Kubernetes Service (AKS) clusters to a fleet as member clusters. Member clusters must reside in the same Microsoft Entra tenant as the fleet, but they can be in different regions, different resource groups, and/or different subscriptions.
3131

32+
### Taints
33+
34+
Member clusters support the specification of taints, which apply to the `MemberCluster` resource. Each taint object consists of the following fields:
35+
36+
* `key`: The key of the taint.
37+
* `value`: The value of the taint.
38+
* `effect`: The effect of the taint, such as `NoSchedule`.
39+
40+
Once a `MemberCluster` is tainted, it lets the [scheduler](./concepts-scheduler-scheduling-framework.md) know that the cluster shouldn't receive resources as part of the [resource propagation](./concepts-resource-propagation.md) from the hub cluster. The `NoSchedule` effect is a signal to the scheduler to avoid scheduling resources from a [`ClusterResourcePlacement`](./concepts-resource-propagation.md#what-is-a-clusterresourceplacement) to the `MemberCluster`.
41+
42+
For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/MemberCluster/README.md).
43+
3244
## What is a hub cluster (preview)?
3345

3446
[!INCLUDE [preview features note](./includes/preview/preview-callout.md)]

‎articles/kubernetes-fleet/concepts-resource-propagation.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,19 @@ The Fleet scheduler prioritizes the stability of existing workload placements. T
330330
331331
Resource-only changes (updating the resources or updating the `ResourceSelector` in the `ClusterResourcePlacement` object) roll out gradually in existing placements but do **not** trigger rescheduling of the workload.
332332
333+
## Tolerations
334+
335+
`ClusterResourcePlacement` objects support the specification of tolerations, which apply to the `ClusterResourcePlacement` object. Each toleration object consists of the following fields:
336+
337+
* `key`: The key of the toleration.
338+
* `value`: The value of the toleration.
339+
* `effect`: The effect of the toleration, such as `NoSchedule`.
340+
* `operator`: The operator of the toleration, such as `Exists` or `Equal`.
341+
342+
Each toleration is used to tolerate one or more specific taints applied on the `ClusterResourcePlacement`. Once all taints on a [`MemberCluster`](./concepts-fleet.md#what-are-member-clusters) are tolerated, the scheduler can then propagate resources to the cluster. You can't update or remove tolerations from a `ClusterResourcePlacement` object once it's created.
343+
344+
For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/ClusterResourcePlacement/README.md#tolerations).
345+
333346
## Access the Kubernetes API of the Fleet resource cluster
334347
335348
If you created an Azure Kubernetes Fleet Manager resource with the hub cluster enabled, you can use it to centrally control scenarios like Kubernetes object propagation. To access the Kubernetes API of the Fleet resource cluster, follow the steps in [Access the Kubernetes API of the Fleet resource cluster with Azure Kubernetes Fleet Manager](./quickstart-access-fleet-kubernetes-api.md).

‎articles/kubernetes-fleet/concepts-scheduler-scheduling-framework.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ The fleet scheduler integrates the following fundamental built-in plugins:
4141
* **Cluster affinity plugin**: Facilitates the affinity clause in the placement policy.
4242
* **Same placement affinity plugin**: Designed specifically for fleet and prevents multiple replicas from being placed within the same cluster.
4343
* **Cluster eligibility plugin**: Enables cluster selection based on specific status criteria.
44+
* **Taint & toleration plugin**: Enables cluster selection based on [taints on the cluster](./concepts-fleet.md#taints) and [tolerations on the `ClusterResourcePlacement`](./concepts-resource-propagation.md#tolerations).
4445

4546
For more information, see the [upstream Fleet Scheduling Framework documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/Scheduling-Framework/README.md).
4647

‎articles/kubernetes-fleet/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@
3434
href: update-orchestration.md
3535
- name: Multi-cluster Layer 4 Load Balancing
3636
href: l4-load-balancing.md
37+
- name: Use taints and tolerations to schedule workloads
38+
href: use-taints-tolerations.md
3739
- name: Resources
3840
items:
3941
- name: Pricing

‎articles/kubernetes-fleet/use-taints-tolerations.md

Lines changed: 305 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,305 @@
1+
---
2+
title: "Use taints on member clusters and tolerations on cluster resource placements in Azure Kubernetes Fleet Manager"
3+
description: Learn how to use taints on `MemberCluster` resources and tolerations on `ClusterResourcePlacement` resources in Azure Kubernetes Fleet Manager.
4+
ms.topic: how-to
5+
ms.date: 04/23/2024
6+
author: schaffererin
7+
ms.author: schaffererin
8+
ms.service: kubernetes-fleet
9+
---
10+
11+
# Use taints on member clusters and tolerations on cluster resource placements
12+
13+
This article explains how to add/remove taints on `MemberCluster` resources and tolerations on `ClusterResourcePlacement` resources in Azure Kubernetes Fleet Manager.
14+
15+
Taints and tolerations work together to ensure member clusters only receive specified resources during resource propagation. Taints are applied to `MemberCluster` resources to prevent resources from being propagated to the member cluster. Tolerations are applied to `ClusterResourcePlacement` resources to allow resources to be propagated to the member cluster, even if the member cluster has a taint.
16+
17+
## Prerequisites
18+
19+
* [!INCLUDE [free trial note](../../includes/quickstarts-free-trial-note.md)]
20+
* Read the conceptual overviews for [taints](./concepts-fleet.md#taints) and [tolerations](./concepts-resource-propagation.md#tolerations).
21+
* You must have a Fleet resource with a hub cluster and member clusters. If you don't have this resource, follow [Quickstart: Create a Fleet resource and join member clusters](quickstart-create-fleet-and-members.md).
22+
* You must gain access to the Kubernetes API of the hub cluster by following the steps in [Access the Kubernetes API of the Fleet resource](./quickstart-access-fleet-kubernetes-api.md).
23+
24+
## Add taints to a member cluster
25+
26+
In this example, we add a taint to a `MemberCluster` resource, then try to propagate resources to the member cluster using a `ClusterResourcePlacement` with a `PickAll` placement policy. The resources shouldn't be propagated to the member cluster because of the taint.
27+
28+
1. Create a namespace to propagate to the member cluster using the `kubectl create ns` command.
29+
30+
```bash
31+
kubectl create ns test-ns
32+
```
33+
34+
2. Create a taint on the `MemberCluster` resource using the following example code:
35+
36+
```yml
37+
apiVersion: cluster.kubernetes-fleet.io/v1beta1
38+
kind: MemberCluster
39+
metadata:
40+
name: kind-cluster-1
41+
spec:
42+
identity:
43+
name: fleet-member-agent-cluster-1
44+
kind: ServiceAccount
45+
namespace: fleet-system
46+
apiGroup: ""
47+
taints: # Add taint to the member cluster
48+
- key: test-key1
49+
value: test-value1
50+
effect: NoSchedule
51+
```
52+
53+
3. Apply the taint to the `MemberCluster` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
54+
55+
```bash
56+
kubectl apply -f member-cluster-taint.yml
57+
```
58+
59+
4. Create a `PickAll` placement policy on the `ClusterResourcePlacement` resource using the following example code:
60+
61+
```yml
62+
resourceSelectors:
63+
- group: ""
64+
kind: Namespace
65+
version: v1
66+
name: test-ns
67+
policy:
68+
placementType: PickAll
69+
```
70+
71+
5. Apply the `ClusterResourcePlacement` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
72+
73+
```bash
74+
kubectl apply -f cluster-resource-placement-pick-all.yml
75+
```
76+
77+
6. Verify that the resources weren't propagated to the member cluster by checking the details of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
78+
79+
```bash
80+
kubectl describe clusterresourceplacement test-ns
81+
```
82+
83+
Your output should look similar to the following example output:
84+
85+
```output
86+
status:
87+
conditions:
88+
- lastTransitionTime: "2024-04-16T19:03:17Z"
89+
message: found all the clusters needed as specified by the scheduling policy
90+
observedGeneration: 2
91+
reason: SchedulingPolicyFulfilled
92+
status: "True"
93+
type: ClusterResourcePlacementScheduled
94+
- lastTransitionTime: "2024-04-16T19:03:17Z"
95+
message: All 0 cluster(s) are synchronized to the latest resources on the hub
96+
cluster
97+
observedGeneration: 2
98+
reason: SynchronizeSucceeded
99+
status: "True"
100+
type: ClusterResourcePlacementSynchronized
101+
- lastTransitionTime: "2024-04-16T19:03:17Z"
102+
message: There are no clusters selected to place the resources
103+
observedGeneration: 2
104+
reason: ApplySucceeded
105+
status: "True"
106+
type: ClusterResourcePlacementApplied
107+
observedResourceIndex: "0"
108+
selectedResources:
109+
- kind: Namespace
110+
name: test-ns
111+
version: v1
112+
```
113+
114+
## Remove taints from a member cluster
115+
116+
In this example, we remove the taint we created in [add taints to a member cluster](#add-taints-to-a-member-cluster). This should automatically trigger the Fleet scheduler to propagate the resources to the member cluster.
117+
118+
1. Open your `MemberCluster` YAML file and remove the taint section.
119+
2. Apply the changes to the `MemberCluster` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
120+
121+
```bash
122+
kubectl apply -f member-cluster-taint.yml
123+
```
124+
125+
3. Verify that the resources were propagated to the member cluster by checking the details of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
126+
127+
```bash
128+
kubectl describe clusterresourceplacement test-ns
129+
```
130+
131+
Your output should look similar to the following example output:
132+
133+
```output
134+
status:
135+
conditions:
136+
- lastTransitionTime: "2024-04-16T20:00:03Z"
137+
message: found all the clusters needed as specified by the scheduling policy
138+
observedGeneration: 2
139+
reason: SchedulingPolicyFulfilled
140+
status: "True"
141+
type: ClusterResourcePlacementScheduled
142+
- lastTransitionTime: "2024-04-16T20:02:57Z"
143+
message: All 1 cluster(s) are synchronized to the latest resources on the hub
144+
cluster
145+
observedGeneration: 2
146+
reason: SynchronizeSucceeded
147+
status: "True"
148+
type: ClusterResourcePlacementSynchronized
149+
- lastTransitionTime: "2024-04-16T20:02:57Z"
150+
message: Successfully applied resources to 1 member clusters
151+
observedGeneration: 2
152+
reason: ApplySucceeded
153+
status: "True"
154+
type: ClusterResourcePlacementApplied
155+
observedResourceIndex: "0"
156+
placementStatuses:
157+
- clusterName: kind-cluster-1
158+
conditions:
159+
- lastTransitionTime: "2024-04-16T20:02:52Z"
160+
message: 'Successfully scheduled resources for placement in kind-cluster-1 (affinity
161+
score: 0, topology spread score: 0): picked by scheduling policy'
162+
observedGeneration: 2
163+
reason: ScheduleSucceeded
164+
status: "True"
165+
type: Scheduled
166+
- lastTransitionTime: "2024-04-16T20:02:57Z"
167+
message: Successfully Synchronized work(s) for placement
168+
observedGeneration: 2
169+
reason: WorkSynchronizeSucceeded
170+
status: "True"
171+
type: WorkSynchronized
172+
- lastTransitionTime: "2024-04-16T20:02:57Z"
173+
message: Successfully applied resources
174+
observedGeneration: 2
175+
reason: ApplySucceeded
176+
status: "True"
177+
type: Applied
178+
selectedResources:
179+
- kind: Namespace
180+
name: test-ns
181+
version: v1
182+
```
183+
184+
## Add tolerations to a cluster resource placement
185+
186+
In this example, we add a toleration to a `ClusterResourcePlacement` resource to propagate resources to a member cluster that has a taint. The toleration allows the resources to be propagated to the member cluster.
187+
188+
1. Create a namespace to propagate to the member cluster using the `kubectl create ns` command.
189+
190+
```bash
191+
kubectl create ns test-ns
192+
```
193+
194+
2. Create a taint on the `MemberCluster` resource using the following example code:
195+
196+
```yml
197+
apiVersion: cluster.kubernetes-fleet.io/v1beta1
198+
kind: MemberCluster
199+
metadata:
200+
name: kind-cluster-1
201+
spec:
202+
identity:
203+
name: fleet-member-agent-cluster-1
204+
kind: ServiceAccount
205+
namespace: fleet-system
206+
apiGroup: ""
207+
taints: # Add taint to the member cluster
208+
- key: test-key1
209+
value: test-value1
210+
effect: NoSchedule
211+
```
212+
213+
3. Apply the taint to the `MemberCluster` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
214+
215+
```bash
216+
kubectl apply -f member-cluster-taint.yml
217+
```
218+
219+
4. Create a toleration on the `ClusterResourcePlacement` resource using the following example code:
220+
221+
```yml
222+
spec:
223+
policy:
224+
placementType: PickAll
225+
tolerations:
226+
- key: test-key1
227+
operator: Exists
228+
resourceSelectors:
229+
- group: ""
230+
kind: Namespace
231+
name: test-ns
232+
version: v1
233+
revisionHistoryLimit: 10
234+
strategy:
235+
type: RollingUpdate
236+
```
237+
238+
5. Apply the `ClusterResourcePlacement` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
239+
240+
```bash
241+
kubectl apply -f cluster-resource-placement-toleration.yml
242+
```
243+
244+
6. Verify that the resources were propagated to the member cluster by checking the details of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
245+
246+
```bash
247+
kubectl describe clusterresourceplacement test-ns
248+
```
249+
250+
Your output should look similar to the following example output:
251+
252+
```output
253+
status:
254+
conditions:
255+
- lastTransitionTime: "2024-04-16T20:16:10Z"
256+
message: found all the clusters needed as specified by the scheduling policy
257+
observedGeneration: 3
258+
reason: SchedulingPolicyFulfilled
259+
status: "True"
260+
type: ClusterResourcePlacementScheduled
261+
- lastTransitionTime: "2024-04-16T20:16:15Z"
262+
message: All 1 cluster(s) are synchronized to the latest resources on the hub
263+
cluster
264+
observedGeneration: 3
265+
reason: SynchronizeSucceeded
266+
status: "True"
267+
type: ClusterResourcePlacementSynchronized
268+
- lastTransitionTime: "2024-04-16T20:16:15Z"
269+
message: Successfully applied resources to 1 member clusters
270+
observedGeneration: 3
271+
reason: ApplySucceeded
272+
status: "True"
273+
type: ClusterResourcePlacementApplied
274+
observedResourceIndex: "0"
275+
placementStatuses:
276+
- clusterName: kind-cluster-1
277+
conditions:
278+
- lastTransitionTime: "2024-04-16T20:16:10Z"
279+
message: 'Successfully scheduled resources for placement in kind-cluster-1 (affinity
280+
score: 0, topology spread score: 0): picked by scheduling policy'
281+
observedGeneration: 3
282+
reason: ScheduleSucceeded
283+
status: "True"
284+
type: Scheduled
285+
- lastTransitionTime: "2024-04-16T20:16:15Z"
286+
message: Successfully Synchronized work(s) for placement
287+
observedGeneration: 3
288+
reason: WorkSynchronizeSucceeded
289+
status: "True"
290+
type: WorkSynchronized
291+
- lastTransitionTime: "2024-04-16T20:16:15Z"
292+
message: Successfully applied resources
293+
observedGeneration: 3
294+
reason: ApplySucceeded
295+
status: "True"
296+
type: Applied
297+
selectedResources:
298+
- kind: Namespace
299+
name: test-ns
300+
version: v1
301+
```
302+
303+
## Next steps
304+
305+
For more information on Azure Kubernetes Fleet Manager, see the [upstream Fleet documentation](https://github.com/Azure/fleet/tree/main/docs).

0 commit comments

Comments
 (0)