Skip to content

Commit 3c42181

Browse files
schaffererinschaffererin
committed
New Fleet doc: Use taints and tolerations to schedule workloadsin Fleet
1 parent 9ad81a4 commit 3c42181

File tree

5 files changed

+332
-1
lines changed

5 files changed

+332
-1
lines changed

‎articles/kubernetes-fleet/concepts-fleet.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,22 @@ A fleet consists of the following components:
2525

2626
## What are member clusters?
2727

28-
The `MemberCluster` represents a cluster-scoped API established within the hub cluster, serving as a representation of a cluster within the fleet. This API offers a dependable, uniform, and automated approach for multi-cluster applications to identify registered clusters within a fleet. It also facilitates applications in querying a list of clusters managed by the fleet or in observing cluster statuses for subsequent actions. For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/MemberCluster/README.md).
28+
The `MemberCluster` represents a cluster-scoped API established within the hub cluster, serving as a representation of a cluster within the fleet. This API offers a dependable, uniform, and automated approach for multi-cluster applications to identify registered clusters within a fleet. It also facilitates applications in querying a list of clusters managed by the fleet or in observing cluster statuses for subsequent actions.
2929

3030
You can join Azure Kubernetes Service (AKS) clusters to a fleet as member clusters. Member clusters must reside in the same Microsoft Entra tenant as the fleet, but they can be in different regions, different resource groups, and/or different subscriptions.
3131

32+
### Taints
33+
34+
Member clusters support the specification of taints, which apply to the `MemberCluster` resource. Each taint object consists of the following fields:
35+
36+
* `key`: The key of the taint.
37+
* `value`: The value of the taint.
38+
* `effect`: The effect of the taint, such as `NoSchedule`.
39+
40+
Once a `MemberCluster` is tainted, it lets the [scheduler](./concepts-scheduler-scheduling-framework.md) know that the cluster shouldn't receive resources as part of the [resource propagation](./concepts-resource-propagation.md) from the hub cluster. The `NoSchedule` effect is a signal to the scheduler to avoid scheduling resources from a [`ClusterResourcePlacement`](./concepts-resource-propagation.md#what-is-a-clusterresourceplacement) to the `MemberCluster`.
41+
42+
For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/MemberCluster/README.md).
43+
3244
## What is a hub cluster (preview)?
3345

3446
[!INCLUDE [preview features note](./includes/preview/preview-callout.md)]

‎articles/kubernetes-fleet/concepts-resource-propagation.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,19 @@ The Fleet scheduler prioritizes the stability of existing workload placements. T
330330
331331
Resource-only changes (updating the resources or updating the `ResourceSelector` in the `ClusterResourcePlacement` object) roll out gradually in existing placements but do **not** trigger rescheduling of the workload.
332332
333+
## Tolerations
334+
335+
`ClusterResourcePlacement` objects support the specification of tolerations, which apply to the `ClusterResourcePlacement` object. Each toleration object consists of the following fields:
336+
337+
* `key`: The key of the toleration.
338+
* `value`: The value of the toleration.
339+
* `effect`: The effect of the toleration, such as `NoSchedule`.
340+
* `operator`: The operator of the toleration, such as `Exists` or `Equal`.
341+
342+
Each toleration is used to tolerate one or more specific taints applied on the `ClusterResourcePlacement`. Once all taints on a [`MemberCluster`](./concepts-fleet.md#what-are-member-clusters) are tolerated, the scheduler can then propagate resources to the cluster. You can't update or remove tolerations from a `ClusterResourcePlacement` object once it's created.
343+
344+
For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/ClusterResourcePlacement/README.md#tolerations).
345+
333346
## Access the Kubernetes API of the Fleet resource cluster
334347
335348
If you created an Azure Kubernetes Fleet Manager resource with the hub cluster enabled, you can use it to centrally control scenarios like Kubernetes object propagation. To access the Kubernetes API of the Fleet resource cluster, follow the steps in [Access the Kubernetes API of the Fleet resource cluster with Azure Kubernetes Fleet Manager](./quickstart-access-fleet-kubernetes-api.md).

‎articles/kubernetes-fleet/concepts-scheduler-scheduling-framework.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ The fleet scheduler integrates the following fundamental built-in plugins:
4141
* **Cluster affinity plugin**: Facilitates the affinity clause in the placement policy.
4242
* **Same placement affinity plugin**: Designed specifically for fleet and prevents multiple replicas from being placed within the same cluster.
4343
* **Cluster eligibility plugin**: Enables cluster selection based on specific status criteria.
44+
* **Taint & toleration plugin**: Enables cluster selection based on [taints on the cluster](./concepts-fleet.md#taints) and [tolerations on the `ClusterResourcePlacement`](./concepts-resource-propagation.md#tolerations).
4445

4546
For more information, see the [upstream Fleet Scheduling Framework documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/Scheduling-Framework/README.md).
4647

‎articles/kubernetes-fleet/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@
3434
href: update-orchestration.md
3535
- name: Multi-cluster Layer 4 Load Balancing
3636
href: l4-load-balancing.md
37+
- name: Use taints and tolerations to schedule workloads
38+
href: use-taints-tolerations.md
3739
- name: Resources
3840
items:
3941
- name: Pricing

‎articles/kubernetes-fleet/use-taints-tolerations.md

Lines changed: 303 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,303 @@
1+
---
2+
title: "Use taints on member clusters and tolerations on cluster resource placements in Azure Kubernetes Fleet Manager"
3+
description: Learn how to use taints on `MemberCluster` resources and tolerations on `ClusterResourcePlacement` resources in Azure Kubernetes Fleet Manager.
4+
ms.topic: how-to
5+
ms.date: 04/19/2024
6+
author: schaffererin
7+
ms.author: schaffererin
8+
ms.service: kubernetes-fleet
9+
---
10+
11+
# Use taints on member clusters and tolerations on cluster resource placements
12+
13+
This article explains how to add/remove taints on `MemberCluster` resources and tolerations on `ClusterResourcePlacement` resources in Azure Kubernetes Fleet Manager.
14+
15+
## Prerequisites
16+
17+
* [!INCLUDE [free trial note](../../includes/quickstarts-free-trial-note.md)]
18+
* Read the conceptual overviews for [taints](./concepts-fleet.md#taints) and [tolerations](./concepts-resource-propagation.md#tolerations).
19+
* You must have a Fleet resource with a hub cluster and member clusters. If you don't have this resource, follow [Quickstart: Create a Fleet resource and join member clusters](quickstart-create-fleet-and-members.md).
20+
* You must gain access to the Kubernetes API of the hub cluster by following the steps in [Access the Kubernetes API of the Fleet resource](./quickstart-access-fleet-kubernetes-api.md).
21+
22+
## Add taints to a member cluster
23+
24+
In this example, we add a taint to a `MemberCluster` resource, then try to propagate resources to the member cluster using a `ClusterResourcePlacement` with a `PickAll` placement policy. The resources shouldn't be propagated to the member cluster because of the taint.
25+
26+
1. Create a namespace to propagate to the member cluster using the `kubectl create ns` command.
27+
28+
```bash
29+
kubectl create ns test-ns
30+
```
31+
32+
2. Create a taint on the `MemberCluster` resource using the following example code:
33+
34+
```yml
35+
apiVersion: cluster.kubernetes-fleet.io/v1beta1
36+
kind: MemberCluster
37+
metadata:
38+
name: kind-cluster-1
39+
spec:
40+
identity:
41+
name: fleet-member-agent-cluster-1
42+
kind: ServiceAccount
43+
namespace: fleet-system
44+
apiGroup: ""
45+
taints: # Add taint to the member cluster
46+
- key: test-key1
47+
value: test-value1
48+
effect: NoSchedule
49+
```
50+
51+
3. Apply the taint to the `MemberCluster` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
52+
53+
```bash
54+
kubectl apply -f member-cluster-taint.yml
55+
```
56+
57+
4. Create a `PickAll` placement policy on the `ClusterResourcePlacement` resource using the following example code:
58+
59+
```yml
60+
resourceSelectors:
61+
- group: ""
62+
kind: Namespace
63+
version: v1
64+
name: test-ns
65+
policy:
66+
placementType: PickAll
67+
```
68+
69+
5. Apply the `ClusterResourcePlacement` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
70+
71+
```bash
72+
kubectl apply -f cluster-resource-placement-pick-all.yml
73+
```
74+
75+
6. Verify that the resources weren't propagated to the member cluster by checking the details of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
76+
77+
```bash
78+
kubectl describe clusterresourceplacement test-ns
79+
```
80+
81+
Your output should look similar to the following example output:
82+
83+
```output
84+
status:
85+
conditions:
86+
- lastTransitionTime: "2024-04-16T19:03:17Z"
87+
message: found all the clusters needed as specified by the scheduling policy
88+
observedGeneration: 2
89+
reason: SchedulingPolicyFulfilled
90+
status: "True"
91+
type: ClusterResourcePlacementScheduled
92+
- lastTransitionTime: "2024-04-16T19:03:17Z"
93+
message: All 0 cluster(s) are synchronized to the latest resources on the hub
94+
cluster
95+
observedGeneration: 2
96+
reason: SynchronizeSucceeded
97+
status: "True"
98+
type: ClusterResourcePlacementSynchronized
99+
- lastTransitionTime: "2024-04-16T19:03:17Z"
100+
message: There are no clusters selected to place the resources
101+
observedGeneration: 2
102+
reason: ApplySucceeded
103+
status: "True"
104+
type: ClusterResourcePlacementApplied
105+
observedResourceIndex: "0"
106+
selectedResources:
107+
- kind: Namespace
108+
name: test-ns
109+
version: v1
110+
```
111+
112+
## Remove taints from a member cluster
113+
114+
In this example, we remove the taint we created in [add taints to a member cluster](#add-taints-to-a-member-cluster). This should automatically trigger the Fleet scheduler to propagate the resources to the member cluster.
115+
116+
1. Open your `MemberCluster` YAML file and remove the taint section.
117+
2. Apply the changes to the `MemberCluster` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
118+
119+
```bash
120+
kubectl apply -f member-cluster-taint.yml
121+
```
122+
123+
3. Verify that the resources were propagated to the member cluster by checking the details of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
124+
125+
```bash
126+
kubectl describe clusterresourceplacement test-ns
127+
```
128+
129+
Your output should look similar to the following example output:
130+
131+
```output
132+
status:
133+
conditions:
134+
- lastTransitionTime: "2024-04-16T20:00:03Z"
135+
message: found all the clusters needed as specified by the scheduling policy
136+
observedGeneration: 2
137+
reason: SchedulingPolicyFulfilled
138+
status: "True"
139+
type: ClusterResourcePlacementScheduled
140+
- lastTransitionTime: "2024-04-16T20:02:57Z"
141+
message: All 1 cluster(s) are synchronized to the latest resources on the hub
142+
cluster
143+
observedGeneration: 2
144+
reason: SynchronizeSucceeded
145+
status: "True"
146+
type: ClusterResourcePlacementSynchronized
147+
- lastTransitionTime: "2024-04-16T20:02:57Z"
148+
message: Successfully applied resources to 1 member clusters
149+
observedGeneration: 2
150+
reason: ApplySucceeded
151+
status: "True"
152+
type: ClusterResourcePlacementApplied
153+
observedResourceIndex: "0"
154+
placementStatuses:
155+
- clusterName: kind-cluster-1
156+
conditions:
157+
- lastTransitionTime: "2024-04-16T20:02:52Z"
158+
message: 'Successfully scheduled resources for placement in kind-cluster-1 (affinity
159+
score: 0, topology spread score: 0): picked by scheduling policy'
160+
observedGeneration: 2
161+
reason: ScheduleSucceeded
162+
status: "True"
163+
type: Scheduled
164+
- lastTransitionTime: "2024-04-16T20:02:57Z"
165+
message: Successfully Synchronized work(s) for placement
166+
observedGeneration: 2
167+
reason: WorkSynchronizeSucceeded
168+
status: "True"
169+
type: WorkSynchronized
170+
- lastTransitionTime: "2024-04-16T20:02:57Z"
171+
message: Successfully applied resources
172+
observedGeneration: 2
173+
reason: ApplySucceeded
174+
status: "True"
175+
type: Applied
176+
selectedResources:
177+
- kind: Namespace
178+
name: test-ns
179+
version: v1
180+
```
181+
182+
## Add tolerations to a cluster resource placement
183+
184+
In this example, we add a toleration to a `ClusterResourcePlacement` resource to propagate resources to a member cluster that has a taint. The toleration allows the resources to be propagated to the member cluster.
185+
186+
1. Create a namespace to propagate to the member cluster using the `kubectl create ns` command.
187+
188+
```bash
189+
kubectl create ns test-ns
190+
```
191+
192+
2. Create a taint on the `MemberCluster` resource using the following example code:
193+
194+
```yml
195+
apiVersion: cluster.kubernetes-fleet.io/v1beta1
196+
kind: MemberCluster
197+
metadata:
198+
name: kind-cluster-1
199+
spec:
200+
identity:
201+
name: fleet-member-agent-cluster-1
202+
kind: ServiceAccount
203+
namespace: fleet-system
204+
apiGroup: ""
205+
taints: # Add taint to the member cluster
206+
- key: test-key1
207+
value: test-value1
208+
effect: NoSchedule
209+
```
210+
211+
3. Apply the taint to the `MemberCluster` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
212+
213+
```bash
214+
kubectl apply -f member-cluster-taint.yml
215+
```
216+
217+
4. Create a toleration on the `ClusterResourcePlacement` resource using the following example code:
218+
219+
```yml
220+
spec:
221+
policy:
222+
placementType: PickAll
223+
tolerations:
224+
- key: test-key1
225+
operator: Exists
226+
resourceSelectors:
227+
- group: ""
228+
kind: Namespace
229+
name: test-ns
230+
version: v1
231+
revisionHistoryLimit: 10
232+
strategy:
233+
type: RollingUpdate
234+
```
235+
236+
5. Apply the `ClusterResourcePlacement` resource using the `kubectl apply` command. Make sure you replace the file name with the name of your file.
237+
238+
```bash
239+
kubectl apply -f cluster-resource-placement-toleration.yml
240+
```
241+
242+
6. Verify that the resources were propagated to the member cluster by checking the details of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
243+
244+
```bash
245+
kubectl describe clusterresourceplacement test-ns
246+
```
247+
248+
Your output should look similar to the following example output:
249+
250+
```output
251+
status:
252+
conditions:
253+
- lastTransitionTime: "2024-04-16T20:16:10Z"
254+
message: found all the clusters needed as specified by the scheduling policy
255+
observedGeneration: 3
256+
reason: SchedulingPolicyFulfilled
257+
status: "True"
258+
type: ClusterResourcePlacementScheduled
259+
- lastTransitionTime: "2024-04-16T20:16:15Z"
260+
message: All 1 cluster(s) are synchronized to the latest resources on the hub
261+
cluster
262+
observedGeneration: 3
263+
reason: SynchronizeSucceeded
264+
status: "True"
265+
type: ClusterResourcePlacementSynchronized
266+
- lastTransitionTime: "2024-04-16T20:16:15Z"
267+
message: Successfully applied resources to 1 member clusters
268+
observedGeneration: 3
269+
reason: ApplySucceeded
270+
status: "True"
271+
type: ClusterResourcePlacementApplied
272+
observedResourceIndex: "0"
273+
placementStatuses:
274+
- clusterName: kind-cluster-1
275+
conditions:
276+
- lastTransitionTime: "2024-04-16T20:16:10Z"
277+
message: 'Successfully scheduled resources for placement in kind-cluster-1 (affinity
278+
score: 0, topology spread score: 0): picked by scheduling policy'
279+
observedGeneration: 3
280+
reason: ScheduleSucceeded
281+
status: "True"
282+
type: Scheduled
283+
- lastTransitionTime: "2024-04-16T20:16:15Z"
284+
message: Successfully Synchronized work(s) for placement
285+
observedGeneration: 3
286+
reason: WorkSynchronizeSucceeded
287+
status: "True"
288+
type: WorkSynchronized
289+
- lastTransitionTime: "2024-04-16T20:16:15Z"
290+
message: Successfully applied resources
291+
observedGeneration: 3
292+
reason: ApplySucceeded
293+
status: "True"
294+
type: Applied
295+
selectedResources:
296+
- kind: Namespace
297+
name: test-ns
298+
version: v1
299+
```
300+
301+
## Next steps
302+
303+
For more information on Azure Kubernetes Fleet Manager, see the [upstream Fleet documentation](https://github.com/Azure/fleet/tree/main/docs).

0 commit comments

Comments
 (0)