Skip to content

Commit c399a5d

Browse files
WendyRingWendyRing
authored
Merge pull request #275369 from MicrosoftDocs/release-build-2024-kubernetes-fleet-manager
[BUILD 2024 Ship Room] release-build-2024-kubernetes-fleet-manager
2 parents 5ba3e7b + 02b8a5d commit c399a5d

13 files changed

+1151
-44
lines changed

‎articles/kubernetes-fleet/cluster-resource-override.md

Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
---
2+
title: "Customize cluster scoped resources in Azure Kubernetes Fleet Manager with cluster resource overrides"
3+
description: This article provides an overview of how to use the Fleet ClusterResourceOverride API to override cluster scoped resources in Azure Kubernetes Fleet Manager.
4+
ms.topic: how-to
5+
ms.date: 05/10/2024
6+
author: schaffererin
7+
ms.author: schaffererin
8+
ms.service: kubernetes-fleet
9+
ms.custom:
10+
- build-2024
11+
---
12+
13+
# Customize cluster scoped resources in Azure Kubernetes Fleet Manager with cluster resource overrides (preview)
14+
15+
This article provides an overview of how to use the Fleet `ClusterResourceOverride` API to customize cluster scoped resources in Azure Kubernetes Fleet Manager.
16+
17+
[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
18+
19+
## Cluster resource override overview
20+
21+
The cluster resource override feature allows you to modify or override specific attributes across cluster-wide resources. With `ClusterResourceOverride`, you can define rules based on cluster labels, specifying changes to be applied to various cluster-wide resources such as namespaces, cluster roles, cluster role bindings, or custom resource definitions (CRDs). These modifications might include updates to permissions, configurations, or other parameters, ensuring consistent management and enforcement of configurations across your Fleet-managed Kubernetes clusters.
22+
23+
## API components
24+
25+
The `ClusterResourceOverride` API consists of the following components:
26+
27+
* **`clusterResourceSelectors`**: Specifies the set of cluster resources selected for overriding.
28+
* **`policy`**: Specifies the set of rules to apply to the selected cluster resources.
29+
30+
### Cluster resource selectors
31+
32+
A `ClusterResourceOverride` object can include one or more cluster resource selectors to specify which resources to override. The `ClusterResourceSelector` object supports the following fields:
33+
34+
> [!NOTE]
35+
> If you select a namespace in the `ClusterResourceSelector`, the override will apply to all resources in the namespace.
36+
37+
* `group`: The API group of the resource.
38+
* `version`: The API version of the resource.
39+
* `kind`: The kind of the resource.
40+
* `name`: The name of the resource.
41+
42+
To add a cluster resource selector to a `ClusterResourceOverride` object, use the `clusterResourceSelectors` field with the following YAML format:
43+
44+
```yaml
45+
apiVersion: placement.kubernetes-fleet.io/v1alpha1
46+
kind: ClusterResourceOverride
47+
metadata:
48+
name: example-cro
49+
spec:
50+
clusterResourceSelectors:
51+
- group: rbac.authorization.k8s.io
52+
kind: ClusterRole
53+
version: v1
54+
name: secret-reader
55+
```
56+
57+
This example selects a `ClusterRole` named `secret-reader` from the `rbac.authorization.k8s.io/v1` API group for overriding.
58+
59+
## Policy
60+
61+
A `Policy` object consists of a set of rules, `overrideRules`, that specify the changes to apply to the selected cluster resources. Each `overrideRule` object supports the following fields:
62+
63+
* `clusterSelector`: Specifies the set of clusters to which the override rule applies.
64+
* `jsonPatchOverrides`: Specifies the changes to apply to the selected resources.
65+
66+
To add an override rule to a `ClusterResourceOverride` object, use the `policy` field with the following YAML format:
67+
68+
```yaml
69+
apiVersion: placement.kubernetes-fleet.io/v1alpha1
70+
kind: ClusterResourceOverride
71+
metadata:
72+
name: example-cro
73+
spec:
74+
clusterResourceSelectors:
75+
- group: rbac.authorization.k8s.io
76+
kind: ClusterRole
77+
version: v1
78+
name: secret-reader
79+
policy:
80+
overrideRules:
81+
- clusterSelector:
82+
clusterSelectorTerms:
83+
- labelSelector:
84+
matchLabels:
85+
env: prod
86+
jsonPatchOverrides:
87+
- op: remove
88+
path: /rules/0/verbs/2
89+
```
90+
91+
This example removes the verb "list" in the `ClusterRole` named `secret-reader` on clusters with the label `env: prod`.
92+
93+
### Cluster selector
94+
95+
You can use the `clusterSelector` field in the `overrideRule` object to specify the clusters to which the override rule applies. The `ClusterSelector` object supports the following field:
96+
97+
* `clusterSelectorTerms`: A list of terms that specify the criteria for selecting clusters. Each term includes a `labelSelector` field that defines a set of labels to match.
98+
99+
> [!IMPORTANT]
100+
> Only `labelSelector` is supported in the `clusterSelectorTerms` field.
101+
102+
### JSON patch overrides
103+
104+
You can use `jsonPatchOverrides` in the `overrideRule` object to specify the changes to apply to the selected resources. The `JsonPatch` object supports the following fields:
105+
106+
* `op`: The operation to perform.
107+
* Supported operations include `add`, `remove`, and `replace`.
108+
* `add`: Adds a new value to the specified path.
109+
* `remove`: Removes the value at the specified path.
110+
* `replace`: Replaces the value at the specified path.
111+
* `path`: The path to the field to modify.
112+
* Guidance on specifying paths includes:
113+
* Must start with a `/` character.
114+
* Can't be empty or contain an empty string.
115+
* Can't be a `TypeMeta` field ("/kind" or "/apiVersion").
116+
* Can't be a `Metadata` field ("/metadata/name" or "/metadata/namespace") except the fields "/metadata/labels" and "/metadata/annotations".
117+
* Can't be any field in the status of the resource.
118+
* Examples of valid paths include:
119+
* `/metadata/labels/new-label`
120+
* `/metadata/annotations/new-annotation`
121+
* `/spec/template/spec/containers/0/resources/limits/cpu`
122+
* `/spec/template/spec/containers/0/resources/requests/memory`
123+
* `value`: The value to add, remove, or replace.
124+
* If the `op` is `remove`, you can't specify a `value`.
125+
126+
### Use multiple override patches
127+
128+
You can add multiple `jsonPatchOverrides` to an `overrideRule` to apply multiple changes to the select cluster resources, as shown in the following example:
129+
130+
```yaml
131+
apiVersion: placement.kubernetes-fleet.io/v1alpha1
132+
kind: ClusterResourceOverride
133+
metadata:
134+
name: cro-1
135+
spec:
136+
clusterResourceSelectors:
137+
- group: rbac.authorization.k8s.io
138+
kind: ClusterRole
139+
version: v1
140+
name: secret-reader
141+
policy:
142+
overrideRules:
143+
- clusterSelector:
144+
clusterSelectorTerms:
145+
- labelSelector:
146+
matchLabels:
147+
env: prod
148+
jsonPatchOverrides:
149+
- op: remove
150+
path: /rules/0/verbs/2
151+
- op: remove
152+
path: /rules/0/verbs/1
153+
```
154+
155+
This example removes the verbs "list" and "watch" in the `ClusterRole` named `secret-reader` on clusters with the label `env: prod`.
156+
157+
`jsonPatchOverrides` apply a JSON patch on the selected resources following [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902).
158+
159+
## Apply the cluster resource override
160+
161+
1. Create a `ClusterResourcePlacement` resource to specify the placement rules for distributing the cluster resource overrides across the cluster infrastructure, as shown in the following example. Make sure you select the appropriate resource.
162+
163+
```yaml
164+
apiVersion: placement.kubernetes-fleet.io/v1beta1
165+
kind: ClusterResourcePlacement
166+
metadata:
167+
name: crp
168+
spec:
169+
resourceSelectors:
170+
- group: rbac.authorization.k8s.io
171+
kind: ClusterRole
172+
version: v1
173+
name: secret-reader
174+
policy:
175+
placementType: PickAll
176+
affinity:
177+
clusterAffinity:
178+
requiredDuringSchedulingIgnoredDuringExecution:
179+
clusterSelectorTerms:
180+
- labelSelector:
181+
matchLabels:
182+
env: prod
183+
```
184+
185+
This example distributes resources across all clusters labeled with `env: prod`. As the changes are implemented, the corresponding `ClusterResourceOverride` configurations will be applied to the designated clusters, triggered by the selection of matching cluster role resource, `secret-reader`.
186+
187+
2. Apply the `ClusterResourcePlacement` using the `kubectl apply` command.
188+
189+
```bash
190+
kubectl apply -f cluster-resource-placement.yaml
191+
```
192+
193+
3. Verify the `ClusterResourceOverride` object applied to the selected resources by checking the status of the `ClusterResourcePlacement` resource using the `kubectl describe` command.
194+
195+
```bash
196+
kubectl describe clusterresourceplacement crp
197+
```
198+
199+
Your output should resemble the following example output:
200+
201+
```output
202+
Status:
203+
Conditions:
204+
...
205+
Last Transition Time: 2024-04-27T04:18:00Z
206+
Message: The selected resources are successfully overridden in the 10 clusters
207+
Observed Generation: 1
208+
Reason: OverriddenSucceeded
209+
Status: True
210+
Type: ClusterResourcePlacementOverridden
211+
...
212+
Observed Resource Index: 0
213+
Placement Statuses:
214+
Applicable Cluster Resource Overrides:
215+
example-cro-0
216+
Cluster Name: member-50
217+
Conditions:
218+
...
219+
Message: Successfully applied the override rules on the resources
220+
Observed Generation: 1
221+
Reason: OverriddenSucceeded
222+
Status: True
223+
Type: Overridden
224+
...
225+
```
226+
227+
The `ClusterResourcePlacementOverridden` condition indicates whether the resource override was successfully applied to the selected resources in the clusters. Each cluster maintains its own `Applicable Cluster Resource Overrides` list, which contains the cluster resource override snapshot if relevant. Individual status messages for each cluster indicate whether the override rules were successfully applied.
228+
229+
## Next steps
230+
231+
To learn more about Fleet, see the following resources:
232+
233+
* [Upstream Fleet documentation](https://github.com/Azure/fleet/tree/main/docs)
234+
* [Azure Kubernetes Fleet Manager overview](./overview.md)

‎articles/kubernetes-fleet/concepts-choosing-fleet.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
title: "Choose an Azure Kubernetes Fleet Manager option"
3+
description: This article provides a conceptual overview of the various Azure Kubernetes Fleet Manager options and why you may choose a specific configuration.
4+
ms.date: 05/01/2024
5+
author: shashankbarsin
6+
ms.author: shasb
7+
ms.service: kubernetes-fleet
8+
ms.topic: conceptual
9+
---
10+
11+
# Choosing an Azure Kubernetes Fleet Manager option
12+
13+
This article provides an overview of the various Azure Kubernetes Fleet Manager (Fleet) options and the considerations you should use to guide your selection of a specific configuration.
14+
15+
## Fleet types
16+
17+
A Kubernetes Fleet resource can be created with or without a hub cluster. A hub cluster is a managed Azure Kubernetes Service (AKS) cluster that acts as a hub to store and propagate Kubernetes resources.
18+
19+
The following table compares the scenarios enabled by the hub cluster:
20+
21+
| Capability | Kubernetes Fleet resource without hub cluster | Kubernetes Fleet resource with hub cluster |
22+
|----|----|----|
23+
|**Hub cluster hosting**|<span class='red-x'>&#10060;</span>|<span class='green-check'>&#9989;</span>|
24+
|**Member cluster limit**|Up to 100 clusters|Up to 20 clusters|
25+
|**Update orchestration**|<span class='green-check'>&#9989;</span>|<span class='green-check'>&#9989;</span>|
26+
|**Workload orchestration**|<span class='red-x'>&#10060;</span>|<span class='green-check'>&#9989;</span>|
27+
|**Layer 4 load balancing**|<span class='red-x'>&#10060;</span>|<span class='green-check'>&#9989;</span>|
28+
|**Billing considerations**|No cost|You pay cost associated with the hub, which is a standard-tier AKS-cluster.|
29+
|**Converting fleet types**|Can be upgraded to a Kubernetes Fleet resource with a hub cluster.|Can't be downgraded to a Kubernetes Fleet resource without a hub cluster.|
30+
31+
## Kubernetes Fleet resource without hub clusters
32+
33+
Without a hub cluster, Kubernetes Fleet acts solely as a grouping entity in Azure Resource Manager (ARM). Certain scenarios, such as update runs, don't require a Kubernetes API and thus don't require a hub cluster. To take full advantage of all the features available, you need a Kubernetes Fleet resource with a hub cluster.
34+
35+
For more information, see [Create a Kubernetes Fleet resource without a hub cluster][create-fleet-without-hub].
36+
37+
## Kubernetes Fleet resource with hub clusters
38+
39+
A Kubernetes Fleet resource with a hub cluster has an associated AKS-managed cluster, which is used to store the configuration for workload orchestration and layer-4 load balancing.
40+
41+
Upon the creation of a Kubernetes Fleet resource with a hub cluster, a hub AKS cluster is automatically created in the same subscription under a managed resource group that begins with `FL_`. To improve reliability, hub clusters are locked down by denying any user-initiated mutations to the corresponding AKS clusters (under the Fleet-managed resource group `FL_`) and their underlying Azure resources (under the AKS-managed resource group `MC_FL_*`), such as virtual machines (VMs), via Azure deny assignments. Control plane operations, such as changing the hub cluster's configuration through Azure Resource Manager (ARM) or deleting the cluster entirely, are denied. Data plane operations, such as connecting to the hub cluster's Kubernetes API server in order to configure workload orchestration, are not denied.
42+
43+
Hub clusters are exempted from [Azure policies][azure-policy-overview] to avoid undesirable policy effects upon hub clusters.
44+
45+
### Network access modes for hub cluster
46+
47+
For a Kubernetes Fleet resource with a hub cluster, there are two network access modes:
48+
49+
- **Public hub clusters** expose the hub cluster to the internet. This means that with the right credentials, anyone on the internet can connect to the hub cluster. This configuration can be useful during the development and testing phase, but represents a security concern, which is largely undesirable in production.
50+
51+
For more information, see [Create a Kubernetes Fleet resource with a public hub cluster][create-public-hub-cluster].
52+
53+
- **Private hub clusters** use a [private AKS cluster][aks-private-cluster] as the hub, which prevents open access over the internet. All considerations for a private AKS cluster apply, so review the prerequisites and limitations to determine whether a Kubernetes Fleet resource with a private hub cluster meets your needs.
54+
55+
Some other details to consider:
56+
57+
- Whether you choose a public or private hub, the type can't be changed after creation.
58+
- When using an AKS private cluster, you have the ability to configure fully qualified domain names (FQDNs) and FQDN subdomains. This functionality doesn't apply to the private hub cluster of the Kubernetes Fleet resource.
59+
- When you connect to a private hub cluster, you can use the same methods that you would use to [connect to any private AKS cluster][aks-private-cluster-connect]. However, connecting using AKS command invoke and private endpoints aren't currently supported.
60+
- When you use private hub clusters, you're required to specify the subnet in which the Kubernetes Fleet hub cluster's node VMs reside. This process differs slightly from the AKS private cluster equivalent. For more information, see [create a Kubernetes Fleet resource with a private hub cluster][create-private-hub-cluster].
61+
62+
63+
## Next steps
64+
65+
Now that you understand the different types of Kubernetes fleet resources, see [Create an Azure Kubernetes Fleet Manager resource and join member clusters][quickstart-create-fleet].
66+
67+
<!-- LINKS -->
68+
[aks-private-cluster]: /azure/aks/private-clusters
69+
[aks-private-cluster-connect]: /azure/aks/private-clusters?tabs=azure-portal#options-for-connecting-to-the-private-cluster
70+
[azure-policy-overview]: /azure/governance/policy/overview
71+
[quickstart-create-fleet]: quickstart-create-fleet-and-members.md
72+
[create-fleet-without-hub]: quickstart-create-fleet-and-members.md?tabs=without-hub-cluster#create-a-fleet-resource
73+
[create-public-hub-cluster]: quickstart-create-fleet-and-members.md?tabs=with-hub-cluster#public-hub-cluster
74+
[create-private-hub-cluster]: quickstart-create-fleet-and-members.md?tabs=with-hub-cluster#private-hub-cluster

‎articles/kubernetes-fleet/concepts-fleet.md

Lines changed: 1 addition & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ This article provides a conceptual overview of fleets, member clusters, and hub
1414

1515
## What are fleets?
1616

17-
A fleet resource acts as a grouping entity for multiple AKS clusters. You can use them to manage multiple AKS clusters as a single entity, orchestrate updates across multiple clusters, propagate Kubernetes resources across multiple clusters, and provide a single pane of glass for managing multiple clusters. You can create a fleet with or without a [hub cluster](#what-is-a-hub-cluster-preview).
17+
A fleet resource acts as a grouping entity for multiple AKS clusters. You can use them to manage multiple AKS clusters as a single entity, orchestrate updates across multiple clusters, propagate Kubernetes resources across multiple clusters, and provide a single pane of glass for managing multiple clusters. You can create a fleet with or without a [hub cluster](concepts-choosing-fleet.md).
1818

1919
A fleet consists of the following components:
2020

@@ -41,40 +41,6 @@ Once a `MemberCluster` is tainted, it lets the [scheduler](./concepts-scheduler-
4141

4242
For more information, see [the upstream Fleet documentation](https://github.com/Azure/fleet/blob/main/docs/concepts/MemberCluster/README.md).
4343

44-
## What is a hub cluster (preview)?
45-
46-
[!INCLUDE [preview features note](./includes/preview/preview-callout.md)]
47-
48-
Certain scenarios of fleet such as update runs don't require a Kubernetes API and thus don't require a hub cluster. Fleet can be created without the hub cluster for such scenarios. In this mode, Fleet just acts as a grouping entity in Azure Resource Manager.
49-
50-
For other scenarios such as Kubernetes resource propagation, a hub cluster is required. This hub cluster is a special AKS cluster whose lifecycle (creation, upgrades, deletion) is managed by the fleet resource. Any Kubernetes objects provided to the hub cluster are only stored as configurations on this cluster. Pod creation is disabled on this locked down hub cluster. Thus Fleet doesn't allow running any user workloads on the hub cluster and instead only allows using hub cluster for storing configurations that need to be propagated to other clusters or configurations that control cross-cluster orchestration.
51-
52-
The following table lists the differences between a fleet without hub cluster and a fleet with hub cluster:
53-
54-
| Feature dimension | Without hub cluster | With hub cluster (preview) |
55-
|-|-|-|
56-
| Hub cluster hosting (preview) | :x: | :white_check_mark: |
57-
| Member cluster limit | Up to 100 clusters | Up to 20 clusters |
58-
| Update orchestration across multiple clusters | :white_check_mark: | :white_check_mark: |
59-
| Kubernetes resource object propagation (preview) | :x: | :white_check_mark: |
60-
| Multi-cluster L4 load balancing (preview) | :x: | :white_check_mark: |
61-
62-
Upon the creation of a fleet, a hub cluster is automatically created in the same subscription as the fleet under a managed resource group named as `FL_*`.
63-
64-
To improve reliability, hub clusters are locked down by denying any user initiated mutations to the corresponding AKS clusters (under the Fleet-managed resource group `FL_*`) and their underlying Azure resources like VMs (under the AKS-managed resource group `MC_FL_*`) via Azure deny assignments.
65-
66-
Hub clusters are exempted from Azure policies to avoid undesirable policy effects upon hub clusters.
67-
68-
## Billing
69-
70-
The fleet resource without hub cluster is currently free of charge. If your fleet contains a hub cluster, the hub cluster is a standard tier AKS cluster created in the fleet subscription and paid by you.
71-
72-
## FAQs
73-
74-
### Can I change a fleet without hub cluster to a fleet with hub cluster?
75-
76-
Not during hub cluster preview. This is planned to be supported once hub clusters become generally available.
77-
7844
## Next steps
7945

8046
* [Create a fleet and join member clusters](./quickstart-create-fleet-and-members.md).

0 commit comments

Comments
 (0)