You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/upgrades/automated.md
+71-52Lines changed: 71 additions & 52 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,35 +2,28 @@
2
2
title: Automated Upgrades
3
3
---
4
4
5
-
###Overview
5
+
## Overview
6
6
7
-
You can manage K3s cluster upgrades using Rancher's system-upgrade-controller. This is a Kubernetes-native approach to cluster upgrades. It leverages a [custom resource definition (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-resources), a `plan`, and a [controller](https://kubernetes.io/docs/concepts/architecture/controller/).
7
+
You can manage K3s cluster upgrades using Rancher's system-upgrade-controller. This is a Kubernetes-native approach to cluster upgrades. It leverages a [`Plan`](https://github.com/rancher/system-upgrade-controller/blob/master/doc/plan.md#planspec) Custom Resource to declaratively describe what nodes to upgrade, and to what version.
8
8
9
-
The plan defines upgrade policies and requirements. It also defines which nodes should be upgraded through a [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). See below for plans with defaults appropriate for upgrading a K3s cluster. For more advanced plan configuration options, please review the [CRD](https://github.com/rancher/system-upgrade-controller/blob/master/pkg/apis/upgrade.cattle.io/v1/types.go).
9
+
The plan defines upgrade policies and requirements. It also defines which nodes should be upgraded through a [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). See below for plans with defaults appropriate for upgrading a K3s cluster. For more advanced plan configuration options, see the Plan documentation linked above.
10
10
11
-
The controller schedules upgrades by monitoring plans and selecting nodes to run upgrade [jobs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) on. When a job has run to completion successfully, the controller will label the node on which it ran accordingly.
12
-
13
-
:::note
14
-
The upgrade job that is launched must be highly privileged. It is configured with the following:
15
-
- Host `IPC`, `NET`, and `PID` namespaces
16
-
- The `CAP_SYS_BOOT` capability
17
-
- Host root mounted at `/host` with read and write permissions
11
+
The System Upgrade controller schedules upgrades by monitoring plans and selecting nodes to run upgrade [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) on. When a Job has run to completion successfully, the controller will label the node on which it ran accordingly.
18
12
13
+
:::warning
14
+
If the K3s cluster is managed by Rancher, you should use the Rancher UI to manage upgrades.
15
+
- If the K3s cluster was imported (registered) into Rancher, Rancher will by default manage the system-upgrade-controller deployment and plans. Do not follow the steps on this page unless you have disabled version management in Rancher.
16
+
See [Configuring Version Management for RKE2 and K3s Clusters](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters#configuring-version-management-for-rke2-and-k3s-clusters) for more information.
17
+
- If the K3s cluster was provisioned by Rancher, Rancher will use system agent to manage version upgrades. Do not follow the steps on this page.
18
+
- If the K3s cluster is *not* managed by Rancher, you may follow the steps below.
19
19
:::
20
20
21
+
## Using the System Upgrade Controller
21
22
22
-
To automate upgrades in this manner, you must do the following:
23
+
To automate upgrades , you must do the following:
23
24
24
25
1. Install the system-upgrade-controller into your cluster
25
-
1. Configure plans
26
-
27
-
:::warning
28
-
If the K3s cluster is managed by Rancher, you should use the Rancher UI to manage upgrades.
29
-
- If the K3s cluster was imported into Rancher, Rancher will manage the system-upgrade-controller deployment and plans. Do not follow the steps on this page.
30
-
- If the K3s cluster was provisioned by Rancher, Rancher will use system agent to manage version upgrades. Do not follow the steps on this page.
31
-
- If the K3s cluster is *not* managed Rancher, you may follow the steps below.
32
-
33
-
:::
26
+
1. Create plans describing which groups of nodes to upgrade, and how.
34
27
35
28
For more details on the design and architecture of the system-upgrade-controller or its integration with K3s, see the following Git repositories:
36
29
@@ -41,23 +34,20 @@ For more details on the design and architecture of the system-upgrade-controller
41
34
When attempting to upgrade to a new version of K3s, the [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy/) applies. Ensure that your plan does not skip intermediate minor versions when upgrading. The system-upgrade-controller itself will not protect against unsupported changes to the Kubernetes version.
42
35
:::
43
36
44
-
### Install the system-upgrade-controller
45
-
46
-
The system-upgrade-controller can be installed as a deployment into your cluster. The deployment requires a service-account, clusterRoleBinding, and a configmap. To install these components, run the following command:
The controller can be configured and customized via the previously mentioned configmap, but the controller must be redeployed for the changes to be applied.
37
+
### Installation
52
38
53
-
To be able to apply plans, the system-upgrade-controller CRD has to be deployed:
39
+
The system-upgrade-controller manifest installs a custom resource definition, deployment, service account, cluster role binding, and configmap. To install these components, run the following command:
The controller can be configured and customized via the previously mentioned configmap, but the controller pod must be deleted for the changes to be applied.
58
45
59
-
### Configure plans
60
-
It is recommended you create at least two plans: a plan for upgrading server (control-plane) nodes and a plan for upgrading agent nodes. You can create additional plans as needed to control the rollout of the upgrade across nodes. Once the plans are created, the controller will pick them up and begin to upgrade your cluster.
46
+
### Configuration
47
+
Server nodes should always be upgraded before agent nodes.
48
+
For this reason, it is recommended you create at least two plans: a plan for upgrading server (control-plane) nodes, and a plan for upgrading agent nodes.
49
+
You can create additional plans as needed to control the rollout of the upgrade across nodes.
50
+
Once the plans are created, the controller will pick them up and begin to upgrade your cluster.
61
51
62
52
The following two example plans will upgrade your cluster to K3s v1.24.6+k3s1:
63
53
@@ -108,33 +98,54 @@ spec:
108
98
109
99
There are a few important things to call out regarding these plans:
110
100
111
-
1) The plans must be created in the same namespace where the controller was deployed.
101
+
1. The plans must be created in the same namespace where the controller was deployed.
102
+
2. The `concurrency` field indicates how many nodes can be upgraded at the same time.
103
+
3. The server-plan targets server nodes by specifying a label selector that selects nodes with the `node-role.kubernetes.io/control-plane` label. The agent-plan targets agent nodes by specifying a label selector that select nodes without that label.
104
+
4. The `prepare` step in the agent-plan will cause upgrade jobs for that plan to wait for the server-plan to complete before they execute. This logic is built into the image run for the prepare step, and is not part of system-upgrade-controller itself.
105
+
5. Both plans have the `version` field set to v1.24.6+k3s1. Alternatively, you can omit the `version` field and set the `channel` field to a URL that resolves to a release of K3s. This will cause the controller to monitor that URL and upgrade the cluster any time it resolves to a new release. This works well with the [release channels](manual.md#release-channels). Thus, you can configure your plans with the following channel to ensure your cluster is always automatically upgraded to the newest stable release of K3s:
The upgrade will begin as soon as the controller detects the target version for a plan has been resolved, either from the version field, or by polling the channel server.
116
+
Modifying a plan will cause the controller to re-evaluate the plan and determine if another upgrade is needed.
117
+
If a channel has been configured, the URL is also polled periodically to check for new versions.
112
118
113
-
2) The `concurrency` field indicates how many nodes can be upgraded at the same time.
114
-
115
-
3) The server-plan targets server nodes by specifying a label selector that selects nodes with the `node-role.kubernetes.io/control-plane` label. The agent-plan targets agent nodes by specifying a label selector that select nodes without that label.
119
+
You can monitor the progress of an upgrade by viewing the plan and jobs via kubectl:
120
+
```bash
121
+
kubectl -n system-upgrade get plans -o wide
122
+
kubectl -n system-upgrade get jobs
123
+
```
116
124
117
-
4) The `prepare` step in the agent-plan will cause upgrade jobs for that plan to wait for the server-plan to complete before they execute.
125
+
### Scheduling Upgrades
118
126
119
-
5) Both plans have the `version` field set to v1.24.6+k3s1. Alternatively, you can omit the `version` field and set the `channel` field to a URL that resolves to a release of K3s. This will cause the controller to monitor that URL and upgrade the cluster any time it resolves to a new release. This works well with the [release channels](manual.md#release-channels). Thus, you can configure your plans with the following channel to ensure your cluster is always automatically upgraded to the newest stable release of K3s:
127
+
Plans can be restricted to occurring within a specific time window by setting the `window` field within the plan spec.
128
+
The time window fields are compatible with and take the same format as [kured schedule options](https://kured.dev/docs/configuration/#setting-a-schedule).
As stated, the upgrade will begin as soon as the controller detects that a plan was created. Updating a plan will cause the controller to re-evaluate the plan and determine if another upgrade is needed.
131
-
132
-
You can monitor the progress of an upgrade by viewing the plan and jobs via kubectl:
133
-
```bash
134
-
kubectl -n system-upgrade get plans -o yaml
135
-
kubectl -n system-upgrade get jobs -o yaml
135
+
# ...
136
+
window:
137
+
days:
138
+
- monday
139
+
- tuesday
140
+
- wednesday
141
+
- thursday
142
+
- friday
143
+
startTime: 19:00
144
+
endTime: 21:00
145
+
timeZone: UTC
136
146
```
137
147
148
+
Jobs to execute upgrades for a plan will not be created outside the time window. Once jobs are created, may continue running once the window has closed.
138
149
139
150
## Downgrade Prevention
140
151
@@ -147,20 +158,28 @@ Kubernetes does not support downgrades of control-plane components. The k3s-upgr
147
158
Here is an example cluster, showing failed upgrade pods and cordoned nodes:
You can return your cordoned nodes to service by either of the following methods:
164
176
* Change the version or channel on your plan to target a release that is the same or newer than what is currently running on the cluster, so that the plan succeeds.
165
177
* Delete the plan and manually uncordon the nodes.
166
178
Use `kubectl get plan -n system-upgrade` to find the plan name, then `kubectl delete plan -n system-upgrade PLAN_NAME` to delete it. Once the plan has been deleted, use `kubectl uncordon NODE_NAME` to uncordon each of the nodes.
179
+
180
+
## Security
181
+
The upgrade job that is launched must be highly privileged in order to effect change to the underlying nodes. By default, it is configured with the following:
182
+
- Host `IPC`, `NET`, and `PID` namespaces
183
+
- The `CAP_SYS_BOOT` capability
184
+
- Host root mounted at `/host` with read and write permissions
0 commit comments