Skip to content

Commit 0039b0c

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #41389 from tengqm/wrap-leader-mig
Tweak line wrapping for leader migration task page
2 parents 75ece85 + 3b2a703 commit 0039b0c

File tree

1 file changed

+134
-41
lines changed

1 file changed

+134
-41
lines changed

‎content/en/docs/tasks/administer-cluster/controller-manager-leader-migration.md

Lines changed: 134 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
reviewers:
33
- jpbetz
44
- cheftako
5-
title: "Migrate Replicated Control Plane To Use Cloud Controller Manager"
5+
title: Migrate Replicated Control Plane To Use Cloud Controller Manager
66
linkTitle: "Migrate Replicated Control Plane To Use Cloud Controller Manager"
77
content_type: task
88
weight: 250
@@ -14,45 +14,92 @@ weight: 250
1414

1515
## Background
1616

17-
As part of the [cloud provider extraction effort](/blog/2019/04/17/the-future-of-cloud-providers-in-kubernetes/), all cloud specific controllers must be moved out of the `kube-controller-manager`. All existing clusters that run cloud controllers in the `kube-controller-manager` must migrate to instead run the controllers in a cloud provider specific `cloud-controller-manager`.
18-
19-
Leader Migration provides a mechanism in which HA clusters can safely migrate "cloud specific" controllers between the `kube-controller-manager` and the `cloud-controller-manager` via a shared resource lock between the two components while upgrading the replicated control plane. For a single-node control plane, or if unavailability of controller managers can be tolerated during the upgrade, Leader Migration is not needed and this guide can be ignored.
20-
21-
Leader Migration can be enabled by setting `--enable-leader-migration` on `kube-controller-manager` or `cloud-controller-manager`. Leader Migration only applies during the upgrade and can be safely disabled or left enabled after the upgrade is complete.
22-
23-
This guide walks you through the manual process of upgrading the control plane from `kube-controller-manager` with built-in cloud provider to running both `kube-controller-manager` and `cloud-controller-manager`. If you use a tool to deploy and manage the cluster, please refer to the documentation of the tool and the cloud provider for specific instructions of the migration.
17+
As part of the [cloud provider extraction effort](/blog/2019/04/17/the-future-of-cloud-providers-in-kubernetes/),
18+
all cloud specific controllers must be moved out of the `kube-controller-manager`.
19+
All existing clusters that run cloud controllers in the `kube-controller-manager`
20+
must migrate to instead run the controllers in a cloud provider specific
21+
`cloud-controller-manager`.
22+
23+
Leader Migration provides a mechanism in which HA clusters can safely migrate "cloud
24+
specific" controllers between the `kube-controller-manager` and the
25+
`cloud-controller-manager` via a shared resource lock between the two components
26+
while upgrading the replicated control plane. For a single-node control plane, or if
27+
unavailability of controller managers can be tolerated during the upgrade, Leader
28+
Migration is not needed and this guide can be ignored.
29+
30+
Leader Migration can be enabled by setting `--enable-leader-migration` on
31+
`kube-controller-manager` or `cloud-controller-manager`. Leader Migration only
32+
applies during the upgrade and can be safely disabled or left enabled after the
33+
upgrade is complete.
34+
35+
This guide walks you through the manual process of upgrading the control plane from
36+
`kube-controller-manager` with built-in cloud provider to running both
37+
`kube-controller-manager` and `cloud-controller-manager`. If you use a tool to deploy
38+
and manage the cluster, please refer to the documentation of the tool and the cloud
39+
provider for specific instructions of the migration.
2440

2541
## {{% heading "prerequisites" %}}
2642

27-
It is assumed that the control plane is running Kubernetes version N and to be upgraded to version N + 1. Although it is possible to migrate within the same version, ideally the migration should be performed as part of an upgrade so that changes of configuration can be aligned to each release. The exact versions of N and N + 1 depend on each cloud provider. For example, if a cloud provider builds a `cloud-controller-manager` to work with Kubernetes 1.24, then N can be 1.23 and N + 1 can be 1.24.
28-
29-
The control plane nodes should run `kube-controller-manager` with Leader Election enabled, which is the default. As of version N, an in-tree cloud provider must be set with `--cloud-provider` flag and `cloud-controller-manager` should not yet be deployed.
30-
31-
The out-of-tree cloud provider must have built a `cloud-controller-manager` with Leader Migration implementation. If the cloud provider imports `k8s.io/cloud-provider` and `k8s.io/controller-manager` of version v0.21.0 or later, Leader Migration will be available. However, for version before v0.22.0, Leader Migration is alpha and requires feature gate `ControllerManagerLeaderMigration` to be enabled in `cloud-controller-manager`.
32-
33-
This guide assumes that kubelet of each control plane node starts `kube-controller-manager` and `cloud-controller-manager` as static pods defined by their manifests. If the components run in a different setting, please adjust the steps accordingly.
34-
35-
For authorization, this guide assumes that the cluster uses RBAC. If another authorization mode grants permissions to `kube-controller-manager` and `cloud-controller-manager` components, please grant the needed access in a way that matches the mode.
43+
It is assumed that the control plane is running Kubernetes version N and to be
44+
upgraded to version N + 1. Although it is possible to migrate within the same
45+
version, ideally the migration should be performed as part of an upgrade so that
46+
changes of configuration can be aligned to each release. The exact versions of N and
47+
N + 1 depend on each cloud provider. For example, if a cloud provider builds a
48+
`cloud-controller-manager` to work with Kubernetes 1.24, then N can be 1.23 and N + 1
49+
can be 1.24.
50+
51+
The control plane nodes should run `kube-controller-manager` with Leader Election
52+
enabled, which is the default. As of version N, an in-tree cloud provider must be set
53+
with `--cloud-provider` flag and `cloud-controller-manager` should not yet be
54+
deployed.
55+
56+
The out-of-tree cloud provider must have built a `cloud-controller-manager` with
57+
Leader Migration implementation. If the cloud provider imports
58+
`k8s.io/cloud-provider` and `k8s.io/controller-manager` of version v0.21.0 or later,
59+
Leader Migration will be available. However, for version before v0.22.0, Leader
60+
Migration is alpha and requires feature gate `ControllerManagerLeaderMigration` to be
61+
enabled in `cloud-controller-manager`.
62+
63+
This guide assumes that kubelet of each control plane node starts
64+
`kube-controller-manager` and `cloud-controller-manager` as static pods defined by
65+
their manifests. If the components run in a different setting, please adjust the
66+
steps accordingly.
67+
68+
For authorization, this guide assumes that the cluster uses RBAC. If another
69+
authorization mode grants permissions to `kube-controller-manager` and
70+
`cloud-controller-manager` components, please grant the needed access in a way that
71+
matches the mode.
3672

3773
<!-- steps -->
3874

3975
### Grant access to Migration Lease
4076

41-
The default permissions of the controller manager allow only accesses to their main Lease. In order for the migration to work, accesses to another Lease are required.
77+
The default permissions of the controller manager allow only accesses to their main
78+
Lease. In order for the migration to work, accesses to another Lease are required.
4279

43-
You can grant `kube-controller-manager` full access to the leases API by modifying the `system::leader-locking-kube-controller-manager` role. This task guide assumes that the name of the migration lease is `cloud-provider-extraction-migration`.
80+
You can grant `kube-controller-manager` full access to the leases API by modifying
81+
the `system::leader-locking-kube-controller-manager` role. This task guide assumes
82+
that the name of the migration lease is `cloud-provider-extraction-migration`.
4483

45-
`kubectl patch -n kube-system role 'system::leader-locking-kube-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge`
84+
```shell
85+
kubectl patch -n kube-system role 'system::leader-locking-kube-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge`
86+
```
4687

4788
Do the same to the `system::leader-locking-cloud-controller-manager` role.
4889

49-
`kubectl patch -n kube-system role 'system::leader-locking-cloud-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge`
90+
```shell
91+
kubectl patch -n kube-system role 'system::leader-locking-cloud-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge`
92+
```
5093

5194
### Initial Leader Migration configuration
5295

53-
Leader Migration optionally takes a configuration file representing the state of controller-to-manager assignment. At this moment, with in-tree cloud provider, `kube-controller-manager` runs `route`, `service`, and `cloud-node-lifecycle`. The following example configuration shows the assignment.
96+
Leader Migration optionally takes a configuration file representing the state of
97+
controller-to-manager assignment. At this moment, with in-tree cloud provider,
98+
`kube-controller-manager` runs `route`, `service`, and `cloud-node-lifecycle`. The
99+
following example configuration shows the assignment.
54100

55-
Leader Migration can be enabled without a configuration. Please see [Default Configuration](#default-configuration) for details.
101+
Leader Migration can be enabled without a configuration. Please see
102+
[Default Configuration](#default-configuration) for details.
56103

57104
```yaml
58105
kind: LeaderMigrationConfiguration
@@ -67,8 +114,9 @@ controllerLeaders:
67114
component: kube-controller-manager
68115
```
69116
70-
Alternatively, because the controllers can run under either controller managers, setting `component` to `*`
71-
for both sides makes the configuration file consistent between both parties of the migration.
117+
Alternatively, because the controllers can run under either controller managers,
118+
setting `component` to `*` for both sides makes the configuration file consistent
119+
between both parties of the migration.
72120

73121
```yaml
74122
# wildcard version
@@ -84,16 +132,25 @@ controllerLeaders:
84132
component: *
85133
```
86134

87-
On each control plane node, save the content to `/etc/leadermigration.conf`, and update the manifest of `kube-controller-manager` so that the file is mounted inside the container at the same location. Also, update the same manifest to add the following arguments:
135+
On each control plane node, save the content to `/etc/leadermigration.conf`, and
136+
update the manifest of `kube-controller-manager` so that the file is mounted inside
137+
the container at the same location. Also, update the same manifest to add the
138+
following arguments:
88139

89140
- `--enable-leader-migration` to enable Leader Migration on the controller manager
90141
- `--leader-migration-config=/etc/leadermigration.conf` to set configuration file
91142

92-
Restart `kube-controller-manager` on each node. At this moment, `kube-controller-manager` has leader migration enabled and is ready for the migration.
143+
Restart `kube-controller-manager` on each node. At this moment,
144+
`kube-controller-manager` has leader migration enabled and is ready for the
145+
migration.
93146

94147
### Deploy Cloud Controller Manager
95148

96-
In version N + 1, the desired state of controller-to-manager assignment can be represented by a new configuration file, shown as follows. Please note `component` field of each `controllerLeaders` changing from `kube-controller-manager` to `cloud-controller-manager`. Alternatively, use the wildcard version mentioned above, which has the same effect.
149+
In version N + 1, the desired state of controller-to-manager assignment can be
150+
represented by a new configuration file, shown as follows. Please note `component`
151+
field of each `controllerLeaders` changing from `kube-controller-manager` to
152+
`cloud-controller-manager`. Alternatively, use the wildcard version mentioned above,
153+
which has the same effect.
97154

98155
```yaml
99156
kind: LeaderMigrationConfiguration
@@ -108,35 +165,70 @@ controllerLeaders:
108165
component: cloud-controller-manager
109166
```
110167

111-
When creating control plane nodes of version N + 1, the content should be deployed to `/etc/leadermigration.conf`. The manifest of `cloud-controller-manager` should be updated to mount the configuration file in the same manner as `kube-controller-manager` of version N. Similarly, add `--enable-leader-migration` and `--leader-migration-config=/etc/leadermigration.conf` to the arguments of `cloud-controller-manager`.
168+
When creating control plane nodes of version N + 1, the content should be deployed to
169+
`/etc/leadermigration.conf`. The manifest of `cloud-controller-manager` should be
170+
updated to mount the configuration file in the same manner as
171+
`kube-controller-manager` of version N. Similarly, add `--enable-leader-migration`
172+
and `--leader-migration-config=/etc/leadermigration.conf` to the arguments of
173+
`cloud-controller-manager`.
112174

113-
Create a new control plane node of version N + 1 with the updated `cloud-controller-manager` manifest, and with the `--cloud-provider` flag set to `external` for `kube-controller-manager`. `kube-controller-manager` of version N + 1 MUST NOT have Leader Migration enabled because, with an external cloud provider, it does not run the migrated controllers anymore, and thus it is not involved in the migration.
175+
Create a new control plane node of version N + 1 with the updated
176+
`cloud-controller-manager` manifest, and with the `--cloud-provider` flag set to
177+
`external` for `kube-controller-manager`. `kube-controller-manager` of version N + 1
178+
MUST NOT have Leader Migration enabled because, with an external cloud provider, it
179+
does not run the migrated controllers anymore, and thus it is not involved in the
180+
migration.
114181

115-
Please refer to [Cloud Controller Manager Administration](/docs/tasks/administer-cluster/running-cloud-controller/) for more detail on how to deploy `cloud-controller-manager`.
182+
Please refer to [Cloud Controller Manager Administration](/docs/tasks/administer-cluster/running-cloud-controller/)
183+
for more detail on how to deploy `cloud-controller-manager`.
116184

117185
### Upgrade Control Plane
118186

119-
The control plane now contains nodes of both version N and N + 1. The nodes of version N run `kube-controller-manager` only, and these of version N + 1 run both `kube-controller-manager` and `cloud-controller-manager`. The migrated controllers, as specified in the configuration, are running under either `kube-controller-manager` of version N or `cloud-controller-manager` of version N + 1 depending on which controller manager holds the migration lease. No controller will ever be running under both controller managers at any time.
187+
The control plane now contains nodes of both version N and N + 1. The nodes of
188+
version N run `kube-controller-manager` only, and these of version N + 1 run both
189+
`kube-controller-manager` and `cloud-controller-manager`. The migrated controllers,
190+
as specified in the configuration, are running under either `kube-controller-manager`
191+
of version N or `cloud-controller-manager` of version N + 1 depending on which
192+
controller manager holds the migration lease. No controller will ever be running
193+
under both controller managers at any time.
120194

121-
In a rolling manner, create a new control plane node of version N + 1 and bring down one of version N until the control plane contains only nodes of version N + 1.
122-
If a rollback from version N + 1 to N is required, add nodes of version N with Leader Migration enabled for `kube-controller-manager` back to the control plane, replacing one of version N + 1 each time until there are only nodes of version N.
195+
In a rolling manner, create a new control plane node of version N + 1 and bring down
196+
one of version N until the control plane contains only nodes of version N + 1.
197+
If a rollback from version N + 1 to N is required, add nodes of version N with Leader
198+
Migration enabled for `kube-controller-manager` back to the control plane, replacing
199+
one of version N + 1 each time until there are only nodes of version N.
123200

124201
### (Optional) Disable Leader Migration {#disable-leader-migration}
125202

126-
Now that the control plane has been upgraded to run both `kube-controller-manager` and `cloud-controller-manager` of version N + 1, Leader Migration has finished its job and can be safely disabled to save one Lease resource. It is safe to re-enable Leader Migration for the rollback in the future.
203+
Now that the control plane has been upgraded to run both `kube-controller-manager`
204+
and `cloud-controller-manager` of version N + 1, Leader Migration has finished its
205+
job and can be safely disabled to save one Lease resource. It is safe to re-enable
206+
Leader Migration for the rollback in the future.
127207

128-
In a rolling manager, update manifest of `cloud-controller-manager` to unset both `--enable-leader-migration` and `--leader-migration-config=` flag, also remove the mount of `/etc/leadermigration.conf`, and finally remove `/etc/leadermigration.conf`. To re-enable Leader Migration, recreate the configuration file and add its mount and the flags that enable Leader Migration back to `cloud-controller-manager`.
208+
In a rolling manager, update manifest of `cloud-controller-manager` to unset both
209+
`--enable-leader-migration` and `--leader-migration-config=` flag, also remove the
210+
mount of `/etc/leadermigration.conf`, and finally remove `/etc/leadermigration.conf`.
211+
To re-enable Leader Migration, recreate the configuration file and add its mount and
212+
the flags that enable Leader Migration back to `cloud-controller-manager`.
129213

130214
### Default Configuration
131215

132-
Starting Kubernetes 1.22, Leader Migration provides a default configuration suitable for the default controller-to-manager assignment.
133-
The default configuration can be enabled by setting `--enable-leader-migration` but without `--leader-migration-config=`.
216+
Starting Kubernetes 1.22, Leader Migration provides a default configuration suitable
217+
for the default controller-to-manager assignment.
218+
The default configuration can be enabled by setting `--enable-leader-migration` but
219+
without `--leader-migration-config=`.
134220

135-
For `kube-controller-manager` and `cloud-controller-manager`, if there are no flags that enable any in-tree cloud provider or change ownership of controllers, the default configuration can be used to avoid manual creation of the configuration file.
221+
For `kube-controller-manager` and `cloud-controller-manager`, if there are no flags
222+
that enable any in-tree cloud provider or change ownership of controllers, the
223+
default configuration can be used to avoid manual creation of the configuration file.
136224

137225
### Special case: migrating the Node IPAM controller {#node-ipam-controller-migration}
138226

139-
If your cloud provider provides an implementation of Node IPAM controller, you should switch to the implementation in `cloud-controller-manager`. Disable Node IPAM controller in `kube-controller-manager` of version N + 1 by adding `--controllers=*,-nodeipam` to its flags. Then add `nodeipam` to the list of migrated controllers.
227+
If your cloud provider provides an implementation of Node IPAM controller, you should
228+
switch to the implementation in `cloud-controller-manager`. Disable Node IPAM
229+
controller in `kube-controller-manager` of version N + 1 by adding
230+
`--controllers=*,-nodeipam` to its flags. Then add `nodeipam` to the list of migrated
231+
controllers.
140232

141233
```yaml
142234
# wildcard version, with nodeipam
@@ -156,5 +248,6 @@ controllerLeaders:
156248

157249
## {{% heading "whatsnext" %}}
158250

159-
- Read the [Controller Manager Leader Migration](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cloud-provider/2436-controller-manager-leader-migration) enhancement proposal.
251+
- Read the [Controller Manager Leader Migration](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cloud-provider/2436-controller-manager-leader-migration)
252+
enhancement proposal.
160253

0 commit comments

Comments
 (0)