1
1
---
2
2
title : Migrate Kubernetes Objects Using Storage Version Migration
3
-
4
3
reviewers :
5
- - deads2k
6
- - jpbetz
7
- - enj
8
- - nilekhc
9
-
4
+ - deads2k
5
+ - jpbetz
6
+ - enj
7
+ - nilekhc
10
8
content_type : task
11
9
min-kubernetes-server-version : v1.30
12
10
weight : 60
13
11
---
14
12
15
13
<!-- overview -->
14
+
16
15
{{< feature-state feature_gate_name="StorageVersionMigrator" >}}
17
16
18
- Kubernetes relies on API data being actively re-written, to support some
19
- maintenance activities related to at rest storage. Two prominent examples are
20
- the versioned schema of stored resources (that is, the preferred storage schema
17
+ Kubernetes relies on API data being actively re-written, to support some
18
+ maintenance activities related to at rest storage. Two prominent examples are
19
+ the versioned schema of stored resources (that is, the preferred storage schema
21
20
changing from v1 to v2 for a given resource) and encryption at rest
22
21
(that is, rewriting stale data based on a change in how the data should be encrypted).
23
22
@@ -27,12 +26,13 @@ Install [`kubectl`](/docs/tasks/tools/#kubectl).
27
26
28
27
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
29
28
30
-
31
29
<!-- steps -->
32
30
33
31
## Re-encrypt Kubernetes secrets using storage version migration
34
- - To begin with, [ configure KMS provider] ( /docs/tasks/administer-cluster/kms-provider/ )
32
+
33
+ - To begin with, [ configure KMS provider] ( /docs/tasks/administer-cluster/kms-provider/ )
35
34
to encrypt data at rest in etcd using following encryption configuration.
35
+
36
36
``` yaml
37
37
kind : EncryptionConfiguration
38
38
apiVersion : apiserver.config.k8s.io/v1
@@ -45,15 +45,21 @@ Install [`kubectl`](/docs/tasks/tools/#kubectl).
45
45
- name : key1
46
46
secret : c2VjcmV0IGlzIHNlY3VyZQ==
47
47
` ` `
48
+
48
49
Make sure to enable automatic reload of encryption
49
- configuration file by setting ` --encryption-provider-config-automatic-reload` to true.
50
+ configuration file by setting ` --encryption-provider-config-automatic-reload` to true.
51
+
50
52
- Create a Secret using kubectl.
53
+
51
54
` ` ` shell
52
55
kubectl create secret generic my-secret --from-literal=key1=supersecret
53
56
` ` `
57
+
54
58
- [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted)
55
59
the serialized data for that Secret object is prefixed with `k8s:enc:aescbc:v1:key1`.
60
+
56
61
- Update the encryption configuration file as follows to rotate the encryption key.
62
+
57
63
` ` ` yaml
58
64
kind: EncryptionConfiguration
59
65
apiVersion: apiserver.config.k8s.io/v1
@@ -70,9 +76,12 @@ configuration file by setting `--encryption-provider-config-automatic-reload` to
70
76
- name: key1
71
77
secret: c2VjcmV0IGlzIHNlY3VyZQ==
72
78
` ` `
79
+
73
80
- To ensure that previously created secret `my-secert` is re-encrypted
74
- with new key `key2`, you will use _Storage Version Migration_.
81
+ with new key `key2`, you will use _Storage Version Migration_.
82
+
75
83
- Create a StorageVersionMigration manifest named `migrate-secret.yaml` as follows :
84
+
76
85
` ` ` yaml
77
86
kind: StorageVersionMigration
78
87
apiVersion: storagemigration.k8s.io/v1alpha1
@@ -84,27 +93,31 @@ with new key `key2`, you will use _Storage Version Migration_.
84
93
version: v1
85
94
resource: secrets
86
95
` ` `
96
+
87
97
Create the object using _kubectl_ as follows :
98
+
88
99
` ` ` shell
89
100
kubectl apply -f migrate-secret.yaml
90
101
` ` `
102
+
91
103
- Monitor migration of Secrets by checking the `.status` of the StorageVersionMigration.
92
104
A successful migration should have its
93
- ` Succeeded` condition set to true. Get the StorageVersionMigration object
94
- as follows :
105
+ `Succeeded` condition set to true. Get the StorageVersionMigration object as follows :
106
+
95
107
` ` ` shell
96
108
kubectl get storageversionmigration.storagemigration.k8s.io/secrets-migration -o yaml
97
109
` ` `
98
110
99
111
The output is similar to :
112
+
100
113
` ` ` yaml
101
114
kind: StorageVersionMigration
102
115
apiVersion: storagemigration.k8s.io/v1alpha1
103
116
metadata:
104
117
name: secrets-migration
105
118
uid: 628f6922-a9cb-4514-b076-12d3c178967c
106
- resourceVersion: '90'
107
- creationTimestamp: ' 2024-03-12T20:29:45Z'
119
+ resourceVersion: "90"
120
+ creationTimestamp: " 2024-03-12T20:29:45Z"
108
121
spec:
109
122
resource:
110
123
group: ""
@@ -113,19 +126,21 @@ as follows:
113
126
status:
114
127
conditions:
115
128
- type: Running
116
- status: ' False'
117
- lastUpdateTime: ' 2024-03-12T20:29:46Z'
129
+ status: " False"
130
+ lastUpdateTime: " 2024-03-12T20:29:46Z"
118
131
reason: StorageVersionMigrationInProgress
119
132
- type: Succeeded
120
- status: ' True'
121
- lastUpdateTime: ' 2024-03-12T20:29:46Z'
133
+ status: " True"
134
+ lastUpdateTime: " 2024-03-12T20:29:46Z"
122
135
reason: StorageVersionMigrationSucceeded
123
- resourceVersion: '84'
136
+ resourceVersion: "84"
124
137
` ` `
138
+
125
139
- [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted)
126
140
the stored secret is now prefixed with `k8s:enc:aescbc:v1:key2`.
127
141
128
142
# # Update the preferred storage schema of a CRD
143
+
129
144
Consider a scenario where a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}}
130
145
(CRD) is created to serve custom resources (CRs) and is set as the preferred storage schema. When it's time
131
146
to introduce v2 of the CRD, it can be added for serving only with a conversion
@@ -136,6 +151,7 @@ version, it's important to ensure that all existing CRs stored as v1 are migrate
136
151
This migration can be achieved through _Storage Version Migration_ to migrate all CRs from v1 to v2.
137
152
138
153
- Create a manifest for the CRD, named `test-crd.yaml`, as follows :
154
+
139
155
` ` ` yaml
140
156
apiVersion: apiextensions.k8s.io/v1
141
157
kind: CustomResourceDefinition
@@ -163,35 +179,46 @@ This migration can be achieved through _Storage Version Migration_ to migrate al
163
179
strategy: Webhook
164
180
webhook:
165
181
clientConfig:
166
- url: https://127.0.0.1:9443/crdconvert
182
+ url: " https://127.0.0.1:9443/crdconvert"
167
183
caBundle: <CABundle info>
168
184
conversionReviewVersions:
169
185
- v1
170
186
- v2
171
187
` ` `
172
- Create CRD using kubectl
188
+
189
+ Create CRD using kubectl :
190
+
173
191
` ` ` shell
174
192
kubectl apply -f test-crd.yaml
175
193
` ` `
194
+
176
195
- Create a manifest for an example testcrd. Name the manifest `cr1.yaml` and use these contents :
196
+
177
197
` ` ` yaml
178
198
apiVersion: stable.example.com/v1
179
199
kind: SelfieRequest
180
200
metadata:
181
201
name: cr1
182
202
namespace: default
183
203
` ` `
184
- Create CR using kubectl
204
+
205
+ Create CR using kubectl :
206
+
185
207
` ` ` shell
186
208
kubectl apply -f cr1.yaml
187
209
` ` `
210
+
188
211
- Verify that CR is written and stored as v1 by getting the object from etcd.
212
+
189
213
` ` ` shell
190
214
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C
191
215
` ` `
216
+
192
217
where `[...]` contains the additional arguments for connecting to the etcd server.
218
+
193
219
- Update the CRD `test-crd.yaml` to include v2 version for serving and storage
194
- and v1 as serving only, as follows :
220
+ and v1 as serving only, as follows :
221
+
195
222
` ` ` yaml
196
223
apiVersion: apiextensions.k8s.io/v1
197
224
kind: CustomResourceDefinition
@@ -230,34 +257,45 @@ This migration can be achieved through _Storage Version Migration_ to migrate al
230
257
strategy: Webhook
231
258
webhook:
232
259
clientConfig:
233
- url: ' https://127.0.0.1:9443/crdconvert'
260
+ url: " https://127.0.0.1:9443/crdconvert"
234
261
caBundle: <CABundle info>
235
262
conversionReviewVersions:
236
263
- v1
237
264
- v2
238
265
` ` `
239
- Update CRD using kubectl
266
+
267
+ Update CRD using kubectl :
268
+
240
269
` ` ` shell
241
270
kubectl apply -f test-crd.yaml
242
271
` ` `
272
+
243
273
- Create CR resource file with name `cr2.yaml` as follows :
274
+
244
275
` ` ` yaml
245
276
apiVersion: stable.example.com/v2
246
277
kind: SelfieRequest
247
278
metadata:
248
279
name: cr2
249
280
namespace: default
250
281
` ` `
251
- - Create CR using kubectl
282
+
283
+ - Create CR using kubectl :
284
+
252
285
` ` ` shell
253
286
kubectl apply -f cr2.yaml
254
287
` ` `
288
+
255
289
- Verify that CR is written and stored as v2 by getting the object from etcd.
290
+
256
291
` ` ` shell
257
292
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr2 [...] | hexdump -C
258
293
` ` `
294
+
259
295
where `[...]` contains the additional arguments for connecting to the etcd server.
296
+
260
297
- Create a StorageVersionMigration manifest named `migrate-crd.yaml`, with the contents as follows :
298
+
261
299
` ` ` yaml
262
300
kind: StorageVersionMigration
263
301
apiVersion: storagemigration.k8s.io/v1alpha1
@@ -269,26 +307,31 @@ This migration can be achieved through _Storage Version Migration_ to migrate al
269
307
version: v1
270
308
resource: SelfieRequest
271
309
` ` `
310
+
272
311
Create the object using _kubectl_ as follows :
312
+
273
313
` ` ` shell
274
314
kubectl apply -f migrate-crd.yaml
275
315
` ` `
316
+
276
317
- Monitor migration of secrets using status. Successful migration should have
277
318
` Succeeded` condition set to "True" in the status field. Get the migration resource
278
319
as follows :
320
+
279
321
` ` ` shell
280
322
kubectl get storageversionmigration.storagemigration.k8s.io/crdsvm -o yaml
281
323
` ` `
282
-
324
+
283
325
The output is similar to :
326
+
284
327
` ` ` yaml
285
328
kind: StorageVersionMigration
286
329
apiVersion: storagemigration.k8s.io/v1alpha1
287
330
metadata:
288
331
name: crdsvm
289
332
uid: 13062fe4-32d7-47cc-9528-5067fa0c6ac8
290
- resourceVersion: ' 111'
291
- creationTimestamp: ' 2024-03-12T22:40:01Z'
333
+ resourceVersion: " 111"
334
+ creationTimestamp: " 2024-03-12T22:40:01Z"
292
335
spec:
293
336
resource:
294
337
group: stable.example.com
@@ -297,17 +340,20 @@ This migration can be achieved through _Storage Version Migration_ to migrate al
297
340
status:
298
341
conditions:
299
342
- type: Running
300
- status: ' False'
301
- lastUpdateTime: ' 2024-03-12T22:40:03Z'
343
+ status: " False"
344
+ lastUpdateTime: " 2024-03-12T22:40:03Z"
302
345
reason: StorageVersionMigrationInProgress
303
346
- type: Succeeded
304
- status: ' True'
305
- lastUpdateTime: ' 2024-03-12T22:40:03Z'
347
+ status: " True"
348
+ lastUpdateTime: " 2024-03-12T22:40:03Z"
306
349
reason: StorageVersionMigrationSucceeded
307
- resourceVersion: ' 106'
350
+ resourceVersion: " 106"
308
351
` ` `
352
+
309
353
- Verify that previously created cr1 is now written and stored as v2 by getting the object from etcd.
354
+
310
355
` ` ` shell
311
356
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C
312
357
` ` `
358
+
313
359
where `[...]` contains the additional arguments for connecting to the etcd server.
0 commit comments