Skip to content

Commit 69a3809

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #45714 from drewhagen/kep_4192_svm_v1alpha1_docs_v1.30_dh
docs: adds documentation for Storage Version Migration
2 parents aebbd2f + 048f53d commit 69a3809

File tree

2 files changed

+327
-0
lines changed

2 files changed

+327
-0
lines changed

‎content/en/docs/reference/command-line-tools-reference/feature-gates/storage-version-migration.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
title: StorageVersionMigration
3+
content_type: feature_gate
4+
_build:
5+
list: never
6+
render: false
7+
8+
stages:
9+
- stage: alpha
10+
defaultValue: false
11+
fromVersion: "1.30"
12+
toVersion: "1.32"
13+
---
14+
Enables storage version migration. See [Migrate Kubernetes Objects Using Storage Version Migration](/docs/tasks/manage-kubernetes-objects/storage-version-migration) for more details.

‎content/en/docs/tasks/manage-kubernetes-objects/storage-version-migration.md

Lines changed: 313 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
1+
---
2+
title: Migrate Kubernetes Objects Using Storage Version Migration
3+
4+
reviewers:
5+
- deads2k
6+
- jpbetz
7+
- enj
8+
- nilekhc
9+
10+
content_type: task
11+
min-kubernetes-server-version: v1.30
12+
weight: 60
13+
---
14+
15+
<!-- overview -->
16+
{{< feature-state feature_gate_name="StorageVersionMigration" >}}
17+
18+
Kubernetes relies on API data being actively re-written, to support some
19+
maintenance activities related to at rest storage. Two prominent examples are
20+
the versioned schema of stored resources (that is, the preferred storage schema
21+
changing from v1 to v2 for a given resource) and encryption at rest
22+
(that is, rewriting stale data based on a change in how the data should be encrypted).
23+
24+
## {{% heading "prerequisites" %}}
25+
26+
Install [`kubectl`](/docs/tasks/tools/#kubectl).
27+
28+
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
29+
30+
31+
<!-- steps -->
32+
33+
## Re-encrypt Kubernetes secrets using storage version migration
34+
- To begin with, [configure KMS provider](/docs/tasks/administer-cluster/kms-provider/)
35+
to encrypt data at rest in etcd using following encryption configuration.
36+
```yaml
37+
kind: EncryptionConfiguration
38+
apiVersion: apiserver.config.k8s.io/v1
39+
resources:
40+
- resources:
41+
- secrets
42+
providers:
43+
- aescbc:
44+
keys:
45+
- name: key1
46+
secret: c2VjcmV0IGlzIHNlY3VyZQ==
47+
```
48+
Make sure to enable automatic reload of encryption
49+
configuration file by setting `--encryption-provider-config-automatic-reload` to true.
50+
- Create a Secret using kubectl.
51+
```shell
52+
kubectl create secret generic my-secret --from-literal=key1=supersecret
53+
```
54+
- [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted)
55+
the serialized data for that Secret object is prefixed with `k8s:enc:aescbc:v1:key1`.
56+
- Update the encryption configuration file as follows to rotate the encryption key.
57+
```yaml
58+
kind: EncryptionConfiguration
59+
apiVersion: apiserver.config.k8s.io/v1
60+
resources:
61+
- resources:
62+
- secrets
63+
providers:
64+
- aescbc:
65+
keys:
66+
- name: key2
67+
secret: c2VjcmV0IGlzIHNlY3VyZSwgaXMgaXQ/
68+
- aescbc:
69+
keys:
70+
- name: key1
71+
secret: c2VjcmV0IGlzIHNlY3VyZQ==
72+
```
73+
- To ensure that previously created secret `my-secert` is re-encrypted
74+
with new key `key2`, you will use _Storage Version Migration_.
75+
- Create a StorageVersionMigration manifest named `migrate-secret.yaml` as follows:
76+
```yaml
77+
kind: StorageVersionMigration
78+
apiVersion: storagemigration.k8s.io/v1alpha1
79+
metadata:
80+
name: secrets-migration
81+
spec:
82+
resource:
83+
group: ""
84+
version: v1
85+
resource: secrets
86+
```
87+
Create the object using _kubectl_ as follows:
88+
```shell
89+
kubectl apply -f migrate-secret.yaml
90+
```
91+
- Monitor migration of Secrets by checking the `.status` of the StorageVersionMigration.
92+
A successful migration should have its
93+
`Succeeded` condition set to true. Get the StorageVersionMigration object
94+
as follows:
95+
```shell
96+
kubectl get storageversionmigration.storagemigration.k8s.io/secrets-migration -o yaml
97+
```
98+
99+
The output is similar to:
100+
```yaml
101+
kind: StorageVersionMigration
102+
apiVersion: storagemigration.k8s.io/v1alpha1
103+
metadata:
104+
name: secrets-migration
105+
uid: 628f6922-a9cb-4514-b076-12d3c178967c
106+
resourceVersion: '90'
107+
creationTimestamp: '2024-03-12T20:29:45Z'
108+
spec:
109+
resource:
110+
group: ""
111+
version: v1
112+
resource: secrets
113+
status:
114+
conditions:
115+
- type: Running
116+
status: 'False'
117+
lastUpdateTime: '2024-03-12T20:29:46Z'
118+
reason: StorageVersionMigrationInProgress
119+
- type: Succeeded
120+
status: 'True'
121+
lastUpdateTime: '2024-03-12T20:29:46Z'
122+
reason: StorageVersionMigrationSucceeded
123+
resourceVersion: '84'
124+
```
125+
- [Verify](/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted)
126+
the stored secret is now prefixed with `k8s:enc:aescbc:v1:key2`.
127+
128+
## Update the preferred storage schema of the resource while moving from _v1_ to _v2_
129+
Consider a scenario where a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}}
130+
(CRD) is created to serve custom resources (CRs) and is set as the preferred storage schema. When it's time
131+
to introduce v2 of the CRD, it can be added for serving only with a conversion
132+
webhook. This enables a smoother transition where users can create CRs using
133+
either the v1 or v2 schema, with the webhook in place to perform the necessary
134+
schema conversion between them. Before setting v2 as the preferred storage schema
135+
version, it's important to ensure that all existing CRs stored as v1 are migrated to v2.
136+
This migration can be achieved through _Storage Version Migration_ to migrate all CRs from v1 to v2.
137+
138+
- Create a manifest for the CRD, named `test-crd.yaml`, as follows:
139+
```yaml
140+
apiVersion: apiextensions.k8s.io/v1
141+
kind: CustomResourceDefinition
142+
metadata:
143+
name: selfierequests.stable.example.com
144+
spec:
145+
group: stable.example.com
146+
names:
147+
plural: SelfieRequests
148+
singular: SelfieRequest
149+
kind: SelfieRequest
150+
listKind: SelfieRequestList
151+
scope: Namespaced
152+
versions:
153+
- name: v1
154+
served: true
155+
storage: true
156+
schema:
157+
openAPIV3Schema:
158+
type: object
159+
properties:
160+
hostPort:
161+
type: string
162+
conversion:
163+
strategy: Webhook
164+
webhook:
165+
clientConfig:
166+
url: https://127.0.0.1:9443/crdconvert
167+
caBundle: <CABundle info>
168+
conversionReviewVersions:
169+
- v1
170+
- v2
171+
```
172+
Create CRD using kubectl
173+
```shell
174+
kubectl apply -f test-crd.yaml
175+
```
176+
- Create a manifest for an example testcrd. Name the manifest `cr1.yaml` and use these contents:
177+
```yaml
178+
apiVersion: stable.example.com/v1
179+
kind: SelfieRequest
180+
metadata:
181+
name: cr1
182+
namespace: default
183+
```
184+
Create CR using kubectl
185+
```shell
186+
kubectl apply -f cr1.yaml
187+
```
188+
- Verify that CR is written and stored as v1 by getting the object from etcd.
189+
```shell
190+
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C
191+
```
192+
where [...] contains the additional arguments for connecting to the etcd server.
193+
- Update the CRD `test-crd.yaml` to include v2 version for serving and storage
194+
and v1 as serving only, as follows:
195+
```yaml
196+
apiVersion: apiextensions.k8s.io/v1
197+
kind: CustomResourceDefinition
198+
metadata:
199+
name: selfierequests.stable.example.com
200+
spec:
201+
group: stable.example.com
202+
names:
203+
plural: SelfieRequests
204+
singular: SelfieRequest
205+
kind: SelfieRequest
206+
listKind: SelfieRequestList
207+
scope: Namespaced
208+
versions:
209+
- name: v2
210+
served: true
211+
storage: true
212+
schema:
213+
openAPIV3Schema:
214+
type: object
215+
properties:
216+
host:
217+
type: string
218+
port:
219+
type: string
220+
- name: v1
221+
served: true
222+
storage: false
223+
schema:
224+
openAPIV3Schema:
225+
type: object
226+
properties:
227+
hostPort:
228+
type: string
229+
conversion:
230+
strategy: Webhook
231+
webhook:
232+
clientConfig:
233+
url: 'https://127.0.0.1:9443/crdconvert'
234+
caBundle: <CABundle info>
235+
conversionReviewVersions:
236+
- v1
237+
- v2
238+
```
239+
Update CRD using kubectl
240+
```shell
241+
kubectl apply -f test-crd.yaml
242+
```
243+
- Create CR resource file with name `cr2.yaml` as follows:
244+
```yaml
245+
apiVersion: stable.example.com/v2
246+
kind: SelfieRequest
247+
metadata:
248+
name: cr2
249+
namespace: default
250+
```
251+
- Create CR using kubectl
252+
```shell
253+
kubectl apply -f cr2.yaml
254+
```
255+
- Verify that CR is written and stored as v2 by getting the object from etcd.
256+
```shell
257+
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr2 [...] | hexdump -C
258+
```
259+
where [...] contains the additional arguments for connecting to the etcd server.
260+
- Create a StorageVersionMigration manifest named `migrate-crd.yaml`, with the contents as follows:
261+
```yaml
262+
kind: StorageVersionMigration
263+
apiVersion: storagemigration.k8s.io/v1alpha1
264+
metadata:
265+
name: crdsvm
266+
spec:
267+
resource:
268+
group: stable.example.com
269+
version: v1
270+
resource: SelfieRequest
271+
```
272+
Create the object using _kubectl_ as follows:
273+
```shell
274+
kubectl apply -f migrate-crd.yaml
275+
```
276+
- Monitor migration of secrets using status. Successful migration should have
277+
_Succeeded_ condition set to _true_ in the status field. Get the migration resource
278+
as follows:
279+
```shell
280+
kubectl get storageversionmigration.storagemigration.k8s.io/crdsvm -o yaml
281+
```
282+
283+
The output is similar to:
284+
```yaml
285+
kind: StorageVersionMigration
286+
apiVersion: storagemigration.k8s.io/v1alpha1
287+
metadata:
288+
name: crdsvm
289+
uid: 13062fe4-32d7-47cc-9528-5067fa0c6ac8
290+
resourceVersion: '111'
291+
creationTimestamp: '2024-03-12T22:40:01Z'
292+
spec:
293+
resource:
294+
group: stable.example.com
295+
version: v1
296+
resource: testcrds
297+
status:
298+
conditions:
299+
- type: Running
300+
status: 'False'
301+
lastUpdateTime: '2024-03-12T22:40:03Z'
302+
reason: StorageVersionMigrationInProgress
303+
- type: Succeeded
304+
status: 'True'
305+
lastUpdateTime: '2024-03-12T22:40:03Z'
306+
reason: StorageVersionMigrationSucceeded
307+
resourceVersion: '106'
308+
```
309+
- Verify that previously created cr1 is now written and stored as v2 by getting the object from etcd.
310+
```shell
311+
ETCDCTL_API=3 etcdctl get /kubernetes.io/stable.example.com/testcrds/default/cr1 [...] | hexdump -C
312+
```
313+
where [...] contains the additional arguments for connecting to the etcd server.

0 commit comments

Comments
 (0)