2
2
layout : blog
3
3
title : " Kubernetes 1.26: StatefulSet Start Ordinal Simplifies Migration"
4
4
date : 2023-01-03
5
- slug : statefulset-migration
5
+ slug : statefulset-start-ordinal
6
6
---
7
7
8
8
** Author** : Peter Schuurman (Google)
@@ -32,11 +32,12 @@ You lose the self-healing benefit of the StatefulSet controller when your Pods
32
32
fail or are evicted.
33
33
34
34
Kubernetes v1.26 enables a StatefulSet to be responsible for a range of ordinals
35
- within a half-open interval ` [0, N) ` (the ordinals 0, 1, ... N-1).
35
+ within a range {0..N-1} (the ordinals 0, 1, ... up to N-1).
36
36
With it, you can scale down a range
37
- ( ` [0, k) ` ) in a source cluster, and scale up the complementary range ( ` [k, N) ` )
37
+ {0..k-1} in a source cluster, and scale up the complementary range {k..N-1}
38
38
in a destination cluster, while maintaining application availability. This
39
- enables you to retain * at most one* semantics and
39
+ enables you to retain * at most one* semantics (meaning there is at most one Pod
40
+ with a given identity running in a StatefulSet) and
40
41
[ Rolling Update] ( /docs/tutorials/stateful-application/basic-stateful-set/#rolling-update )
41
42
behavior when orchestrating a migration across clusters.
42
43
@@ -61,42 +62,50 @@ to a different cluster. There are many reasons why you would need to do this:
61
62
Enable the ` StatefulSetStartOrdinal ` feature gate on a cluster, and create a
62
63
StatefulSet with a customized ` .spec.ordinals.start ` .
63
64
64
- ## Try it for yourself
65
+ ## Try it out
65
66
66
- In this demo, you 'll use the ` StatefulSetStartOrdinal ` feature to migrate a
67
- StatefulSet from one Kubernetes cluster to another. For this demo, the
67
+ In this demo, I 'll use the new mechanism to migrate a
68
+ StatefulSet from one Kubernetes cluster to another. The
68
69
[ redis-cluster] ( https://github.com/bitnami/charts/tree/main/bitnami/redis-cluster )
69
- Bitnami Helm chart is used to install Redis.
70
+ Bitnami Helm chart will be used to install Redis.
70
71
71
72
Tools Required:
72
73
* [ yq] ( https://github.com/mikefarah/yq )
73
74
* [ helm] ( https://helm.sh/docs/helm/helm_install/ )
74
75
75
- Pre-requisites: Two Kubernetes clusters named ` source ` and ` destination ` .
76
- * ` StatefulSetStartOrdinal ` feature gate is enabled on both clusters
77
- * The same default ` StorageClass ` is installed on both clusters. This
78
- ` StorageClass ` should provision underlying storage that is accessible from
79
- both clusters.
80
- * A flat network topology that allows for pods to be accessible across both
81
- Kubernetes clusters. If creating clusters on a cloud provider, this
82
- configuration may be called private cloud or private network.
76
+ ### Pre-requisites {#demo-pre-requisites}
83
77
84
- 1 . Create a demo namespace on both clusters.
78
+ To do this, I need two Kubernetes clusters that can both access common
79
+ networking and storage; I've named my clusters ` source ` and ` destination ` .
80
+ Specifically, I need:
81
+
82
+ * The ` StatefulSetStartOrdinal ` feature gate enabled on both clusters.
83
+ * Client configuration for ` kubectl ` that lets me access both clusters as an
84
+ administrator.
85
+ * The same ` StorageClass ` installed on both clusters, and set as the default
86
+ StorageClass for both clusters. This ` StorageClass ` should provision
87
+ underlying storage that is accessible from either or both clusters.
88
+ * A flat network topology that allows for pods to send and receive packets to
89
+ and from Pods in either clusters. If you are creating clusters on a cloud
90
+ provider, this configuration may be called private cloud or private network.
91
+
92
+ 1 . Create a demo namespace on both clusters:
85
93
86
94
```
87
95
kubectl create ns kep-3335
88
96
```
89
97
90
- 2 . Deploy a Redis cluster on ` source ` .
98
+ 2 . Deploy a Redis cluster with six replicas in the source cluster:
91
99
92
100
```
93
101
helm repo add bitnami https://charts.bitnami.com/bitnami
94
102
helm install redis --namespace kep-3335 \
95
103
bitnami/redis-cluster \
96
- --set persistence.size=1Gi
104
+ --set persistence.size=1Gi \
105
+ --set cluster.nodes=6
97
106
```
98
107
99
- 3 . On ` source ` , check the replication status.
108
+ 3 . Check the replication status in the source cluster:
100
109
101
110
```
102
111
kubectl exec -it redis-redis-cluster-0 -- /bin/bash -c \
@@ -112,7 +121,7 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
112
121
2cff613d763b22c180cd40668da8e452edef3fc8 10.104.0.17:6379@16379 master - 0 1669764410000 2 connected 5461-10922
113
122
```
114
123
115
- 4 . On ` destination ` , deploy Redis with zero replicas.
124
+ 4 . Deploy a Redis cluster with zero replicas in the destination cluster:
116
125
117
126
```
118
127
helm install redis --namespace kep-3335 \
@@ -123,19 +132,20 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
123
132
--set existingSecret=redis-redis-cluster
124
133
```
125
134
126
- 5 . Scale down replica ` redis-redis-cluster-5 ` in the source cluster.
135
+ 5 . Scale down the ` redis-redis-cluster ` StatefulSet in the source cluster by 1,
136
+ to remove the replica ` redis-redis-cluster-5 ` :
127
137
128
138
```
129
139
kubectl patch sts redis-redis-cluster -p '{"spec": {"replicas": 5}}'
130
140
```
131
141
132
- 6 . Migrate dependencies from ` source ` to ` destination ` .
142
+ 6 . Migrate dependencies from the source cluster to the destination cluster:
133
143
134
144
The following commands copy resources from ` source ` to ` destionation ` . Details
135
145
that are not relevant in ` destination ` cluster are removed (eg: ` uid ` ,
136
146
` resourceVersion ` , ` status ` ).
137
147
138
- #### Source cluster
148
+ ** Steps for the source cluster**
139
149
140
150
Note: If using a ` StorageClass ` with ` reclaimPolicy: Delete ` configured, you
141
151
should patch the PVs in ` source ` with ` reclaimPolicy: Retain ` prior to
@@ -149,7 +159,7 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
149
159
kubectl get secret redis-redis-cluster -o yaml | yq 'del(.metadata.uid, .metadata.resourceVersion)' > /tmp/secret-redis-redis-cluster.yaml
150
160
```
151
161
152
- #### Destination cluster
162
+ ** Steps for the destination cluster**
153
163
154
164
Note: For the PV/PVC, this procedure only works if the underlying storage system
155
165
that your PVs use can support being copied into ` destination ` . Storage
@@ -164,42 +174,49 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
164
174
kubectl create -f /tmp/secret-redis-redis-cluster.yaml
165
175
```
166
176
167
- 7 . Scale up replica ` redis-redis-cluster-5 ` in the destination cluster, with a
168
- start ordinal of 5:
177
+ 7 . Scale up the ` redis-redis-cluster ` StatefulSet in the destination cluster by
178
+ 1, with a start ordinal of 5:
169
179
170
180
```
171
181
kubectl patch sts redis-redis-cluster -p '{"spec": {"ordinals": {"start": 5}, "replicas": 1}}'
172
182
```
173
183
174
- 8 . On the source cluster, check the replication status.
184
+ 8 . Check the replication status in the destination cluster:
175
185
176
186
```
177
- kubectl exec -it redis-redis-cluster-0 -- /bin/bash -c \
187
+ kubectl exec -it redis-redis-cluster-5 -- /bin/bash -c \
178
188
"redis-cli -c -h redis-redis-cluster -a $(kubectl get secret redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d) CLUSTER NODES;"
179
189
```
180
190
181
- You should see that the new replica's address has joined the Redis cluster.
191
+ I should see that the new replica (labeled ` myself ` ) has joined the Redis
192
+ cluster (the IP address belongs to a different CIDR block than the
193
+ replicas in the source cluster).
182
194
183
195
```
184
- 2cff613d763b22c180cd40668da8e452edef3fc8 10.104.0.17:6379@16379 myself, master - 0 1669766684000 2 connected 5461-10922
185
- 7136e37d8864db983f334b85d2b094be47c830e5 10.108.0.22:6379@16379 slave 2cff613d763b22c180cd40668da8e452edef3fc8 0 1669766685609 2 connected
196
+ 2cff613d763b22c180cd40668da8e452edef3fc8 10.104.0.17:6379@16379 master - 0 1669766684000 2 connected 5461-10922
197
+ 7136e37d8864db983f334b85d2b094be47c830e5 10.108.0.22:6379@16379 myself, slave 2cff613d763b22c180cd40668da8e452edef3fc8 0 1669766685609 2 connected
186
198
2ce30362c188aabc06f3eee5d92892d95b1da5c3 10.104.0.14:6379@16379 master - 0 1669766684000 3 connected 10923-16383
187
199
961f35e37c4eea507cfe12f96e3bfd694b9c21d4 10.104.0.18:6379@16379 slave a8765caed08f3e185cef22bd09edf409dc2bcc61 0 1669766683600 1 connected
188
200
a8765caed08f3e185cef22bd09edf409dc2bcc61 10.104.0.19:6379@16379 master - 0 1669766685000 1 connected 0-5460
189
201
7743661f60b6b17b5c71d083260419588b4f2451 10.104.0.16:6379@16379 slave 2ce30362c188aabc06f3eee5d92892d95b1da5c3 0 1669766686613 3 connected
190
202
```
191
203
204
+ 9 . Repeat steps #5 to #7 for the remainder of the replicas, until the
205
+ Redis StatefulSet in the source cluster is scaled to 0, and the Redis
206
+ StatefulSet in the destination cluster is healthy with 6 total replicas.
207
+
192
208
## What's Next?
193
209
194
210
This feature provides a building block for a StatefulSet to be split up across
195
211
clusters, but does not prescribe the mechanism as to how the StatefulSet should
196
212
be migrated. Migration requires coordination of StatefulSet replicas, along with
197
213
orchestration of the storage and network layer. This is dependent on the storage
198
214
and connectivity requirements of the application installed by the StatefulSet.
199
- Additionally, many StatefulSets are controlled by Operators, which adds another
215
+ Additionally, many StatefulSets are managed by
216
+ [ operators] ( /docs/concepts/extend-kubernetes/operator/ ) , which adds another
200
217
layer of complexity to migration.
201
218
202
- If you're interested in building blocks to make these processes easier, get
203
- involved with
219
+ If you're interested in building enhancements to make these processes easier,
220
+ get involved with
204
221
[ SIG Multicluster] ( https://github.com/kubernetes/community/blob/master/sig-multicluster )
205
222
to contribute!
0 commit comments