Skip to content

Commit 13f1c8a

Browse files
committed
Update formatting and wording for StatefulSet Migration Redis demo
1 parent 8ca5a5d commit 13f1c8a

File tree

1 file changed

+52
-35
lines changed

1 file changed

+52
-35
lines changed

content/en/blog/_posts/2022-12-16-statefulset-migration.md

Lines changed: 52 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
layout: blog
33
title: "Kubernetes 1.26: StatefulSet Start Ordinal Simplifies Migration"
44
date: 2023-01-03
5-
slug: statefulset-migration
5+
slug: statefulset-start-ordinal
66
---
77

88
**Author**: Peter Schuurman (Google)
@@ -32,11 +32,12 @@ You lose the self-healing benefit of the StatefulSet controller when your Pods
3232
fail or are evicted.
3333

3434
Kubernetes v1.26 enables a StatefulSet to be responsible for a range of ordinals
35-
within a half-open interval `[0, N)` (the ordinals 0, 1, ... N-1).
35+
within a range {0..N-1} (the ordinals 0, 1, ... up to N-1).
3636
With it, you can scale down a range
37-
(`[0, k)`) in a source cluster, and scale up the complementary range (`[k, N)`)
37+
{0..k-1} in a source cluster, and scale up the complementary range {k..N-1}
3838
in a destination cluster, while maintaining application availability. This
39-
enables you to retain *at most one* semantics and
39+
enables you to retain *at most one* semantics (meaning there is at most one Pod
40+
with a given identity running in a StatefulSet) and
4041
[Rolling Update](/docs/tutorials/stateful-application/basic-stateful-set/#rolling-update)
4142
behavior when orchestrating a migration across clusters.
4243

@@ -61,42 +62,50 @@ to a different cluster. There are many reasons why you would need to do this:
6162
Enable the `StatefulSetStartOrdinal` feature gate on a cluster, and create a
6263
StatefulSet with a customized `.spec.ordinals.start`.
6364

64-
## Try it for yourself
65+
## Try it out
6566

66-
In this demo, you'll use the `StatefulSetStartOrdinal` feature to migrate a
67-
StatefulSet from one Kubernetes cluster to another. For this demo, the
67+
In this demo, I'll use the new mechanism to migrate a
68+
StatefulSet from one Kubernetes cluster to another. The
6869
[redis-cluster](https://github.com/bitnami/charts/tree/main/bitnami/redis-cluster)
69-
Bitnami Helm chart is used to install Redis.
70+
Bitnami Helm chart will be used to install Redis.
7071

7172
Tools Required:
7273
* [yq](https://github.com/mikefarah/yq)
7374
* [helm](https://helm.sh/docs/helm/helm_install/)
7475

75-
Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
76-
* `StatefulSetStartOrdinal` feature gate is enabled on both clusters
77-
* The same default `StorageClass` is installed on both clusters. This
78-
`StorageClass` should provision underlying storage that is accessible from
79-
both clusters.
80-
* A flat network topology that allows for pods to be accessible across both
81-
Kubernetes clusters. If creating clusters on a cloud provider, this
82-
configuration may be called private cloud or private network.
76+
### Pre-requisites {#demo-pre-requisites}
8377

84-
1. Create a demo namespace on both clusters.
78+
To do this, I need two Kubernetes clusters that can both access common
79+
networking and storage; I've named my clusters `source` and `destination`.
80+
Specifically, I need:
81+
82+
* The `StatefulSetStartOrdinal` feature gate enabled on both clusters.
83+
* Client configuration for `kubectl` that lets me access both clusters as an
84+
administrator.
85+
* The same `StorageClass` installed on both clusters, and set as the default
86+
StorageClass for both clusters. This `StorageClass` should provision
87+
underlying storage that is accessible from either or both clusters.
88+
* A flat network topology that allows for pods to send and receive packets to
89+
and from Pods in either clusters. If you are creating clusters on a cloud
90+
provider, this configuration may be called private cloud or private network.
91+
92+
1. Create a demo namespace on both clusters:
8593

8694
```
8795
kubectl create ns kep-3335
8896
```
8997

90-
2. Deploy a Redis cluster on `source`.
98+
2. Deploy a Redis cluster with six replicas in the source cluster:
9199

92100
```
93101
helm repo add bitnami https://charts.bitnami.com/bitnami
94102
helm install redis --namespace kep-3335 \
95103
bitnami/redis-cluster \
96-
--set persistence.size=1Gi
104+
--set persistence.size=1Gi \
105+
--set cluster.nodes=6
97106
```
98107

99-
3. On `source`, check the replication status.
108+
3. Check the replication status in the source cluster:
100109

101110
```
102111
kubectl exec -it redis-redis-cluster-0 -- /bin/bash -c \
@@ -112,7 +121,7 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
112121
2cff613d763b22c180cd40668da8e452edef3fc8 10.104.0.17:6379@16379 master - 0 1669764410000 2 connected 5461-10922
113122
```
114123

115-
4. On `destination`, deploy Redis with zero replicas.
124+
4. Deploy a Redis cluster with zero replicas in the destination cluster:
116125

117126
```
118127
helm install redis --namespace kep-3335 \
@@ -123,19 +132,20 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
123132
--set existingSecret=redis-redis-cluster
124133
```
125134

126-
5. Scale down replica `redis-redis-cluster-5` in the source cluster.
135+
5. Scale down the `redis-redis-cluster` StatefulSet in the source cluster by 1,
136+
to remove the replica `redis-redis-cluster-5`:
127137

128138
```
129139
kubectl patch sts redis-redis-cluster -p '{"spec": {"replicas": 5}}'
130140
```
131141

132-
6. Migrate dependencies from `source` to `destination`.
142+
6. Migrate dependencies from the source cluster to the destination cluster:
133143

134144
The following commands copy resources from `source` to `destionation`. Details
135145
that are not relevant in `destination` cluster are removed (eg: `uid`,
136146
`resourceVersion`, `status`).
137147

138-
#### Source cluster
148+
**Steps for the source cluster**
139149

140150
Note: If using a `StorageClass` with `reclaimPolicy: Delete` configured, you
141151
should patch the PVs in `source` with `reclaimPolicy: Retain` prior to
@@ -149,7 +159,7 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
149159
kubectl get secret redis-redis-cluster -o yaml | yq 'del(.metadata.uid, .metadata.resourceVersion)' > /tmp/secret-redis-redis-cluster.yaml
150160
```
151161

152-
#### Destination cluster
162+
**Steps for the destination cluster**
153163

154164
Note: For the PV/PVC, this procedure only works if the underlying storage system
155165
that your PVs use can support being copied into `destination`. Storage
@@ -164,42 +174,49 @@ Pre-requisites: Two Kubernetes clusters named `source` and `destination`.
164174
kubectl create -f /tmp/secret-redis-redis-cluster.yaml
165175
```
166176

167-
7. Scale up replica `redis-redis-cluster-5` in the destination cluster, with a
168-
start ordinal of 5:
177+
7. Scale up the `redis-redis-cluster` StatefulSet in the destination cluster by
178+
1, with a start ordinal of 5:
169179

170180
```
171181
kubectl patch sts redis-redis-cluster -p '{"spec": {"ordinals": {"start": 5}, "replicas": 1}}'
172182
```
173183

174-
8. On the source cluster, check the replication status.
184+
8. Check the replication status in the destination cluster:
175185

176186
```
177-
kubectl exec -it redis-redis-cluster-0 -- /bin/bash -c \
187+
kubectl exec -it redis-redis-cluster-5 -- /bin/bash -c \
178188
"redis-cli -c -h redis-redis-cluster -a $(kubectl get secret redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d) CLUSTER NODES;"
179189
```
180190

181-
You should see that the new replica's address has joined the Redis cluster.
191+
I should see that the new replica (labeled `myself`) has joined the Redis
192+
cluster (the IP address belongs to a different CIDR block than the
193+
replicas in the source cluster).
182194

183195
```
184-
2cff613d763b22c180cd40668da8e452edef3fc8 10.104.0.17:6379@16379 myself,master - 0 1669766684000 2 connected 5461-10922
185-
7136e37d8864db983f334b85d2b094be47c830e5 10.108.0.22:6379@16379 slave 2cff613d763b22c180cd40668da8e452edef3fc8 0 1669766685609 2 connected
196+
2cff613d763b22c180cd40668da8e452edef3fc8 10.104.0.17:6379@16379 master - 0 1669766684000 2 connected 5461-10922
197+
7136e37d8864db983f334b85d2b094be47c830e5 10.108.0.22:6379@16379 myself,slave 2cff613d763b22c180cd40668da8e452edef3fc8 0 1669766685609 2 connected
186198
2ce30362c188aabc06f3eee5d92892d95b1da5c3 10.104.0.14:6379@16379 master - 0 1669766684000 3 connected 10923-16383
187199
961f35e37c4eea507cfe12f96e3bfd694b9c21d4 10.104.0.18:6379@16379 slave a8765caed08f3e185cef22bd09edf409dc2bcc61 0 1669766683600 1 connected
188200
a8765caed08f3e185cef22bd09edf409dc2bcc61 10.104.0.19:6379@16379 master - 0 1669766685000 1 connected 0-5460
189201
7743661f60b6b17b5c71d083260419588b4f2451 10.104.0.16:6379@16379 slave 2ce30362c188aabc06f3eee5d92892d95b1da5c3 0 1669766686613 3 connected
190202
```
191203

204+
9. Repeat steps #5 to #7 for the remainder of the replicas, until the
205+
Redis StatefulSet in the source cluster is scaled to 0, and the Redis
206+
StatefulSet in the destination cluster is healthy with 6 total replicas.
207+
192208
## What's Next?
193209

194210
This feature provides a building block for a StatefulSet to be split up across
195211
clusters, but does not prescribe the mechanism as to how the StatefulSet should
196212
be migrated. Migration requires coordination of StatefulSet replicas, along with
197213
orchestration of the storage and network layer. This is dependent on the storage
198214
and connectivity requirements of the application installed by the StatefulSet.
199-
Additionally, many StatefulSets are controlled by Operators, which adds another
215+
Additionally, many StatefulSets are managed by
216+
[operators](/docs/concepts/extend-kubernetes/operator/), which adds another
200217
layer of complexity to migration.
201218

202-
If you're interested in building blocks to make these processes easier, get
203-
involved with
219+
If you're interested in building enhancements to make these processes easier,
220+
get involved with
204221
[SIG Multicluster](https://github.com/kubernetes/community/blob/master/sig-multicluster)
205222
to contribute!

0 commit comments

Comments
 (0)