19
19
- [ Design Details] ( #design-details )
20
20
- [ Objects Associated with the StatefulSet] ( #objects-associated-with-the-statefulset )
21
21
- [ Volume delete policy for the StatefulSet created PVCs] ( #volume-delete-policy-for-the-statefulset-created-pvcs )
22
- - [ <code >OnScaleDown </code > policy of <code >Delete</code >.] ( #-policy-of- )
23
- - [ <code >OnSetDeletion </code > policy of <code >Delete</code >.] ( #-policy-of--1 )
22
+ - [ <code >whenScaled </code > policy of <code >Delete</code >.] ( #-policy-of- )
23
+ - [ <code >whenDeleted </code > policy of <code >Delete</code >.] ( #-policy-of--1 )
24
24
- [ Non-Cascading Deletion] ( #non-cascading-deletion )
25
- - [ Mutating <code >PersistentVolumeClaimPolicy </code >] ( #mutating- )
25
+ - [ Mutating <code >PersistentVolumeClaimRetentionPolicy </code >] ( #mutating- )
26
26
- [ Cluster role change for statefulset controller] ( #cluster-role-change-for-statefulset-controller )
27
27
- [ Test Plan] ( #test-plan )
28
28
- [ Graduation Criteria] ( #graduation-criteria )
@@ -86,7 +86,7 @@ delete the PVC.
86
86
### Non-Goals
87
87
88
88
This proposal does not plan to address how the underlying PVs are treated on PVC deletion.
89
- That functionality will continue to be governed by the ReclaimPolicy of the storage class.
89
+ That functionality will continue to be governed by the reclaim policy of the storage class.
90
90
91
91
## Proposal
92
92
@@ -102,10 +102,10 @@ delete the PVCs created by the controller from the StatefulSet's VolumeClaimTemp
102
102
103
103
The following changes are required:
104
104
105
- 1 . Add ` PersistentVolumeClaimPolicy ` to the StatefulSet spec with the following fields.
106
- * ` OnSetDeletion ` - specifies if the VolumeClaimTemplate PVCs are deleted when
105
+ 1 . Add ` persistentVolumeClaimRetentionPolicy ` to the StatefulSet spec with the following fields.
106
+ * ` whenDeleted ` - specifies if the VolumeClaimTemplate PVCs are deleted when
107
107
their StatefulSet is deleted.
108
- * ` OnScaleDown ` - specifies if VolumeClaimTemplate PVCs are deleted when
108
+ * ` whenScaled ` - specifies if VolumeClaimTemplate PVCs are deleted when
109
109
their corresponding pod is deleted on a StatefulSet scale-down, that is,
110
110
when the number of pods in a StatefulSet is reduced via the Replicas field.
111
111
@@ -122,7 +122,7 @@ The following changes are required:
122
122
123
123
#### Story 0
124
124
The user is happy with legacy behavior of a stateful set. They leave all fields
125
- of ` PersistentVolumeClaimPolicy ` to ` Retain ` . Nothing traditional
125
+ of ` PersistentVolumeClaimRetentionPolicy ` to ` Retain ` . Nothing traditional
126
126
StatefulSet behavior changes neither on set deletion nor on scale-down.
127
127
128
128
#### Story 1
@@ -133,7 +133,7 @@ so that scale-up can leverage the existing volumes. When the application is fini
133
133
volumes created by the StatefulSet are no longer needed and can be automatically
134
134
reclaimed.
135
135
136
- The user would set ` PersistentVolumeClaimPolicy.OnSetDeletion ` to `Delete, which
136
+ The user would set ` persistentVolumeClaimRetentionPolicy.whenDeleted ` to `Delete, which
137
137
would ensure that the PVCs created automatically during the StatefulSet
138
138
activation is deleted once the StatefulSet is deleted.
139
139
@@ -143,8 +143,8 @@ scale-down, because scaling events are rare, and volume data can be
143
143
reconstructed, albeit slowly, during a scale up. However, it is necessary to
144
144
bring down the StatefulSet temporarily by deleting it, and then bring it back up
145
145
by reusing the volumes. This is accomplished by setting
146
- ` PersistentVolumeClaimPolicy.OnScaleDown ` to ` Delete ` , and leaving
147
- ` PersistentVolumeClaimPolicy.OnSetDeletion ` at ` Retain ` .
146
+ ` persistentVolumeClaimRetentionPolicy.whenScaled ` to ` Delete ` , and leaving
147
+ ` persistentVolumeClaimRetentionPolicy.whenDeleted ` at ` Retain ` .
148
148
149
149
#### Story 3
150
150
User is very cost conscious, and can sustain slower scale-up speeds even after a
@@ -156,8 +156,8 @@ recreated, like a rolling update or node disruptions, the data on volumes is
156
156
persisted. This is a key property that ephemeral storage, like emptyDir, cannot
157
157
provide.
158
158
159
- User would set the ` PersistentVolumeClaimPolicy.OnScaleDown ` as well as
160
- ` PersistentVolumeClaimPolicy.OnSetDeletion ` to ` Delete ` , ensuring PVCs are
159
+ User would set the ` persistentVolumeClaimRetentionPolicy.whenScaled ` as well as
160
+ ` persistentVolumeClaimRetentionPolicy.whenDeleted ` to ` Delete ` , ensuring PVCs are
161
161
deleted when corresponding Pods are deleted. New Pods created during scale-up
162
162
followed by a scale-down will wait for freshly created PVCs. PVCs are deleted as
163
163
well when the set is deleted, reclaiming volumes as quickly as possible and
@@ -175,7 +175,7 @@ VolumeClaimTemplate it will be deleted according to the deletion policy.
175
175
### Risks and Mitigations
176
176
177
177
Currently the PVCs created by StatefulSet are not deleted automatically. Using
178
- ` OnScaleDown ` or ` OnSetDeletion ` set to ` Delete ` would delete the PVCs
178
+ ` whenScaled ` or ` whenDeleted ` set to ` Delete ` would delete the PVCs
179
179
automatically. Since this involves persistent data being deleted, users should
180
180
take appropriate care using this feature. Having the ` Retain ` behaviour as
181
181
default will ensure that the PVCs remain intact by default and only a conscious
@@ -211,13 +211,13 @@ are affected.
211
211
212
212
### Volume delete policy for the StatefulSet created PVCs
213
213
214
- A new field named ` PersistentVolumeClaimPolicy ` of the type
215
- ` StatefulSetPersistentVolumeClaimPolicy ` will be added to the StatefulSet. This
214
+ A new field named ` PersistentVolumeClaimRetentionPolicy ` of the type
215
+ ` StatefulSetPersistentVolumeClaimRetentionPolicy ` will be added to the StatefulSet. This
216
216
will represent the user indication for which circumstances the associated PVCs
217
217
can be automatically deleted or not, as described above. The default policy
218
218
would be to retain PVCs in all cases.
219
219
220
- The ` PersistentVolumeClaimPolicy ` object will be mutable. The deletion
220
+ The ` PersistentVolumeClaimRetentionPolicy ` object will be mutable. The deletion
221
221
mechanism will be based on reconciliation, so as long as the field is changed
222
222
far from StatefulSet deletion or scale-down, the policy will work as
223
223
expected. Mutability does introduce race conditions if it is changed while a
@@ -230,9 +230,9 @@ manually deleting PVCs. The latter case will result in lost data, but only in
230
230
PVCs that were originally declared to have been deleted. Life does not always
231
231
have an undo button.
232
232
233
- #### ` OnScaleDown ` policy of ` Delete ` .
233
+ #### ` whenScaled ` policy of ` Delete ` .
234
234
235
- If ` PersistentVolumeClaimPolicy.OnScaleDown ` is set to ` Delete ` , the Pod will be
235
+ If ` persistentVolumeClaimRetentionPolicy.whenScaled ` is set to ` Delete ` , the Pod will be
236
236
set as the owner of the PVCs created from the ` VolumeClaimTemplates ` just before
237
237
the scale-down is performed by the StatefulSet controller. When a Pod is
238
238
deleted, the PVC owned by the Pod is also deleted.
@@ -247,17 +247,17 @@ the PVC was referred to by the deleted Pod and is in the process of getting
247
247
deleted. The controller will skip the reconcile loop until PVC deletion finishes, avoiding
248
248
a race condition.
249
249
250
- #### ` OnSetDeletion ` policy of ` Delete ` .
250
+ #### ` whenDeleted ` policy of ` Delete ` .
251
251
252
- When ` PersistentVolumeClaimPolicy.OnSetDeletion ` is set to ` Delete ` , when a
252
+ When ` persistentVolumeClaimRetentionPolicy.whenDeleted ` is set to ` Delete ` , when a
253
253
VolumeClaimTemplate PVC is created, an owner reference in PVC will be added to
254
254
point to the StatefulSet. When a scale-up or scale-down occurs, the PVC is
255
255
unchanged. PVCs previously in use before scale-down will be used again when the
256
256
scale-up occurs.
257
257
258
258
In the existing StatefulSet reconcile loop, the associated VolumeClaimTemplate
259
259
PVCs will be checked to see if the ownerRef is correct according to the
260
- ` PersistentVolumeClaimPolicy ` and updated accordingly. This includes PVCs
260
+ ` persistentVolumeClaimRetentionPolicy ` and updated accordingly. This includes PVCs
261
261
that have been manually provisioned. It will be most consistent and easy
262
262
to reason about if all VolumeClaimTemplate PVCs are treated uniformly rather
263
263
than trying to guess at their provenance.
@@ -279,7 +279,7 @@ When StatefulSet is deleted without cascading, eg `kubectl delete --cascade=fals
279
279
existing behavior is retained and no PVC will be deleted. Only the StatefulSet resource
280
280
will be affected.
281
281
282
- #### Mutating ` PersistentVolumeClaimPolicy `
282
+ #### Mutating ` PersistentVolumeClaimRetentionPolicy `
283
283
284
284
Recall that as defined above, the PVCs associated with a StatefulSet are found
285
285
by the StatefulSet volumeClaimTemplate static naming scheme. The Pods associated
@@ -308,7 +308,7 @@ In order to update the PVC ownerReference, the `buildControllerRoles` will be up
308
308
1 . Unit tests
309
309
310
310
1 . e2e/integration tests
311
- - With ` Delete ` policies for ` OnSetDeletion ` and ` OnScaleDown `
311
+ - With ` Delete ` policies for ` whenScaled ` and ` whenDeleted `
312
312
1 . Create 2 pod statefulset, scale to 1 pod, confirm PVC deleted
313
313
1 . Create 2 pod statefulset, add data to PVs, scale to 1 pod, scale back to 2, confirm PV empty.
314
314
1 . Create 2 pod statefulset, delete stateful set, confirm PVCs deleted.
@@ -317,7 +317,7 @@ In order to update the PVC ownerReference, the `buildControllerRoles` will be up
317
317
1 . Create 2 pod statefulset, add data to PVs, manually delete one pod, immediately scale down to one pod, confirm PVC is deleted.
318
318
1 . Create 2 pod statefulset, add data to PVs, manually delete one pod, immediately scale down to one pod, scale back to two pods, confirm PV is empty.
319
319
1 . Create 2 pod statefulset, add data to PVs, perform rolling confirm PVC don't get deleted and PV contents remain intact and useful in the updated pods.
320
- - With ` Delete ` policy for ` OnSetDeletion ` only
320
+ - With ` Delete ` policy for ` whenDeleted ` only
321
321
1 . Create 2 pod statefulset, scale to 1 pod, confirm PVC still exists,
322
322
1 . Create 2 pod statefulset, add data to PVs, scale to 1 pod, scale back to 2, confirm PV has data (PVC not deleted).
323
323
1 . Create 2 pod statefulset, delete stateful set, confirm PVCs deleted
@@ -350,7 +350,7 @@ In order to update the PVC ownerReference, the `buildControllerRoles` will be up
350
350
This features adds a new field to the StatefulSet. The default value for the new field
351
351
maintains the existing behavior of StatefulSets.
352
352
353
- On a downgrade, the ` PersistentVolumeClaimPolicy ` field will be hidden on
353
+ On a downgrade, the ` PersistentVolumeClaimRetentionPolicy ` field will be hidden on
354
354
any StatefulSets. The behavior in this case will be identical to mutating they
355
355
policy field to ` Retain ` , as described above, including the edge cases
356
356
introduced if this is done during a scale-down or StatefulSet deletion.
@@ -382,9 +382,8 @@ are not involved so there is no version skew between nodes and the control plane
382
382
Yes. Disabling the feature gate will cause the new field to be ignored. If the feature
383
383
gate is re-enabled, the new behavior will start working.
384
384
385
- When the ` PersistentVolumeClaimDeletePolicy ` is set to
386
- ` DeleteOnStatefulSetDeletion ` , then VolumeClaimTemplate PVCs ownerRefs must be
387
- removed.
385
+ When the ` PersistentVolumeClaimRetentionPolicy ` has ` WhenDeleted ` set to
386
+ ` Delete ` , then VolumeClaimTemplate PVCs ownerRefs must be removed.
388
387
389
388
There are new corner cases here. For example, if a StatefulSet deletion is in
390
389
process when the feature is disabled or enabled, the appropriate ownerRefs
@@ -394,8 +393,8 @@ are not involved so there is no version skew between nodes and the control plane
394
393
395
394
* ** What happens if we reenable the feature if it was previously rolled back?**
396
395
In the simple case of reenabling the feature without concurrent StatefulSet
397
- deletion or scale-down, nothing needs to be done when the deletion policy is
398
- ` DeleteOnScaleDown ` . On a policy of ` DeleteOnStatefulSetDeletion ` , the
396
+ deletion or scale-down, nothing needs to be done when the deletion policy has
397
+ ` whenScaled ` set to ` Delete ` . When the policy has ` whenDeleted ` set to ` Delete ` , the
399
398
VolumeClaimTemplate PVC ownerRefs must be set to the StatefulSet.
400
399
401
400
As above, if there is a concurrent scale-down or StatefulSet deletion, more
0 commit comments