Skip to content

Commit cdef287

Browse files
deepakkinnideepakkinni
committed
Add blogpost for honor PV reclaim policy fix moving to Beta
Co-authored-by: 杨朱 · Kiki <[email protected]> Co-authored-by: Nate W <[email protected]> Co-authored-by: Qiming Teng <[email protected]> Co-authored-by: Tim Bannister <[email protected]> Signed-off-by: Deepak Kinni <[email protected]>
1 parent 0e55112 commit cdef287

File tree

1 file changed

+196
-0
lines changed

1 file changed

+196
-0
lines changed

‎content/en/blog/_posts/2024-08-16-Honor-PV-Reclaim-Policy-Beta.md

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
---
2+
layout: blog
3+
title: 'Kubernetes 1.31: Prevent PersistentVolume Leaks When Deleting out of Order'
4+
date: 2024-08-16
5+
slug: kubernetes-1-31-prevent-persistentvolume-leaks-when-deleting-out-of-order
6+
author: >
7+
Deepak Kinni (Broadcom)
8+
---
9+
10+
[PersistentVolume](/docs/concepts/storage/persistent-volumes/) (or PVs for short) are
11+
associated with [Reclaim Policy](/docs/concepts/storage/persistent-volumes/#reclaim-policy).
12+
The reclaim policy is used to determine the actions that need to be taken by the storage
13+
backend on deletion of the PVC Bound to a PV.
14+
When the reclaim policy is `Delete`, the expectation is that the storage backend
15+
releases the storage resource allocated for the PV. In essence, the reclaim
16+
policy needs to be honored on PV deletion.
17+
18+
With the recent Kubernetes v1.31 release, a beta feature lets you configure your
19+
cluster to behave that way and honor the configured reclaim policy.
20+
21+
22+
## How did reclaim work in previous Kubernetes releases?
23+
24+
[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#Introduction) (or PVC for short) is
25+
a user's request for storage. A PV and PVC are considered [Bound](/docs/concepts/storage/persistent-volumes/#Binding)
26+
if a newly created PV or a matching PV is found. The PVs themselves are
27+
backed by volumes allocated by the storage backend.
28+
29+
Normally, if the volume is to be deleted, then the expectation is to delete the
30+
PVC for a bound PV-PVC pair. However, there are no restrictions on deleting a PV
31+
before deleting a PVC.
32+
33+
First, I'll demonstrate the behavior for clusters running an older version of Kubernetes.
34+
35+
#### Retrieve a PVC that is bound to a PV
36+
37+
Retrieve an existing PVC `example-vanilla-block-pvc`
38+
```
39+
kubectl get pvc example-vanilla-block-pvc
40+
```
41+
The following output shows the PVC and its bound PV; the PV is shown under the `VOLUME` column:
42+
```
43+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
44+
example-vanilla-block-pvc Bound pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO example-vanilla-block-sc 19s
45+
```
46+
47+
#### Delete PV
48+
49+
When I try to delete a bound PV, the kubectl session blocks and the `kubectl`
50+
tool does not return back control to the shell; for example:
51+
52+
```
53+
kubectl delete pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
54+
```
55+
56+
```
57+
persistentvolume "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" deleted
58+
^C
59+
```
60+
61+
#### Retrieving the PV
62+
```
63+
kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
64+
```
65+
66+
It can be observed that the PV is in a `Terminating` state
67+
```
68+
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
69+
pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO Delete Terminating default/example-vanilla-block-pvc example-vanilla-block-sc 2m23s
70+
```
71+
72+
#### Delete PVC
73+
74+
```
75+
kubectl delete pvc example-vanilla-block-pvc
76+
```
77+
78+
The following output is seen if the PVC gets successfully deleted:
79+
```
80+
persistentvolumeclaim "example-vanilla-block-pvc" deleted
81+
```
82+
83+
The PV object from the cluster also gets deleted. When attempting to retrieve the PV
84+
it will be observed that the PV is no longer found:
85+
86+
```
87+
kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
88+
```
89+
90+
```
91+
Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found
92+
```
93+
94+
Although the PV is deleted, the underlying storage resource is not deleted and
95+
needs to be removed manually.
96+
97+
To sum up, the reclaim policy associated with the PersistentVolume is currently
98+
ignored under certain circumstances. For a `Bound` PV-PVC pair, the ordering of PV-PVC
99+
deletion determines whether the PV reclaim policy is honored. The reclaim policy
100+
is honored if the PVC is deleted first; however, if the PV is deleted prior to
101+
deleting the PVC, then the reclaim policy is not exercised. As a result of this behavior,
102+
the associated storage asset in the external infrastructure is not removed.
103+
104+
## PV reclaim policy with Kubernetes v1.31
105+
106+
The new behavior ensures that the underlying storage object is deleted from the backend when users attempt to delete a PV manually.
107+
108+
#### How to enable new behavior?
109+
110+
To take advantage of the new behavior, you must have upgraded your cluster to the v1.31 release of Kubernetes
111+
and run the CSI [`external-provisioner`](https://github.com/kubernetes-csi/external-provisioner) version `5.0.1` or later.
112+
113+
#### How does it work?
114+
115+
For CSI volumes, the new behavior is achieved by adding a [finalizer](/docs/concepts/overview/working-with-objects/finalizers/) `external-provisioner.volume.kubernetes.io/finalizer`
116+
on new and existing PVs. The finalizer is only removed after the storage from the backend is deleted.
117+
`
118+
119+
An example of a PV with the finalizer, notice the new finalizer in the finalizers list
120+
121+
```
122+
kubectl get pv pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53 -o yaml
123+
```
124+
125+
```yaml
126+
apiVersion: v1
127+
kind: PersistentVolume
128+
metadata:
129+
annotations:
130+
pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
131+
creationTimestamp: "2021-11-17T19:28:56Z"
132+
finalizers:
133+
- kubernetes.io/pv-protection
134+
- external-provisioner.volume.kubernetes.io/finalizer
135+
name: pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53
136+
resourceVersion: "194711"
137+
uid: 087f14f2-4157-4e95-8a70-8294b039d30e
138+
spec:
139+
accessModes:
140+
- ReadWriteOnce
141+
capacity:
142+
storage: 1Gi
143+
claimRef:
144+
apiVersion: v1
145+
kind: PersistentVolumeClaim
146+
name: example-vanilla-block-pvc
147+
namespace: default
148+
resourceVersion: "194677"
149+
uid: a7b7e3ba-f837-45ba-b243-dec7d8aaed53
150+
csi:
151+
driver: csi.vsphere.vmware.com
152+
fsType: ext4
153+
volumeAttributes:
154+
storage.kubernetes.io/csiProvisionerIdentity: 1637110610497-8081-csi.vsphere.vmware.com
155+
type: vSphere CNS Block Volume
156+
volumeHandle: 2dacf297-803f-4ccc-afc7-3d3c3f02051e
157+
persistentVolumeReclaimPolicy: Delete
158+
storageClassName: example-vanilla-block-sc
159+
volumeMode: Filesystem
160+
status:
161+
phase: Bound
162+
```
163+
164+
The [finalizer](/docs/concepts/overview/working-with-objects/finalizers/) prevents this
165+
PersistentVolume from being removed from the
166+
cluster. As stated previously, the finalizer is only removed from the PV object
167+
after it is successfully deleted from the storage backend. To learn more about
168+
finalizers, please refer to [Using Finalizers to Control Deletion](/blog/2021/05/14/using-finalizers-to-control-deletion/).
169+
170+
Similarly, the finalizer `kubernetes.io/pv-controller` is added to dynamically provisioned in-tree plugin volumes.
171+
172+
#### What about CSI migrated volumes?
173+
174+
The fix applies to CSI migrated volumes as well.
175+
176+
### Some caveats
177+
178+
The fix does not apply to statically provisioned in-tree plugin volumes.
179+
180+
### References
181+
182+
* [KEP-2644](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2644-honor-pv-reclaim-policy)
183+
* [Volume leak issue](https://github.com/kubernetes-csi/external-provisioner/issues/546)
184+
185+
### How do I get involved?
186+
187+
The Kubernetes Slack channel [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams.
188+
189+
Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution:
190+
191+
* Fan Baofa (carlory)
192+
* Jan Šafránek (jsafrane)
193+
* Xing Yang (xing-yang)
194+
* Matthew Wong (wongma7)
195+
196+
Join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage) if you're interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system. We’re rapidly growing and always welcome new contributors.

0 commit comments

Comments
 (0)