Skip to content

Commit 19e2c2a

Browse files
xing-yangxing-yang
committed
Add a blog for volume group snapshots beta
1 parent c2f1dd9 commit 19e2c2a

File tree

1 file changed

+382
-0
lines changed

1 file changed

+382
-0
lines changed

‎content/en/blog/_posts/2024-12-11-volume-group-snapshot-beta.md

Lines changed: 382 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,382 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes 1.32: Moving Volume Group Snapshots to Beta"
4+
date: 2024-12-11
5+
slug: kubernetes-1-32-volume-group-snapshot-beta
6+
draft: true
7+
author: >
8+
Xing Yang (VMware by Broadcom)
9+
---
10+
11+
Volume group snapshots were [introduced](/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/)
12+
as an Alpha feature with the Kubernetes 1.27 release.
13+
The recent release of Kubernetes v1.32 moved that support to **beta**.
14+
The support for volume group snapshots relies on a set of
15+
[extension APIs for group snapshots](https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html#volume-group-snapshot-apis).
16+
These APIs allow users to take crash consistent snapshots for a set of volumes.
17+
Behind the scenes, Kubernetes uses a label selector to group multiple PersistentVolumeClaims
18+
for snapshotting.
19+
A key aim is to allow you restore that set of snapshots to new volumes and
20+
recover your workload based on a crash consistent recovery point.
21+
22+
This new feature is only supported for [CSI](https://kubernetes-csi.github.io/docs/) volume drivers.
23+
24+
## An overview of volume group snapshots
25+
26+
Some storage systems provide the ability to create a crash consistent snapshot of
27+
multiple volumes. A group snapshot represents _copies_ made from multiple volumes, that
28+
are taken at the same point-in-time. A group snapshot can be used either to rehydrate
29+
new volumes (pre-populated with the snapshot data) or to restore existing volumes to
30+
a previous state (represented by the snapshots).
31+
32+
## Why add volume group snapshots to Kubernetes?
33+
34+
The Kubernetes volume plugin system already provides a powerful abstraction that
35+
automates the provisioning, attaching, mounting, resizing, and snapshotting of block
36+
and file storage.
37+
38+
Underpinning all these features is the Kubernetes goal of workload portability:
39+
Kubernetes aims to create an abstraction layer between distributed applications and
40+
underlying clusters so that applications can be agnostic to the specifics of the
41+
cluster they run on and application deployment requires no cluster specific knowledge.
42+
43+
There was already a [VolumeSnapshot](/docs/concepts/storage/volume-snapshots/) API
44+
that provides the ability to take a snapshot of a persistent volume to protect against
45+
data loss or data corruption. However, there are other snapshotting functionalities
46+
not covered by the VolumeSnapshot API.
47+
48+
Some storage systems support consistent group snapshots that allow a snapshot to be
49+
taken from multiple volumes at the same point-in-time to achieve write order consistency.
50+
This can be useful for applications that contain multiple volumes. For example,
51+
an application may have data stored in one volume and logs stored in another volume.
52+
If snapshots for the data volume and the logs volume are taken at different times,
53+
the application will not be consistent and will not function properly if it is restored
54+
from those snapshots when a disaster strikes.
55+
56+
It is true that you can quiesce the application first, take an individual snapshot from
57+
each volume that is part of the application one after the other, and then unquiesce the
58+
application after all the individual snapshots are taken. This way, you would get
59+
application consistent snapshots.
60+
61+
However, sometimes the application quiesce can be so time consuming that you want to do it less frequently,
62+
or it may not be possible to quiesce an application at all.
63+
For example, a user may want to run weekly backups with application quiesce
64+
and nightly backups without application quiesce but with consistent group support which
65+
provides crash consistency across all volumes in the group.
66+
67+
## Kubernetes APIs for volume group snapshots
68+
69+
Kubernetes' support for _volume group snapshots_ relies on three API kinds that
70+
are used
71+
for managing snapshots:
72+
73+
VolumeGroupSnapshot
74+
: Created by a Kubernetes user (or perhaps by your own automation) to request
75+
creation of a volume group snapshot for multiple persistent volume claims.
76+
It contains information about the volume group snapshot operation such as the
77+
timestamp when the volume group snapshot was taken and whether it is ready to use.
78+
The creation and deletion of this object represents a desire to create or delete a
79+
cluster resource (a group snapshot).
80+
81+
VolumeGroupSnapshotContent
82+
: Created by the snapshot controller for a dynamically created VolumeGroupSnapshot.
83+
It contains information about the volume group snapshot including the volume group
84+
snapshot ID.
85+
This object represents a provisioned resource on the cluster (a group snapshot).
86+
The VolumeGroupSnapshotContent object binds to the VolumeGroupSnapshot for which it
87+
was created with a one-to-one mapping.
88+
89+
VolumeGroupSnapshotClass
90+
: Created by cluster administrators to describe how volume group snapshots should be
91+
created, including the driver information, the deletion policy, etc.
92+
93+
These three API kinds are defined as
94+
[CustomResourceDefinitions](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
95+
(CRDs).
96+
These CRDs must be installed in a Kubernetes cluster for a CSI Driver to support
97+
volume group snapshots.
98+
99+
## What components are needed to support volume group snapshots
100+
101+
Volume group snapshots are implemented in the
102+
[external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter) repository.
103+
Implementing volume group snapshots meant adding or changing several components:
104+
105+
* Added new CustomResourceDefinitions for VolumeGroupSnapshot and two supporting APIs.
106+
* Volume group snapshot controller logic is added to the common snapshot controller.
107+
* Adding logic to make CSI calls into the snapshotter sidecar controller.
108+
109+
The volume snapshot controller and CRDs are deployed once per
110+
cluster, while the sidecar is bundled with each CSI driver.
111+
112+
Therefore, it makes sense to deploy the volume snapshot controller and CRDs as a cluster addon.
113+
114+
The Kubernetes project recommends that Kubernetes distributors
115+
bundle and deploy the volume snapshot controller and CRDs as part
116+
of their Kubernetes cluster management process (independent of any CSI Driver).
117+
118+
## What's new in Beta?
119+
120+
* The VolumeGroupSnapshot feature in CSI spec moved to GA in the [v1.11.0 release](https://github.com/container-storage-interface/spec/releases/tag/v1.11.0).
121+
122+
* The snapshot validation webhook was deprecated in external-snapshotter v8.0.0 and it is now removed.
123+
Most of the validation webhook logic was added as validation rules into the CRDs.
124+
Minimum required Kubernetes version is 1.25 for these validation rules.
125+
One thing in the validation webhook not moved to CRDs is the prevention of creating
126+
multiple default volume snapshot classes and multiple default volume group snapshot classes
127+
for the same CSI driver.
128+
With the removal of the validation webhook, an error will still be raised when dynamically
129+
provisioning a VolumeSnapshot or VolumeGroupSnapshot when multiple default volume snapshot
130+
classes or multiple default volume group snapshot classes for the same CSI driver exist.
131+
132+
* The `enable-volumegroup-snapshot` flag in the snapshot-controller and the CSI snapshotter
133+
sidecar has been replaced by a feature gate.
134+
Since VolumeGroupSnapshot is a new API, the feature moves to Beta but the feature gate is
135+
disabled by default.
136+
To use this feature, enable the feature gate by adding the flag `--feature-gates=CSIVolumeGroupSnapshot=true`
137+
when starting the snapshot-controller and the CSI snapshotter sidecar.
138+
139+
* The logic to dynamically create the VolumeGroupSnapshot and its corresponding individual
140+
VolumeSnapshot and VolumeSnapshotContent objects are moved from the CSI snapshotter to the common
141+
snapshot-controller.
142+
New RBAC rules are added to the common snapshot-controller and some RBAC rules are removed from
143+
the CSI snapshotter sidecar accordingly.
144+
145+
## How do I use Kubernetes volume group snapshots
146+
147+
### Creating a new group snapshot with Kubernetes
148+
149+
Once a VolumeGroupSnapshotClass object is defined and you have volumes you want to
150+
snapshot together, you may request a new group snapshot by creating a VolumeGroupSnapshot
151+
object.
152+
153+
The source of the group snapshot specifies whether the underlying group snapshot
154+
should be dynamically created or if a pre-existing VolumeGroupSnapshotContent
155+
should be used.
156+
157+
A pre-existing VolumeGroupSnapshotContent is created by a cluster administrator.
158+
It contains the details of the real volume group snapshot on the storage system which
159+
is available for use by cluster users.
160+
161+
One of the following members in the source of the group snapshot must be set.
162+
163+
* `selector` - a label query over PersistentVolumeClaims that are to be grouped
164+
together for snapshotting. This selector will be used to match the label
165+
added to a PVC.
166+
* `volumeGroupSnapshotContentName` - specifies the name of a pre-existing
167+
VolumeGroupSnapshotContent object representing an existing volume group snapshot.
168+
169+
#### Dynamically provision a group snapshot
170+
171+
In the following example, there are two PVCs.
172+
173+
```console
174+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
175+
pvc-0 Bound pvc-6e1f7d34-a5c5-4548-b104-01e72c72b9f2 100Mi RWO csi-hostpath-sc <unset> 2m15s
176+
pvc-1 Bound pvc-abc640b3-2cc1-4c56-ad0c-4f0f0e636efa 100Mi RWO csi-hostpath-sc <unset> 2m7s
177+
```
178+
179+
Label the PVCs.
180+
```console
181+
% kubectl label pvc pvc-0 group=myGroup
182+
persistentvolumeclaim/pvc-0 labeled
183+
184+
% kubectl label pvc pvc-1 group=myGroup
185+
persistentvolumeclaim/pvc-1 labeled
186+
```
187+
188+
For dynamic provisioning, a selector must be set so that the snapshot controller can find PVCs
189+
with the matching labels to be snapshotted together.
190+
191+
```yaml
192+
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
193+
kind: VolumeGroupSnapshot
194+
metadata:
195+
name: snapshot-daily-20241217
196+
namespace: demo-namespace
197+
spec:
198+
volumeGroupSnapshotClassName: csi-groupSnapclass
199+
source:
200+
selector:
201+
matchLabels:
202+
group: myGroup
203+
```
204+
205+
In the VolumeGroupSnapshot spec, a user can specify the VolumeGroupSnapshotClass which
206+
has the information about which CSI driver should be used for creating the group snapshot.
207+
A VolumGroupSnapshotClass is required for dynamic provisioning.
208+
209+
```yaml
210+
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
211+
kind: VolumeGroupSnapshotClass
212+
metadata:
213+
name: csi-groupSnapclass
214+
annotations:
215+
kubernetes.io/description: "Example group snapshot class"
216+
driver: example.csi.k8s.io
217+
deletionPolicy: Delete
218+
```
219+
220+
As a result of the volume group snapshot creation, a corresponding VolumeGroupSnapshotContent
221+
object will be created with a volumeGroupSnapshotHandle pointing to a resource on the storage
222+
system.
223+
224+
Two individual volume snapshots will be created as part of the volume group snapshot creation.
225+
226+
```console
227+
NAME READYTOUSE SOURCEPVC RESTORESIZE SNAPSHOTCONTENT AGE
228+
snapshot-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0 true pvc-0 100Mi snapcontent-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0 16m
229+
snapshot-da577d76bd2106c410616b346b2e72440f6ec7b12a75156263b989192b78caff true pvc-1 100Mi snapcontent-da577d76bd2106c410616b346b2e72440f6ec7b12a75156263b989192b78caff 16m
230+
```
231+
232+
#### Importing an existing group snapshot with Kubernetes
233+
234+
To import a pre-existing volume group snapshot into Kubernetes, you must also import
235+
the corresponding individual volume snapshots.
236+
237+
Identify the individual volume snapshot handles, manually construct a
238+
VolumeSnapshotContent object first, then create a VolumeSnapshot object pointing to
239+
the VolumeSnapshotContent object. Repeat this for every individual volume snapshot.
240+
241+
Then manually create a VolumeGroupSnapshotContent object, specifying the
242+
volumeGroupSnapshotHandle and individual volumeSnapshotHandles already existing
243+
on the storage system.
244+
245+
```yaml
246+
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
247+
kind: VolumeGroupSnapshotContent
248+
metadata:
249+
name: static-group-content
250+
spec:
251+
deletionPolicy: Delete
252+
driver: hostpath.csi.k8s.io
253+
source:
254+
groupSnapshotHandles:
255+
volumeGroupSnapshotHandle: e8779136-a93e-11ef-9549-66940726f2fd
256+
volumeSnapshotHandles:
257+
- e8779147-a93e-11ef-9549-66940726f2fd
258+
- e8783cd0-a93e-11ef-9549-66940726f2fd
259+
volumeGroupSnapshotRef:
260+
name: static-group-snapshot
261+
namespace: demo-namespace
262+
```
263+
264+
After that create a VolumeGroupSnapshot object pointing to the VolumeGroupSnapshotContent
265+
object.
266+
267+
```yaml
268+
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
269+
kind: VolumeGroupSnapshot
270+
metadata:
271+
name: static-group-snapshot
272+
namespace: demo-namespace
273+
spec:
274+
source:
275+
volumeGroupSnapshotContentName: static-group-content
276+
```
277+
278+
### How to use group snapshot for restore in Kubernetes
279+
280+
At restore time, the user can request a new PersistentVolumeClaim to be created from
281+
a VolumeSnapshot object that is part of a VolumeGroupSnapshot. This will trigger
282+
provisioning of a new volume that is pre-populated with data from the specified
283+
snapshot. The user should repeat this until all volumes are created from all the
284+
snapshots that are part of a group snapshot.
285+
286+
```yaml
287+
apiVersion: v1
288+
kind: PersistentVolumeClaim
289+
metadata:
290+
name: examplepvc-restored-2024-12-17
291+
namespace: demo-namespace
292+
spec:
293+
storageClassName: example-foo-nearline
294+
dataSource:
295+
name: snapshot-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0
296+
kind: VolumeSnapshot
297+
apiGroup: snapshot.storage.k8s.io
298+
accessModes:
299+
- ReadWriteOncePod
300+
resources:
301+
requests:
302+
storage: 100Mi # must be enough storage to fit the existing snapshot
303+
```
304+
305+
## As a storage vendor, how do I add support for group snapshots to my CSI driver?
306+
307+
To implement the volume group snapshot feature, a CSI driver **must**:
308+
309+
* Implement a new group controller service.
310+
* Implement group controller RPCs: `CreateVolumeGroupSnapshot`, `DeleteVolumeGroupSnapshot`, and `GetVolumeGroupSnapshot`.
311+
* Add group controller capability `CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT`.
312+
313+
See the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md)
314+
and the [Kubernetes-CSI Driver Developer Guide](https://kubernetes-csi.github.io/docs/)
315+
for more details.
316+
317+
As mentioned earlier, it is strongly recommended that Kubernetes distributors
318+
bundle and deploy the volume snapshot controller and CRDs as part
319+
of their Kubernetes cluster management process (independent of any CSI Driver).
320+
321+
As part of this recommended deployment process, the Kubernetes team provides a number of
322+
sidecar (helper) containers, including the
323+
[external-snapshotter sidecar container](https://kubernetes-csi.github.io/docs/external-snapshotter.html)
324+
which has been updated to support volume group snapshot.
325+
326+
The external-snapshotter watches the Kubernetes API server for
327+
VolumeGroupSnapshotContent objects, and triggers `CreateVolumeGroupSnapshot` and
328+
`DeleteVolumeGroupSnapshot` operations against a CSI endpoint.
329+
330+
## What are the limitations?
331+
332+
The beta implementation of volume group snapshots for Kubernetes has the following limitations:
333+
334+
* Does not support reverting an existing PVC to an earlier state represented by
335+
a snapshot (only supports provisioning a new volume from a snapshot).
336+
* No application consistency guarantees beyond any guarantees provided by the storage system
337+
(e.g. crash consistency). See this [doc](https://github.com/kubernetes/community/blob/30d06f49fba22273f31b3c616b74cf8745c19b3d/wg-data-protection/data-protection-workflows-white-paper.md#quiesce-and-unquiesce-hooks)
338+
for more discussions on application consistency.
339+
340+
## What’s next?
341+
342+
Depending on feedback and adoption, the Kubernetes project plans to push the volume
343+
group snapshot implementation to general availability (GA) in a future release.
344+
345+
## How can I learn more?
346+
347+
- The [design spec](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot)
348+
for the volume group snapshot feature.
349+
- The [code repository](https://github.com/kubernetes-csi/external-snapshotter) for volume group
350+
snapshot APIs and controller.
351+
- CSI [documentation](https://kubernetes-csi.github.io/docs/) on the group snapshot feature.
352+
353+
## How do I get involved?
354+
355+
This project, like all of Kubernetes, is the result of hard work by many contributors
356+
from diverse backgrounds working together. On behalf of SIG Storage, I would like to
357+
offer a huge thank you to the contributors who stepped up these last few quarters
358+
to help the project reach beta:
359+
360+
* Ben Swartzlander ([bswartz](https://github.com/bswartz))
361+
* Cici Huang ([cici37](https://github.com/cici37))
362+
* Hemant Kumar ([gnufied](https://github.com/gnufied))
363+
* James Defelice ([jdef](https://github.com/jdef))
364+
* Jan Šafránek ([jsafrane](https://github.com/jsafrane))
365+
* Madhu Rajanna ([Madhu-1](https://github.com/Madhu-1))
366+
* Manish M Yathnalli ([manishym](https://github.com/manishym))
367+
* Michelle Au ([msau42](https://github.com/msau42))
368+
* Niels de Vos ([nixpanic](https://github.com/nixpanic))
369+
* Leonardo Cecchi ([leonardoce](https://github.com/leonardoce))
370+
* Rakshith R ([Rakshith-R](https://github.com/Rakshith-R))
371+
* Raunak Shah ([RaunakShah](https://github.com/RaunakShah))
372+
* Saad Ali ([saad-ali](https://github.com/saad-ali))
373+
* Xing Yang ([xing-yang](https://github.com/xing-yang))
374+
* Yati Padia ([yati1998](https://github.com/yati1998))
375+
376+
For those interested in getting involved with the design and development of CSI or
377+
any part of the Kubernetes Storage system, join the
378+
[Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG).
379+
We always welcome new contributors.
380+
381+
We also hold regular [Data Protection Working Group meetings](https://github.com/kubernetes/community/tree/master/wg-data-protection).
382+
New attendees are welcome to join our discussions.

0 commit comments

Comments
 (0)