You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+47-27Lines changed: 47 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,39 +4,33 @@
4
4
5
5
The CSI snapshotter is part of Kubernetes implementation of [Container Storage Interface (CSI)](https://github.com/container-storage-interface/spec).
6
6
7
-
The volume snapshot feature supports CSI v1.0 and higher. It was introduced as an Alpha feature in Kubernetes v1.12 and has been promoted to an Beta feature in Kubernetes 1.17.
7
+
The volume snapshot feature supports CSI v1.0 and higher. It was introduced as an Alpha feature in Kubernetes v1.12 and has been promoted to an Beta feature in Kubernetes 1.17. In Kubernetes 1.20, the volume snapshot feature moves to GA.
8
8
9
9
> :warning:**WARNING**: There is a new validating webhook server which provides tightened validation on snapshot objects. This SHOULD be installed by all users of this feature. More details [below](#validating-webhook).
10
10
11
11
12
12
## Overview
13
13
14
-
With the promotion of Volume Snapshot to beta, the feature is now enabled by default on standard Kubernetes deployments instead of being opt-in.
14
+
With the promotion of Volume Snapshot to GA, the feature is enabled by default on standard Kubernetes deployments and cannot be turned off.
15
15
16
-
The move of the Kubernetes Volume Snapshot feature to beta also means:
17
-
* A revamp of volume snapshot APIs.
18
-
* The CSI external-snapshotter sidecar is split into two controllers, a snapshot controller and a CSI external-snapshotter sidecar.
19
-
20
-
The snapshot controller is deployed by the Kubernetes distributions and is responsible for watching the VolumeSnapshot CRD objects and manages the creation and deletion lifecycle of snapshots.
21
-
22
-
The CSI external-snapshotter sidecar watches Kubernetes VolumeSnapshotContent CRD objects and triggers CreateSnapshot/DeleteSnapshot against a CSI endpoint.
23
-
24
-
Blog post for the beta feature can be found [here](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-cis-volume-snapshot-beta)
16
+
Blog post for the GA feature can be found [here](https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/)
25
17
26
18
27
19
## Compatibility
28
20
29
21
This information reflects the head of this branch.
30
22
31
-
| Compatible with CSI Version | Container Image |[Min K8s Version](https://kubernetes-csi.github.io/docs/kubernetes-compatibility.html#minimum-version)| Snapshot CRD version |
Note: snapshot-controller, snapshot-validation-webhook, csi-snapshotter v4.0 requires v1 snapshot CRDs to be installed, but it supports both v1 and v1beta1 snapshot objects.
36
30
37
31
## Feature Status
38
32
39
-
The `VolumeSnapshotDataSource` feature gate was introduced in Kubernetes 1.12 and it is enabled by default in Kubernetes 1.17 when the volume snapshot feature is promoted to beta.
33
+
The `VolumeSnapshotDataSource` feature gate was introduced in Kubernetes 1.12 and it is enabled by default in Kubernetes 1.17 when the volume snapshot feature is promoted to beta. In Kubernetes 1.20, the feature gate is enabled by default on standard Kubernetes deployments and cannot be turned off.
40
34
41
35
42
36
## Design
@@ -47,18 +41,21 @@ The CSI external-snapshotter sidecar only watches for `VolumeSnapshotContent` cr
47
41
48
42
The CSI external-snapshotter sidecar talks to CSI over socket (/run/csi/socket by default, configurable by -csi-address).
49
43
50
-
### Hightlights in the snapshot v1beta1 APIs
44
+
### Snapshot v1 APIs
51
45
52
-
* DeletionPolicy is a required field in both VolumeSnapshotClass and VolumeSnapshotContent. This way the user has to explicitly specify it, leaving no room for confusion.
53
-
* VolumeSnapshotSpec has a required Source field. Source may be either a PersistentVolumeClaimName (if dynamically provisioning a snapshot) or VolumeSnapshotContentName (if pre-provisioning a snapshot).
54
-
* VolumeSnapshotContentSpec has a required Source field. This Source may be either a VolumeHandle (if dynamically provisioning a snapshot) or a SnapshotHandle (if pre-provisioning volume snapshots).
55
-
* VolumeSnapshot contains a Status to indicate the current state of the volume snapshot. It has a field BoundVolumeSnapshotContentName to indicate the VolumeSnapshot object is bound to a VolumeSnapshotContent.
56
-
* VolumeSnapshotContent contains a Status to indicate the current state of the volume snapshot content. It has a field SnapshotHandle to indicate that the VolumeSnapshotContent represents a snapshot on the storage system.
46
+
Other than introducing tightening validation, there is no difference between the v1beta1 and v1 Kubernetes volume snapshot API. In the current release, both v1 and v1beta1 are served while the stored API version is still v1beta1. Future releases will switch the stored version to v1 and gradually remove v1beta1 support.
57
47
58
48
59
49
## Usage
60
50
61
-
The Volume Snapshot feature now depends on a new, volume snapshot controller in addition to the volume snapshot CRDs. Both the volume snapshot controller and the CRDs are independent of any CSI driver. Regardless of the number CSI drivers deployed on the cluster, there must be only one instance of the volume snapshot controller running and one set of volume snapshot CRDs installed per cluster.
51
+
Volume Snapshot feature contains the following components:
* CSI Driver along with [CSI Snapshotter sidecar](https://github.com/kubernetes-csi/external-snapshotter/tree/master/pkg/sidecar-controller)
57
+
58
+
The Volume Snapshot feature depends on a volume snapshot controller and the volume snapshot CRDs. Both the volume snapshot controller and the CRDs are independent of any CSI driver. Regardless of the number CSI drivers deployed on the cluster, there must be only one instance of the volume snapshot controller running and one set of volume snapshot CRDs installed per cluster.
62
59
63
60
Therefore, it is strongly recommended that Kubernetes distributors bundle and deploy the controller and CRDs as part of their Kubernetes cluster management process (independent of any CSI Driver).
64
61
@@ -87,6 +84,22 @@ Install CSI Driver:
87
84
88
85
The snapshot validating webhook is an HTTP callback which responds to [admission requests](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). It is part of a larger [plan](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1900-volume-snapshot-validation-webhook) to tighten validation for volume snapshot objects. This webhook introduces the [ratcheting validation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1900-volume-snapshot-validation-webhook#backwards-compatibility) mechanism targeting the tighter validation. The cluster admin or Kubernetes distribution admin should install the webhook alongside the snapshot controllers and CRDs.
89
86
87
+
Along with the validation webhook, the volume snapshot controller will start [labeling](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1900-volume-snapshot-validation-webhook#automatic-labelling-of-invalid-objects) invalid snapshot objects which already existed. This is to enable quick identification of invalid snapshot objects in the system by running:
88
+
```
89
+
kubectl get volumesnapshots --selector=snapshot.storage.kubernetes.io/invalid-snapshot-resource: ""
90
+
kubectl get volumesnapshotcontents --selector=snapshot.storage.kubernetes.io/invalid-snapshot-content-resource: ""
91
+
```
92
+
93
+
Users should run this to identify, remove any invalid objects, and correct their workflows before upgrading to v1. Once the API has been switched to the v1 type, those invalid objects will not be deletable from the system.
94
+
95
+
If there are no existing invalid v1beta1 objects, after upgrading to v1, the webhook and schema validation will prevent the user from creating new invalid v1 and v1beta1 objects.
96
+
97
+
If there are existing invalid v1beta1 objects, the user should install the validation webhook before upgrading to v1 so that those invalid objects will be labeled and can be identified easily and removed before upgrading to v1.
98
+
99
+
If there are existing invalid v1beta1 objects and the user didn't install the validation webhook before upgrading to v1, those existing invalid v1beta1 objects will not be labeled by the snapshot controller.
100
+
101
+
So the recommendation is that before upgrading to v1, the user should install the validation webhook so that all existing invalid objects will be labeled and can be easily identified and deleted.
102
+
90
103
> :warning:**WARNING**: Cluster admins choosing not to install the webhook server and participate in the phased release process can cause future problems when upgrading from `v1beta1` to `v1` volumesnapshot API, if there are currently persisted objects which fail the new stricter validation. Potential impacts include being unable to delete invalid snapshot objects.
91
104
92
105
Read more about how to install the example webhook [here](deploy/kubernetes/webhook-example/README.md).
@@ -157,7 +170,9 @@ The external-snapshotter optionally exposes an HTTP endpoint at address:port spe
157
170
158
171
* Leader election health check at `/healthz/leader-election`. It is recommended to run a liveness probe against this endpoint when leader election is used to kill external-provisioner leader that fails to connect to the API server to renew its leadership. See https://github.com/kubernetes-csi/csi-lib-utils/issues/66 for details.
159
172
160
-
## Upgrade from v1alpha1 to v1beta1
173
+
## Upgrade
174
+
175
+
### Upgrade from v1alpha1 to v1beta1
161
176
162
177
The change from v1alpha1 to v1beta1 snapshot APIs is not backward compatible.
163
178
@@ -167,6 +182,11 @@ If you have already deployed v1alpha1 snapshot APIs and external-snapshotter sid
Validation webhook should be installed before upgrading to v1. Potential impacts of not installing the validation webhook before upgrading to v1 include being unable to delete invalid snapshot objects. See the section on Validation Webhook for details.
188
+
189
+
Change from v1beta1 to v1 is backward compatible. Both v1 and v1beta1 are served while the stored API version is still v1beta1. Future releases will switch the stored version to v1 and gradually remove v1beta1 support.
170
190
171
191
## Testing
172
192
@@ -180,9 +200,9 @@ go test -timeout 30s github.com/kubernetes-csi/external-snapshotter/pkg/sidecar
180
200
181
201
## CRDs and Client Library
182
202
183
-
Volume snapshot APIs and client library are now in a separate sub-module: `github.com/kubernetes-csi/external-snapshotter/client/v3`.
203
+
Volume snapshot APIs and client library are now in a separate sub-module: `github.com/kubernetes-csi/external-snapshotter/client/v4`.
184
204
185
-
Use the command `go get -u github.com/kubernetes-csi/external-snapshotter/client/v3@v3.0.0` to get the client library.
205
+
Use the command `go get -u github.com/kubernetes-csi/external-snapshotter/client/v4@v4.0.0` to get the client library.
186
206
187
207
### Setting Quota limits with Snapshot custom resources
188
208
[`ResourceQuotas`](https://kubernetes.io/docs/concepts/policy/resource-quotas/) are namespaced objects that can be used to set limits on objects of a particular [`Group.Version.Kind`](https://book.kubebuilder.io/cronjob-tutorial/gvks.html). Before we set resource quota, make sure that snapshot CRDs are installed in the cluster. If not please follow [this guide](https://github.com/kubernetes-csi/external-snapshotter#usage).
0 commit comments