You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -80,7 +83,7 @@ The rest of the document includes required information missing from the original
80
83
* Snapshot Object in Use Protection
81
84
* Separate the common controller from the sidecar controller
82
85
* Add secrets field to list-snapshots RPC in the CSI spec. Add “data-source-secret” in create-volume intended for accessing the data source. Implement them in external-snapshotter and external-provisioner.
83
-
* Add metrics support
86
+
* Add metrics support to the external-snapshotter sidecar.
84
87
* Unit and e2e tests implemented
85
88
* Update snapshot CRDs to v1beta1 and enable VolumeSnapshotDataSource feature gate by default.
86
89
@@ -89,28 +92,150 @@ The rest of the document includes required information missing from the original
89
92
* Snapshot feature is used as a basic building block in other advanced applications.
90
93
* Feature deployed in production and have gone through at least one K8s upgrade.
91
94
92
-
## Changes
93
-
94
-
### Changes Implemented
95
+
## Snapshot Beta
96
+
97
+
### API Changes
98
+
99
+
A number of changes were made to the Kubernetes volume snapshot API between alpha to beta. These changes are not backward compatible and the alpha API is no longer supported. The purpose of these changes was to make API definitions clear and easier to use.
100
+
101
+
The following changes are made:
102
+
103
+
* DeletionPolicy is now a required field rather than optional in both VolumeSnapshotClass and VolumeSnapshotContent. This way the user has to explicitly specify it, leaving no room for confusion.
104
+
* VolumeSnapshotSpec has a new required Source field. Source may be either a PersistentVolumeClaimName (if dynamically provisioning a snapshot) or VolumeSnapshotContentName (if pre-provisioning a snapshot).
105
+
* VolumeSnapshotContentSpec also has a new required Source field. This Source may be either a VolumeHandle (if dynamically provisioning a snapshot) or a SnapshotHandle (if pre-provisioning volume snapshots).
106
+
* VolumeSnapshotStatus now contains a BoundVolumeSnapshotContentName to indicate the VolumeSnapshot object is bound to a VolumeSnapshotContent.
107
+
* VolumeSnapshotContentnow contains a Status to indicate the current state of the content. It has a field SnapshotHandle to indicate that the VolumeSnapshotContent represents a snapshot on the storage system.
108
+
109
+
The beta VolumeSnapshot API object is as follows:
110
+
111
+
```
112
+
type VolumeSnapshot struct {
113
+
metav1.TypeMeta
114
+
metav1.ObjectMeta
115
+
116
+
Spec VolumeSnapshotSpec
117
+
Status *VolumeSnapshotStatus
118
+
}
119
+
```
120
+
121
+
```
122
+
type VolumeSnapshotSpec struct {
123
+
Source VolumeSnapshotSource
124
+
VolumeSnapshotClassName *string
125
+
}
126
+
// Exactly one of its members MUST be specified
127
+
type VolumeSnapshotSource struct {
128
+
// +optional
129
+
PersistentVolumeClaimName *string
130
+
// +optional
131
+
VolumeSnapshotContentName *string
132
+
}
133
+
```
134
+
135
+
```
136
+
type VolumeSnapshotStatus struct {
137
+
BoundVolumeSnapshotContentName *string
138
+
CreationTime *metav1.Time
139
+
ReadyToUse *bool
140
+
RestoreSize *resource.Quantity
141
+
Error *VolumeSnapshotError
142
+
}
143
+
```
144
+
145
+
The beta VolumeSnapshotContent API object is as follows:
146
+
147
+
```
148
+
type VolumeSnapshotContent struct {
149
+
metav1.TypeMeta
150
+
metav1.ObjectMeta
151
+
152
+
Spec VolumeSnapshotContentSpec
153
+
Status *VolumeSnapshotContentStatus
154
+
}
155
+
```
156
+
157
+
```
158
+
type VolumeSnapshotContentSpec struct {
159
+
VolumeSnapshotRef core_v1.ObjectReference
160
+
Source VolumeSnapshotContentSource
161
+
DeletionPolicy DeletionPolicy
162
+
Driver string
163
+
VolumeSnapshotClassName *string
164
+
}
165
+
```
166
+
167
+
```
168
+
type VolumeSnapshotContentSource struct {
169
+
// +optional
170
+
VolumeHandle *string
171
+
// +optional
172
+
SnapshotHandle *string
173
+
}
174
+
```
175
+
176
+
```
177
+
type VolumeSnapshotContentStatus struct {
178
+
CreationTime *int64
179
+
ReadyToUse *bool
180
+
RestoreSize *int64
181
+
Error *VolumeSnapshotError
182
+
SnapshotHandle *string
183
+
}
184
+
```
185
+
186
+
The beta Kubernetes VolumeSnapshotClass API object is as follows:
187
+
188
+
```
189
+
type VolumeSnapshotClass struct {
190
+
metav1.TypeMeta
191
+
metav1.ObjectMeta
192
+
193
+
Driver string
194
+
Parameters map[string]string
195
+
DeletionPolicy DeletionPolicy
196
+
}
197
+
```
198
+
199
+
### Controller Split
200
+
201
+
When Volume Snapshot is promoted to Beta in Kubernetes 1.17, the CSI external-snapshotter sidecar controller is split into two controllers: a snapshot-controller and a CSI external-snapshotter sidecar.
202
+
203
+
The snapshot controller will be watching the Kubernetes API server for `VolumeSnapshot` and `VolumeSnapshotContent` CRD objects. The CSI `external-snapshotter` sidecar only watches the Kubernetes API server for `VolumeSnapshotContent` CRD objects.
204
+
205
+
The creation of a new `VolumeSnapshot` object referencing a `SnapshotClass` CRD object corresponding to this driver causes the snapshot controller to trigger the creation of a Kubernetes `VolumeSnapshotContent` object to represent the to-be-created new snapshot.
206
+
207
+
The creation of a new `VolumeSnapshotContent` object causes the sidecar container to trigger a `CreateSnapshot` operation against the specified CSI endpoint to provision a new snapshot. When a new snapshot is successfully provisioned, the sidecar container updates the status field of the `VolumeSnapshotContent` object to represent the new snapshot.
208
+
209
+
The snapshot controller will be updating the status field of the `VolumeSnapshot` object accordingly based on the status field of the `VolumeSnapshotContent` object to indicate the new snapshot is ready to be used.
210
+
211
+
The deletion event of a `VolumeSnapshot` object bound to a `VolumeSnapshotContent` corresponding to this driver with a `delete` deletion policy causes the snapshot controller to start deleting the `VolumeSnapshotContent` object and add an annotation to the object to indicate it is being deleted. Note that both the `VolumeSnapshot` object and the `VolumeSnapshotContent` object will not be deleted immediately due to the finalizers. When the sidecar container detects this update on the `VolumeSnapshotContent` object, it triggers a `DeleteSnapshot` operation against the specified CSI endpoint to delete the snapshot. Once the snapshot is successfully deleted, the sidecar container removes the finalizer on the `VolumeSnapshotContent` object which leads to the deletion of the object from Kubernetes. The snapshot controller then removes the finalizer on the `VolumeSnapshot` object and as a result the object will be deleted from Kubernetes.
212
+
213
+
### Other Changes Implemented
95
214
96
215
Here are the changes since the original design proposal:
97
216
98
217
* Renamed `Ready` to `ReadyToUse` in the `Status` field of `VolumeSnapshot` API object.
99
218
* Changed type of `RestoreSize` in `CSIVolumeSnapshotSource` from `*resource.Quantity` to `*int64`.
100
219
* Lease based Leader Election support is added.
101
220
* Added `VolumeSnapshotContent` deletion policy which is also specified in `VolumeSnapshotClass`.
102
-
* Added `VolumeSnapshot` and `VolumeSnapshotContent` in Use Protection using Finalizers.
103
221
* Added Finalizer on the snapshot source PVC to prevent it from being deleted when a snapshot is being created from it.
222
+
* Added Finalizer on the `VolumeSnapshotContent` object to prevent it from being deleted when it is bound to the `VolumeSnapshot` object.
223
+
* Added Finalizer on the `VolumeSnapshot` object to prevent it from being deleted when it is being used as a source to create a PVC.
224
+
* Added Finalizer on the `VolumeSnapshot` object to prevent it from being deleted when it is bound to the `VolumeSnapshotContent` object.
104
225
* Added check to see whether ListSnapshots is supported by the CSI driver. If it is supported, ListSnapshots will be called to find out the status of a snapshot during static binding; otherwise it is assumed the snapshot ID provided by the admin is valid.
226
+
* Added deletion secret as annotation to volume snapshot content.
227
+
* Added prometheus metrics to CSI external-snapshotter under the /metrics endpoint.
228
+
* Removed createSnapshotContentRetryCount and createSnapshotContentInterval
229
+
from command line options.
230
+
* Added a prefix "external-snapshotter-leader" and the driver name to the snapshotter leader election lock name. Rolling update from an earlier version to v2.0.0 will not work if leader election is enabled because the lock name is changed in v2.0.0.
105
231
106
232
### Work in Progress
107
233
108
-
There are other things we are working on for Beta:
234
+
There are other things we are working on before promoting the snapshot feature to GA:
109
235
110
236
* If snapshot creation times out, VolumeSnapshot status will not be marked as failed so that controller will continue to retry to create until the operation either succeeds or fails. It is up to the user or an upper level application that uses the VolumeSnapshot to determine what to do with the snapshot. This work is on-going.
111
-
* Investigation is on-going to determine whether we should separate common controller logic from other logic that belongs to the sidecar. Can be in the same external-snapshotter repo. The common controller should not be deployed with the driver. It should be deployed by the cluster deployer, or we can provide a way to deploy it as a separate Statefulset, not together with the driver. CRD installation should also be done by the cluster deployer.
112
237
* Add secrets field to list-snapshots RPC in the CSI spec. Add “data-source-secret” in create-volume intended for accessing the data source. Implement them in external-snapshotter and external-provisioner.
113
-
* Add metrics support in the snapshot controller.
238
+
* Add metrics support in the snapshot controller (metrics is already added to the external-snapshotter sidecar).
0 commit comments