Skip to content

Commit 0c52080

Browse files
PrasadG193PrasadG193
committed
Add documentation for testing SnapshotMetadata service
Signed-off-by: Prasad Ghangal <[email protected]>
1 parent 6886d26 commit 0c52080

File tree

1 file changed

+268
-0
lines changed

1 file changed

+268
-0
lines changed

‎docs/example-snapshot-metadata.md

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
## Snapshot Changed Block Metadata Support
2+
3+
The CSI HostPath driver now includes support for the CSI [SnapshotMetadata](https://github.com/container-storage-interface/spec/blob/master/csi.proto#L130) service. This service provides APIs to retrieve metadata about the allocated blocks of a CSI VolumeSnapshot or the changed blocks between any two CSI VolumeSnapshot objects of the same PersistentVolume.
4+
5+
This document outlines the steps to test this feature on a Kubernetes cluster.
6+
7+
### Deploying CSI Hostpath driver with SnapshotMetadata service
8+
9+
Setting up CSI Hostpath driver with SnapshotMetadata service requires provisioning TLS certificates, creating TLS secrets, SnapshotMetadata custom resource, patching up csi-hostpathplugin deployments, etc. These steps are automated in `deploy.sh` script used to deploy CSI Hostpath driver.
10+
11+
Follow the steps below to deploy CSI Hostpath driver with SnapshotMetadata service:
12+
13+
a. Create `SnapshotMetadata` CRD
14+
15+
```
16+
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/client/config/crd/cbt.storage.k8s.io_snapshotmetadataservices.yaml
17+
```
18+
19+
b. Execute deploy script to setup hostpath plugin driver with external-snapshot-metadata change
20+
21+
```
22+
$ SNAPSHOT_METADATA_TESTS=true ./deploy/kubernetes-1.27/deploy.sh
23+
```
24+
25+
### Sample SnapshotMetadata client
26+
27+
The `SnapshotMetadata` service implements gRPC APIs. A gRPC client can query these APIs to retrieve metadata about the allocated blocks of a CSI VolumeSnapshot or the changed blocks between any two CSI VolumeSnapshot objects.
28+
29+
For our testing, we will be using a sample client implementation in Go provided as a example in [external-snapshot-metadata](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/examples/snapshot-metadata-lister) repo.
30+
31+
Follow the following steps to setup client with all the required permissions:
32+
33+
1. Setup RBAC
34+
35+
a. Create `ClusterRole` containing all the required permissions for the client
36+
37+
```bash
38+
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/deploy/snapshot-metadata-client-cluster-role.yaml
39+
```
40+
41+
b. Create a namespace to deploy client
42+
43+
```
44+
$ kubectl create namespace csi-client
45+
```
46+
47+
c. Create service account
48+
49+
```
50+
$ kubectl create serviceaccount csi-client-sa -n csi-client
51+
```
52+
53+
d. Bind the clusterrole to the service account
54+
55+
```
56+
$ kubectl create clusterrolebinding csi-client-cluster-role-binding --clusterrole=external-snapshot-metadata-client-runner --serviceaccount=csi-client:csi-client-sa
57+
```
58+
59+
2. Deploy sample client pod
60+
61+
Create a pod in which we'll be installing and running the client
62+
63+
```
64+
$ kubectl apply -f - <<EOF
65+
apiVersion: v1
66+
kind: Pod
67+
metadata:
68+
name: csi-client
69+
namespace: csi-client
70+
spec:
71+
containers:
72+
- name: golang
73+
image: golang:1.23.4
74+
command: ["tail", "-f", "/dev/null"]
75+
serviceAccountName: csi-client-sa
76+
EOF
77+
```
78+
79+
3. Install SnapshotMetadata client in the pod
80+
81+
Exec into the pod, clone the client repo and build the client from source
82+
83+
```
84+
$ kubectl exec -ti -n csi-client csi-client -- bash
85+
86+
## Install snapshot-metadata-lister
87+
root@csi-client:/go# go install github.com/kubernetes-csi/external-snapshot-metadata/examples/snapshot-metadata-lister@latest
88+
```
89+
90+
91+
This client performs following actions:
92+
1. Find Driver name for the snapshot.
93+
2. Discover `SnapshotMetadataService` resource for the driver which contains endpoint, audience and CA cert.
94+
3. Create SA Token with expected audience and permissions.
95+
4. Make gRPC call `GetMetadataAllocated` and `GetMetadataDelta` with appropriate params from `SnapshotMetadataService` resource, generated SA token.
96+
5. Stream response and print on console.
97+
98+
99+
### Test GetMetadataAllocated
100+
101+
1. Create CSI Hostpath storageclass
102+
103+
```
104+
$ kubectl create -f examples/csi-storageclass.yaml
105+
```
106+
107+
2. Create a volume with Block mode access
108+
109+
```
110+
kubectl apply -f - <<EOF
111+
kind: PersistentVolumeClaim
112+
apiVersion: v1
113+
metadata:
114+
name: pvc-raw
115+
spec:
116+
accessModes:
117+
- ReadWriteOnce
118+
storageClassName: csi-hostpath-sc
119+
volumeMode: Block
120+
resources:
121+
requests:
122+
storage: 10Mi
123+
EOF
124+
```
125+
126+
3. Mount the PVC to a pod
127+
128+
```
129+
kubectl apply -f - <<EOF
130+
apiVersion: v1
131+
kind: Pod
132+
metadata:
133+
name: pod-raw
134+
labels:
135+
name: busybox-test
136+
spec:
137+
restartPolicy: Always
138+
containers:
139+
- image: gcr.io/google_containers/busybox
140+
command: ["/bin/sh", "-c"]
141+
args: [ "tail -f /dev/null" ]
142+
name: busybox
143+
volumeDevices:
144+
- name: vol
145+
devicePath: /dev/loop3
146+
volumes:
147+
- name: vol
148+
persistentVolumeClaim:
149+
claimName: pvc-raw
150+
EOF
151+
```
152+
153+
4. Snapshot PVC `pvc-raw`
154+
155+
```
156+
kubectl apply -f - <<EOF
157+
apiVersion: snapshot.storage.k8s.io/v1
158+
kind: VolumeSnapshot
159+
metadata:
160+
name: raw-pvc-snap-1
161+
spec:
162+
volumeSnapshotClassName: csi-hostpath-snapclass
163+
source:
164+
persistentVolumeClaimName: pvc-raw
165+
EOF
166+
```
167+
168+
Wait for snapshot to be ready
169+
170+
```
171+
$ kg vs snapshot-1
172+
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
173+
snapshot-1 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-70b40b27-80d4-448b-bd9a-a87079c1a248 28s 29s
174+
```
175+
176+
5. Now, inside `csi-client` pod which is created in previous steps, use `snapshot-metadata-lister` tool query allocated blocks metadata
177+
178+
```
179+
$ kubectl exec -n csi-client csi-client -- snapshot-metadata-lister -n default -s raw-pvc-snap-1
180+
181+
Record# VolCapBytes BlockMetadataType ByteOffset SizeBytes
182+
------- -------------- ----------------- -------------- --------------
183+
1 10485760 FIXED_LENGTH 0 4096
184+
1 10485760 FIXED_LENGTH 4096 4096
185+
1 10485760 FIXED_LENGTH 8192 4096
186+
1 10485760 FIXED_LENGTH 12288 4096
187+
1 10485760 FIXED_LENGTH 16384 4096
188+
1 10485760 FIXED_LENGTH 20480 4096
189+
1 10485760 FIXED_LENGTH 24576 4096
190+
1 10485760 FIXED_LENGTH 28672 4096
191+
1 10485760 FIXED_LENGTH 32768 4096
192+
1 10485760 FIXED_LENGTH 36864 4096
193+
1 10485760 FIXED_LENGTH 40960 4096
194+
.
195+
.
196+
.
197+
.
198+
10 10485760 FIXED_LENGTH 10452992 4096
199+
10 10485760 FIXED_LENGTH 10457088 4096
200+
10 10485760 FIXED_LENGTH 10461184 4096
201+
10 10485760 FIXED_LENGTH 10465280 4096
202+
10 10485760 FIXED_LENGTH 10469376 4096
203+
10 10485760 FIXED_LENGTH 10473472 4096
204+
10 10485760 FIXED_LENGTH 10477568 4096
205+
10 10485760 FIXED_LENGTH 10481664 4096
206+
```
207+
208+
209+
### Test GetMetadataDelta
210+
211+
1. Change couple of blocks in the mounted device file in `pod-raw` Pod
212+
213+
```
214+
$ kubectl exec -ti pod-raw -- sh
215+
216+
### change blocks 12, 13, 15 and 20
217+
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=12 conv=notrunc
218+
1+0 records in
219+
1+0 records out
220+
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=13 conv=notrunc
221+
1+0 records in
222+
1+0 records out
223+
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=15 conv=notrunc
224+
1+0 records in
225+
1+0 records out
226+
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=20 conv=notrunc
227+
1+0 records in
228+
1+0 records out
229+
230+
```
231+
232+
2. Snapshot `pvc-raw` again
233+
234+
```
235+
kubectl apply -f - <<EOF
236+
apiVersion: snapshot.storage.k8s.io/v1
237+
kind: VolumeSnapshot
238+
metadata:
239+
name: raw-pvc-snap-2
240+
spec:
241+
volumeSnapshotClassName: csi-hostpath-snapclass
242+
source:
243+
persistentVolumeClaimName: pvc-raw
244+
EOF
245+
```
246+
247+
Wait for snapshot to be ready
248+
249+
```
250+
$ kubectl get vs
251+
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
252+
raw-pvc-snap-1 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-ef10f725-4261-4e80-af37-906708796700 7m40s 7m40s
253+
raw-pvc-snap-2 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-188562cb-03b3-4b70-b12d-28900527bca8 23s 23s
254+
```
255+
256+
3. Using `external-snapshot-metadata-client` which uses `GetMetadataDelta` gRPC to allocated blocks metadata
257+
258+
```
259+
$ kubectl exec -n csi-client csi-client -- snapshot-metadata-lister -n default -s raw-pvc-snap-1 -p raw-pvc-snap-2
260+
261+
262+
Record# VolCapBytes BlockMetadataType ByteOffset SizeBytes
263+
------- -------------- ----------------- -------------- --------------
264+
1 10485760 FIXED_LENGTH 49152 4096
265+
1 10485760 FIXED_LENGTH 53248 4096
266+
1 10485760 FIXED_LENGTH 61440 4096
267+
1 10485760 FIXED_LENGTH 81920 4096
268+
```

0 commit comments

Comments
 (0)