Skip to content

Commit 4f3b02a

Browse files
authored
Merge pull request #353 from AndiLi99/AndiLi99/webhook
Add snapshot webhook build and deployment. Modify controller to label invalid objects.
2 parents 6838d02 + 8e8ab7b commit 4f3b02a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+12821
-21
lines changed

Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
.PHONY: all snapshot-controller csi-snapshotter clean test
15+
.PHONY: all snapshot-controller csi-snapshotter validation-webhook clean test
1616

17-
CMDS=snapshot-controller csi-snapshotter
17+
CMDS=snapshot-controller csi-snapshotter validation-webhook
1818
all: build
1919
include release-tools/build.make

README.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ The CSI snapshotter is part of Kubernetes implementation of [Container Storage I
66

77
The volume snapshot feature supports CSI v1.0 and higher. It was introduced as an Alpha feature in Kubernetes v1.12 and has been promoted to an Beta feature in Kubernetes 1.17.
88

9+
> :warning: **WARNING**: There is a new validating webhook server which provides tightened validation on snapshot objects. This SHOULD be installed by all users of this feature. More details [below](#validating-webhook).
10+
911

1012
## Overview
1113

@@ -83,6 +85,22 @@ Install CSI Driver:
8385
* kubectl create -f deploy/kubernetes/csi-snapshotter
8486
* https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/csi-snapshotter
8587

88+
### Validating Webhook
89+
90+
The snapshot validating webhook is an HTTP callback which responds to [admission requests](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). It is part of a larger [plan](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md) to tighten validation for volume snapshot objects. This webhook introduces the [ratcheting validation](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md#backwards-compatibility) mechanism targeting the tighter validation. The cluster admin or Kubernetes distribution admin should install the webhook alongside the snapshot controllers and CRDs.
91+
92+
> :warning: **WARNING**: Cluster admins choosing not to install the webhook server and participate in the phased release process can cause future problems when upgrading from `v1beta1` to `v1` volumesnapshot API, if there are currently persisted objects which fail the new stricter validation. Potential impacts include being unable to delete invalid snapshot objects.
93+
94+
Read more about how to install the example webhook [here](deploy/kubernetes/webhook-example/README.md).
95+
96+
#### Validating Webhook Command Line Options
97+
98+
* `--tls-cert-file`: File containing the x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). Required.
99+
100+
* `--tls-private-key-file`: File containing the x509 private key matching --tls-cert-file. Required.
101+
102+
* `--port`: Secure port that the webhook listens on (default 443)
103+
86104
### Snapshot controller command line options
87105

88106
#### Important optional arguments that are highly recommended to be used

cmd/validation-webhook/Dockerfile

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
FROM gcr.io/distroless/base:latest
2+
LABEL maintainers="Kubernetes Authors"
3+
LABEL description="Snapshot Validation Webhook"
4+
ARG binary=./bin/validation-webhook
5+
6+
COPY ${binary} validation-webhook
7+
ENTRYPOINT ["/validation-webhook"]

cmd/validation-webhook/main.go

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
/*
2+
Copyright 2020 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package main
18+
19+
import (
20+
"flag"
21+
22+
webhook "github.com/kubernetes-csi/external-snapshotter/v2/pkg/validation-webhook"
23+
"k8s.io/klog"
24+
)
25+
26+
func main() {
27+
rootCmd := webhook.CmdWebhook
28+
29+
loggingFlags := &flag.FlagSet{}
30+
klog.InitFlags(loggingFlags)
31+
rootCmd.PersistentFlags().AddGoFlagSet(loggingFlags)
32+
rootCmd.Execute()
33+
}
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Validating Webhook
2+
3+
The snapshot validating webhook is an HTTP callback which responds to [admission requests](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). It is part of a larger [plan](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md) to tighten validation for volume snapshot objects. This webhook introduces the [ratcheting validation](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md#backwards-compatibility) mechanism targeting the tighter validation. The cluster admin or Kubernetes distribution admin should install the webhook alongside the snapshot controllers and CRDs.
4+
5+
> :warning: **WARNING**: Cluster admins choosing not to install the webhook server and participate in the phased release process can cause future problems when upgrading from `v1beta1` to `v1` volumesnapshot API, if there are currently persisted objects which fail the new stricter validation. Potential impacts include being unable to delete invalid snapshot objects.
6+
7+
## Prerequisites
8+
9+
The following are prerequisites to use this validating webhook:
10+
11+
- K8s version 1.17+ (v1.9+ to use `admissionregistration.k8s.io/v1beta1`, v1.16+ to use `admissionregistration.k8s.io/v1`, v1.17+ to use `snapshot.storage.k8s.io/v1beta1`)
12+
- ValidatingAdmissionWebhook is [enabled](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites). (in v1.18+ it will be enabled by default)
13+
- API `admissionregistration.k8s.io/v1beta1` or `admissionregistration.k8s.io/v1` is enabled.
14+
15+
## How to build the webhook
16+
17+
Build the binary
18+
19+
```bash
20+
make
21+
```
22+
23+
Build the docker image
24+
25+
```bash
26+
docker build -t validation-webhook:latest -f ./cmd/validation-webhook/Dockerfile .
27+
```
28+
29+
## How to deploy the webhook
30+
31+
The webhook server is provided as an image which can be built from this repository. It can be deployed anywhere, as long as the api server is able to reach it over HTTPS. It is recommended to deploy the webhook server in the cluster as snapshotting is latency sensitive. A `ValidatingWebhookConfiguration` object is needed to configure the api server to contact the webhook server. Please see the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) for more details. The webhook server code is adapted from the [webhook server](https://github.com/kubernetes/kubernetes/tree/v1.18.6/test/images/agnhost/webhook) used in the kubernetes/kubernetes end to end testing code.
32+
33+
### Example in-cluster deployment using Kubernetes Secrets
34+
35+
Please note this is not considered to be a production ready method to deploy the certificates and is only provided for demo purposes. This is only one of many ways to deploy the certificates, it is your responsibility to ensure the security of your cluster. TLS certificates and private keys should be handled with care and you may not want to keep them in plain Kubernetes secrets.
36+
37+
This method was heavily adapted from [banzai cloud](https://banzaicloud.com/blog/k8s-admission-webhooks/).
38+
39+
#### Method
40+
41+
These commands should be run from the top level directory.
42+
43+
1. Run the `create-cert.sh` script. Note using the default namespace will allow anyone with access to that namespace to read your secret. It is recommended to change the namespace in all the files and the commands given below.
44+
45+
46+
```bash
47+
# This script will create a TLS certificate signed by the [cluster](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/). It will place the public and private key into a secret on the cluster.
48+
./deploy/kubernetes/webhook-example/create-cert.sh --service snapshot-validation-service --secret snapshot-validation-secret --namespace default # Make sure to use a different namespace
49+
```
50+
51+
2. Patch the `ValidatingWebhookConfiguration` file from the template, filling in the CA bundle field.
52+
53+
```bash
54+
cat ./deploy/kubernetes/webhook-example/admission-configuration-template | ./deploy/kubernetes/webhook-example/patch-ca-bundle.sh > ./deploy/kubernetes/webhook-example/admission-configuration.yaml
55+
```
56+
57+
3. Change the namespace in the generated `admission-configuration.yaml` file. Change the namespace in the service and deployment in the `webhook.yaml` file.
58+
59+
4. Create the deployment, service and admission configuration objects on the cluster.
60+
61+
```bash
62+
kubectl apply -f ./deploy/kubernetes/webhook-example
63+
```
64+
65+
Once all the pods from the deployment are up and running, you should be ready to go.
66+
67+
#### Verify the webhook works
68+
69+
Try to create an invalid snapshot object, the snapshot creation should fail.
70+
71+
```bash
72+
kubectl create -f ./examples/kubernetes/invalid-snapshot.yaml
73+
```
74+
75+
### Other methods to deploy the webhook server
76+
77+
Look into [cert-manager](https://cert-manager.io/) to handle the certificates, and this kube-builder [tutorial](https://book.kubebuilder.io/cronjob-tutorial/cert-manager.html) on how to deploy a webhook.
78+
79+
#### Important
80+
81+
Please see the deployment [yaml](./webhook.yaml) for the arguments expected by the webhook server. The snapshot validation webhook is served at the path `/snapshot`.
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
apiVersion: admissionregistration.k8s.io/v1
2+
kind: ValidatingWebhookConfiguration
3+
metadata:
4+
name: "validation-webhook.snapshot.storage.k8s.io"
5+
namespace: "default"
6+
webhooks:
7+
- name: "validation-webhook.snapshot.storage.k8s.io"
8+
rules:
9+
- apiGroups: ["snapshot.storage.k8s.io"]
10+
apiVersions: ["v1beta1"]
11+
operations: ["CREATE", "UPDATE"]
12+
resources: ["volumesnapshots", "volumesnapshotcontents"]
13+
scope: "*"
14+
clientConfig:
15+
service:
16+
namespace: "default"
17+
name: "snapshot-validation-service"
18+
path: "/volumesnapshot"
19+
caBundle: ${CA_BUNDLE}
20+
admissionReviewVersions: ["v1", "v1beta1"]
21+
sideEffects: None
22+
failurePolicy: Ignore # We recommend switching to Fail only after successful installation of the webhook server and webhook.
23+
timeoutSeconds: 2 # This will affect the latency and performance. Finetune this value based on your application's tolerance.
Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
#!/bin/bash
2+
# File originally from https://github.com/banzaicloud/admission-webhook-example/blob/blog/deployment/webhook-create-signed-cert.sh
3+
4+
set -e
5+
6+
usage() {
7+
cat <<EOF
8+
Generate certificate suitable for use with a webhook service.
9+
This script uses k8s' CertificateSigningRequest API to a generate a
10+
certificate signed by k8s CA suitable for use with webhook
11+
services. This requires permissions to create and approve CSR. See
12+
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
13+
detailed explantion and additional instructions.
14+
The server key/cert k8s CA cert are stored in a k8s secret.
15+
usage: ${0} [OPTIONS]
16+
The following flags are required.
17+
--service Service name of webhook.
18+
--namespace Namespace where webhook service and secret reside.
19+
--secret Secret name for CA certificate and server certificate/key pair.
20+
EOF
21+
exit 1
22+
}
23+
24+
while [[ $# -gt 0 ]]; do
25+
case ${1} in
26+
--service)
27+
service="$2"
28+
shift
29+
;;
30+
--secret)
31+
secret="$2"
32+
shift
33+
;;
34+
--namespace)
35+
namespace="$2"
36+
shift
37+
;;
38+
*)
39+
usage
40+
;;
41+
esac
42+
shift
43+
done
44+
45+
[ -z ${service} ] && service=admission-webhook-example-svc
46+
[ -z ${secret} ] && secret=admission-webhook-example-certs
47+
[ -z ${namespace} ] && namespace=default
48+
49+
if [ ! -x "$(command -v openssl)" ]; then
50+
echo "openssl not found"
51+
exit 1
52+
fi
53+
54+
csrName=${service}.${namespace}
55+
tmpdir=$(mktemp -d)
56+
echo "creating certs in tmpdir ${tmpdir} "
57+
58+
cat <<EOF >> ${tmpdir}/csr.conf
59+
[req]
60+
req_extensions = v3_req
61+
distinguished_name = req_distinguished_name
62+
[req_distinguished_name]
63+
[ v3_req ]
64+
basicConstraints = CA:FALSE
65+
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
66+
extendedKeyUsage = serverAuth
67+
subjectAltName = @alt_names
68+
[alt_names]
69+
DNS.1 = ${service}
70+
DNS.2 = ${service}.${namespace}
71+
DNS.3 = ${service}.${namespace}.svc
72+
EOF
73+
74+
openssl genrsa -out ${tmpdir}/server-key.pem 2048
75+
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
76+
77+
# clean-up any previously created CSR for our service. Ignore errors if not present.
78+
kubectl delete csr ${csrName} 2>/dev/null || true
79+
80+
# create server cert/key CSR and send to k8s API
81+
cat <<EOF | kubectl create -f -
82+
apiVersion: certificates.k8s.io/v1beta1
83+
kind: CertificateSigningRequest
84+
metadata:
85+
name: ${csrName}
86+
spec:
87+
groups:
88+
- system:authenticated
89+
request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
90+
usages:
91+
- digital signature
92+
- key encipherment
93+
- server auth
94+
EOF
95+
96+
# verify CSR has been created
97+
while true; do
98+
kubectl get csr ${csrName}
99+
if [ "$?" -eq 0 ]; then
100+
break
101+
fi
102+
done
103+
104+
# approve and fetch the signed certificate
105+
kubectl certificate approve ${csrName}
106+
# verify certificate has been signed
107+
for x in $(seq 10); do
108+
serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
109+
if [[ ${serverCert} != '' ]]; then
110+
break
111+
fi
112+
sleep 1
113+
done
114+
if [[ ${serverCert} == '' ]]; then
115+
echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
116+
exit 1
117+
fi
118+
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem
119+
120+
121+
# create the secret with CA cert and server cert/key
122+
kubectl create secret generic ${secret} \
123+
--from-file=key.pem=${tmpdir}/server-key.pem \
124+
--from-file=cert.pem=${tmpdir}/server-cert.pem \
125+
--dry-run -o yaml |
126+
kubectl -n ${namespace} apply -f -
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
# File originally from https://github.com/banzaicloud/admission-webhook-example/blob/blog/deployment/webhook-patch-ca-bundle.sh
3+
4+
ROOT=$(cd $(dirname $0)/../../; pwd)
5+
6+
set -o errexit
7+
set -o nounset
8+
set -o pipefail
9+
10+
export CA_BUNDLE=$(kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"')
11+
12+
if command -v envsubst >/dev/null 2>&1; then
13+
envsubst
14+
else
15+
sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g"
16+
fi
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: snapshot-validation-deployment
5+
namespace: default # NOTE: change the namespace
6+
labels:
7+
app: snapshot-validation
8+
spec:
9+
replicas: 3
10+
selector:
11+
matchLabels:
12+
app: snapshot-validation
13+
template:
14+
metadata:
15+
labels:
16+
app: snapshot-validation
17+
spec:
18+
containers:
19+
- name: snapshot-validation
20+
image: k8s.gcr.io/sig-storage/validation-webhook:v2.2.0 # change the image if you wish to use your own custom validation server image
21+
args: ['--tls-cert-file=/etc/snapshot-validation-webhook/certs/cert.pem', '--tls-private-key-file=/etc/snapshot-validation-webhook/certs/key.pem']
22+
ports:
23+
- containerPort: 443 # change the port as needed
24+
volumeMounts:
25+
- name: snapshot-validation-webhook-certs
26+
mountPath: /etc/snapshot-validation-webhook/certs
27+
readOnly: true
28+
volumes:
29+
- name: snapshot-validation-webhook-certs
30+
secret:
31+
secretName: snapshot-validation-secret
32+
---
33+
apiVersion: v1
34+
kind: Service
35+
metadata:
36+
name: snapshot-validation-service
37+
namespace: default # NOTE: change the namespace
38+
spec:
39+
selector:
40+
app: snapshot-validation
41+
ports:
42+
- protocol: TCP
43+
port: 443 # Change if needed
44+
targetPort: 443 # Change if the webserver image expects a different port
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
apiVersion: snapshot.storage.k8s.io/v1beta1
2+
kind: VolumeSnapshot
3+
metadata:
4+
name: new-snapshot-demo
5+
spec:
6+
volumeSnapshotClassName: csi-hostpath-snapclass
7+
source: # Only one of the two fields should be set for a snapshot. Therefore this snapshot is invalid.
8+
persistentVolumeClaimName: pvc
9+
volumeSnapshotContentName: vsc

0 commit comments

Comments
 (0)