Skip to content

Commit e308167

Browse files
carretercarreter
committed
Make kind quickstart script more easily configurable by user
1 parent 6bc47e9 commit e308167

File tree

8 files changed

+211
-95
lines changed

8 files changed

+211
-95
lines changed

‎examples/kind-multinode-kcp/README.md

Lines changed: 76 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,12 @@
1-
# Use apiserver-network-proxy with KIND
1+
# Set up KIND cluster with multiple KCP and worker nodes running konnectivity
22

3+
Change to the `examples/kind-multinode-kcp` folder and run `./quickstart-kind`. This script
4+
performs the following operations:
35

4-
Change to the `examples/kind` folder and create a `kind` cluster with the `kind.config` file
5-
6-
```sh
7-
$ kind create cluster --config kind.config
8-
Creating cluster "kind" ...
9-
DEBUG: docker/images.go:58] Image: kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72 present locally
10-
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
11-
⠎⠁ Preparing nodes 📦 📦 📦
12-
13-
This node has joined the cluster:
14-
* Certificate signing request was sent to apiserver and a response was received.
15-
* The Kubelet was informed of the new secure connection details.
16-
17-
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
18-
✓ Joining worker nodes 🚜
19-
Set kubectl context to "kind-kind"
20-
You can now use your cluster with:
21-
22-
kubectl cluster-info --context kind-kind
23-
24-
Have a nice day! 👋
25-
```
26-
27-
Once the cluster is ready install the `apiserver-network-proxy` components:
28-
29-
```sh
30-
$ kubectl apply -f konnectivity-server.yaml
31-
clusterrolebinding.rbac.authorization.k8s.io/system:konnectivity-server created
32-
daemonset.apps/konnectivity-server created
33-
34-
$ kubectl apply -f konnectivity-agent-ds.yaml
35-
serviceaccount/konnectivity-agent created
36-
```
6+
1. Render config templates in `templates/` using provided values.
7+
2. Create a new `kind` cluster with the desired number of KCP and worker nodes.
8+
3. Changes `kubectl` context to point to the new `kind` cluster.
9+
4. Deploys `konnectivity` proxy servers and agents to the KCP and worker nodes.
3710

3811
To validate that it works, run a custom image and get pod logs (it goes through the konnectivity proxy):
3912
```sh
@@ -49,3 +22,72 @@ $ kubectl logs test
4922
...
5023
[Tue Apr 09 20:58:36.756720 2024] [mpm_event:notice] [pid 1:tid 139788897408896] AH00489: Apache/2.4.59 (Unix) configured -- resuming normal operations
5124
```
25+
26+
## `./quickstart-kind.sh` command-line flags
27+
- `--cluster-name <NAME>`: Name of the `kind` cluster to be created Default: `knp-test-cluster`
28+
- `--overwrite-cluster`: Overwrite existing `kind` cluster if necessary. Default: do not overwrite.
29+
- `--server-image <IMAGE_NAME>[:<IMAGE_TAG>]`: Proxy server image to deploy. Default: `gcr.io/k8s-staging-kas-network-proxy/proxy-server:master`
30+
- `--agent-image <IMAGE_NAME>[:<IMAGE_TAG>]`: Proxy server image to deploy. Default: `gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master`
31+
- `--num-kcp-nodes <NUM>`: Number of control plane nodes to spin up. Default: 2.
32+
- `--num-worker-nodes <NUM>`: Number of worker nodes to spin up. Default: 1.
33+
- `--sideload-images`: Use `kind load ...` to sideload custom proxy server and agent images with the names set by `--server-image` and `--agent-image` into the kind cluster. Default: do not sideload.
34+
- Use this if you don't want to publish your custom KNP images to a public registry.
35+
- NOTE: You MUST specify an image tag (i.e. `my-image-name:my-image-tag` and not just `my-image-name`) and the image tag MUST NOT be `:latest` for this to work! See [`kind` docs](https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster) for why this is necessary.
36+
37+
## Example usage to deploy custom local KNP images
38+
In the repo root, build KNP and its docker images with the following:
39+
```shell
40+
make clean
41+
make certs
42+
make gen
43+
make build
44+
make docker-build
45+
```
46+
47+
Verify that the new images are available in the local docker registry with `docker images`. Then, bring up the cluster:
48+
49+
```shell
50+
cd examples/kind-multinode-kcp
51+
52+
# These are the default values of the registry, image name, and tag used by the Makefile.
53+
# Edit them if necessary.
54+
REGISTRY=gcr.io/$(gcloud config get-value project)
55+
TAG=$(git rev-parse HEAD)
56+
TARGET_ARCH="amd64"
57+
SERVER_IMAGE="$REGISTRY/proxy-server-$TARGET_ARCH:$TAG"
58+
AGENT_IMAGE="$REGISTRY/proxy-agent-$TARGET_ARCH:$TAG"
59+
60+
# Bring up the cluster!
61+
./quickstart-kind.sh --cluster-name custom-knp-test --server-image "$SERVER_IMAGE" --agent-image "$AGENT_IMAGE" \
62+
--num-kcp-nodes 3 --num-worker-nodes 2 --sideload-images
63+
```
64+
65+
Check that the `konnectivity` pods are up and running:
66+
```shell
67+
kubectl --namespace kube-system get pods | grep konnectivity
68+
# Output:
69+
# konnectivity-agent-4db5j 1/1 Running 0 34m
70+
# konnectivity-agent-c7gj5 1/1 Running 0 34m
71+
# konnectivity-agent-h86l9 1/1 Running 0 34m
72+
# konnectivity-server-9bl45 1/1 Running 0 34m
73+
# konnectivity-server-dcfz8 1/1 Running 0 34m
74+
# konnectivity-server-klww5 1/1 Running 0 34m
75+
# konnectivity-server-nrfz8 1/1 Running 0 34m
76+
```
77+
78+
Then create a test pod on a worker node and verify you can get logs from it:
79+
```shell
80+
kubectl run test --image httpd:2
81+
# Output:
82+
# pod/test created
83+
kubectl get pods
84+
# Output:
85+
# NAME READY STATUS RESTARTS AGE
86+
# test 1/1 Running 0 34s
87+
kubectl logs test
88+
# Output:
89+
# AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.5.3. Set the 'ServerName' directive globally to suppress this message
90+
# AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.5.3. Set the 'ServerName' directive globally to suppress this message
91+
# [Wed Jun 12 20:42:06.471169 2024] [mpm_event:notice] [pid 1:tid 139903660291968] AH00489: Apache/2.4.59 (Unix) configured -- resuming normal operations
92+
# [Wed Jun 12 20:42:06.471651 2024] [core:notice] [pid 1:tid 139903660291968] AH00094: Command line: 'httpd -D FOREGROUND'
93+
```

‎examples/kind-multinode-kcp/kind.config

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎examples/kind-multinode-kcp/quickstart-kind.sh

Lines changed: 104 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,115 @@
22

33
set -e
44

5-
CLUSTER_NAME=$1
5+
# DEFAULT ARGS
6+
CLUSTER_NAME="knp-test-cluster"
7+
AGENT_IMAGE="gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master"
8+
SERVER_IMAGE="gcr.io/k8s-staging-kas-network-proxy/proxy-server:master"
9+
NUM_WORKER_NODES=1
10+
NUM_KCP_NODES=2
11+
OVERWRITE_CLUSTER=false
12+
SIDELOAD_IMAGES=false
13+
14+
# FUNCTION DEFINITIONS
15+
# For escaping sed replacement strings. Taken from https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed.
16+
quoteSubst() {
17+
IFS= read -d '' -r < <(sed -e ':a' -e '$!{N;ba' -e '}' -e 's/[&/\]/\\&/g; s/\n/\\&/g' <<<"$1")
18+
printf %s "${REPLY%$'\n'}"
19+
}
20+
21+
# Provide usage info
22+
usage() {
23+
printf "USAGE:\n./quickstart-kind.sh\n\t[--cluster-name <NAME>]\n\t[--server-image <IMAGE_NAME>[:<IMAGE_TAG>]]\n\t[--agent-image <IMAGE_NAME>[:<IMAGE_TAG>]]\n\t[--num-worker-nodes <NUM>]\n\t[--num-kcp-nodes <NUM>]\n\t[--overwrite-cluster]\n"
24+
}
25+
26+
# ARG PARSING
27+
VALID_ARGS=$(getopt --options "h" --longoptions "sideload-images,cluster-name:,agent-image:,server-image:,num-worker-nodes:,num-kcp-nodes:,help,overwrite-cluster" --name "$0" -- "$@") || exit 2
28+
29+
eval set -- "$VALID_ARGS"
30+
while true; do
31+
case "$1" in
32+
--cluster-name)
33+
CLUSTER_NAME=$2
34+
shift 2
35+
;;
36+
--agent-image)
37+
AGENT_IMAGE=$2
38+
shift 2
39+
;;
40+
--server-image)
41+
SERVER_IMAGE=$2
42+
shift 2
43+
;;
44+
--num-worker-nodes)
45+
NUM_WORKER_NODES=$2
46+
shift 2
47+
;;
48+
--num-kcp-nodes)
49+
NUM_KCP_NODES=$2
50+
shift 2
51+
;;
52+
--overwrite-cluster)
53+
OVERWRITE_CLUSTER=true
54+
shift 1
55+
;;
56+
--sideload-images)
57+
SIDELOAD_IMAGES=true
58+
shift 1
59+
;;
60+
--)
61+
shift
62+
break
63+
;;
64+
*|-h|--help)
65+
usage
66+
exit
67+
;;
68+
esac
69+
done
70+
71+
# RENDER CONFIG TEMPLATES
72+
echo "Rendering config templates..."
73+
if [ ! -d rendered ]; then
74+
echo "Creating ./rendered"
75+
mkdir rendered
76+
fi
77+
echo "Adding $NUM_KCP_NODES control plane nodes and $NUM_WORKER_NODES worker nodes to kind.config..."
78+
cp templates/kind/kind.config rendered/kind.config
79+
for i in $(seq 0 "$NUM_KCP_NODES")
80+
do
81+
cat templates/kind/control-plane.config >> rendered/kind.config
82+
done
83+
for i in $(seq 0 "$NUM_WORKER_NODES")
84+
do
85+
cat templates/kind/worker.config >> rendered/kind.config
86+
done
87+
88+
echo "Setting server image to $SERVER_IMAGE and agent image to $AGENT_IMAGE"
89+
sed "s/image: .*/image: $(quoteSubst "$AGENT_IMAGE")/" <templates/k8s/konnectivity-agent-ds.yaml >rendered/konnectivity-agent-ds.yaml
90+
sed "s/image: .*/image: $(quoteSubst "$SERVER_IMAGE")/" <templates/k8s/konnectivity-server.yaml >rendered/konnectivity-server.yaml
91+
92+
93+
# CLUSTER CREATION
94+
if [ $OVERWRITE_CLUSTER = true ] && kind get clusters | grep -q "$CLUSTER_NAME"; then
95+
echo "Deleting old cluster $CLUSTER_NAME..."
96+
kind delete clusters "$CLUSTER_NAME"
97+
fi
698

799
echo "Creating cluster $CLUSTER_NAME..."
8-
kind create cluster --config kind.config --name $CLUSTER_NAME
100+
kind create cluster --config rendered/kind.config --name $CLUSTER_NAME
9101

10102
echo "Successfully created cluster. Switching kubectl context to kind-$CLUSTER_NAME"
11103
kubectl cluster-info --context kind-$CLUSTER_NAME
12104

105+
# SIDELOAD IMAGES IF REQUESTED
106+
if [ $SIDELOAD_IMAGES = true ]; then
107+
echo "Sideloading images into the kind cluster..."
108+
kind --name "$CLUSTER_NAME" load docker-image "$SERVER_IMAGE"
109+
kind --name "$CLUSTER_NAME" load docker-image "$AGENT_IMAGE"
110+
fi
111+
112+
# DEPLOY KONNECTIVITY
13113
echo "Requesting creation of konnectivity proxy servers on cluster $CLUSTER_NAME..."
14-
kubectl apply -f konnectivity-server.yaml
114+
kubectl apply -f rendered/konnectivity-server.yaml
15115
echo "Requesting creation of konnectivity proxy agents on cluster $CLUSTER_NAME..."
16-
kubectl apply -f konnectivity-agent-ds.yaml
116+
kubectl apply -f rendered/konnectivity-agent-ds.yaml

‎examples/kind-multinode-kcp/konnectivity-agent-ds.yaml renamed to ‎examples/kind-multinode-kcp/templates/k8s/konnectivity-agent-ds.yaml

File renamed without changes.

‎examples/kind-multinode-kcp/konnectivity-server.yaml renamed to ‎examples/kind-multinode-kcp/templates/k8s/konnectivity-server.yaml

File renamed without changes.

‎examples/kind-multinode-kcp/templates/kind/control-plane.config

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
- role: control-plane
2+
kubeadmConfigPatchesJSON6902:
3+
- kind: ClusterConfiguration
4+
patch: |
5+
- op: add
6+
path: /apiServer/certSANs/-
7+
value: konnectivity-server.kube-system.svc.cluster.local
8+
kubeadmConfigPatches:
9+
- |
10+
kind: ClusterConfiguration
11+
apiServer:
12+
extraArgs:
13+
"egress-selector-config-file": "/etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml"
14+
extraVolumes:
15+
- name: egress-selector-config-file
16+
hostPath: "/etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml"
17+
mountPath: "/etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml"
18+
readOnly: true
19+
- name: konnectivity-server
20+
hostPath: "/etc/kubernetes/konnectivity-server"
21+
mountPath: "/etc/kubernetes/konnectivity-server"
22+
readOnly: true
23+
extraMounts:
24+
- hostPath: ./egress_selector_configuration.yaml
25+
containerPath: /etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml

‎examples/kind-multinode-kcp/templates/kind/kind.config

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
kind: Cluster
2+
apiVersion: kind.x-k8s.io/v1alpha4
3+
networking:
4+
ipFamily: ipv4
5+
nodes:

‎examples/kind-multinode-kcp/templates/kind/worker.config

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- role: worker

0 commit comments

Comments
 (0)