feat: properly support kube-vip upgrades (#1062)

**What problem does this PR solve?**:
See
https://docs.google.com/document/d/1RolIuLXf5qD5l2lya0ZtY-ZcGkh07ctiRiG8jjXMthU/edit?usp=sharing
with additional details.

We ran into an issue where upgrading CAREN causes an unwanted rollout of
all control plane Nodes.
This happens when CAREN is shipped with a new kube-vip version and the
difference in the static Pod spec changes the `file` contents in KCP and
forces new Nodes to be created.

We do not want this behavior and only want this rollout to happen at
controlled times.

In a recent change we introduced a mechanism of making backend changes
by using versioned handlers.
7bc9094

This change follows a similar pattern. A new handler is introduced that
will instead of reading the kube-vip template from a global ConfigMap
will read it directly from the KCPTemplate associated with the
ClusterClass.
We can then continue to change the kube-vip version (or any other spec
changes there) and downstream projects will just pick up whatever
changes through a new ClusterClass.

Another important goal with this approach was to not require consumers
of CAREN to maintain some reference to the "correct" kube-vip template.
We want to keep upgrades as simple as possible by only changing the
Kubernetes version, and let CAREN handle its own business logic.

**Alternative Considered Solutions**

1. Add a ConfigMap ref of the static Pod template to the API. This would
require the clients to keep track of and update another field on
upgrades. We want to keep the upgrade changes as simple as possible and
leave this kind of implementation detail to CAREN.
2. Add logic to only apply new changes when upgrading. This approach was
considered for a previous scenario, but we decided against it due to
complexity and instead introduced versioned ClusterClass and templates.

**Which issue(s) this PR fixes**:
Fixes #

**How Has This Been Tested?**:
<!--
Please describe the tests that you ran to verify your changes.
Provide output from the tests and any manual steps needed to replicate
the tests.
-->
Tested locally by creating a cluster with the new and old handlers.

**Special notes for your reviewer**:
<!--
Use this to provide any additional information to the reviewers.
This may include:
- Best way to review the PR.
- Where the author wants the most review attention on.
- etc.
-->
This PR introduces a "v3" version of the handlers with the modified
kube-vip handler, and preserves the previous behavior from v2.

Author: dkoshkin

.pre-commit-config.yaml

@@ -153,7 +153,7 @@ repos:
     name: License headers - YAML and Makefiles
     stages: [pre-commit]
     files: (^Makefile|\.(ya?ml|mk))$
-    exclude: ^(internal/test|pkg/handlers/.+/embedded|examples|charts/cluster-api-runtime-extensions-nutanix/(defaultclusterclasses|addons))/.+\.ya?ml|docs/static/helm/index\.yaml|charts/cluster-api-runtime-extensions-nutanix/templates/helm-config.yaml$
+    exclude: ^(internal/test|pkg/handlers/.+/embedded|examples|charts/cluster-api-runtime-extensions-nutanix/(defaultclusterclasses|addons))/.+\.ya?ml|docs/static/helm/index\.yaml|charts/cluster-api-runtime-extensions-nutanix/templates/helm-config.yaml|hack/examples/files/kube-vip.yaml$
     args:
       - --license-filepath
       - hack/license-header.txt

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/aws-cluster-class.yaml

@@ -23,7 +23,7 @@ spec:
   patches:
   - external:
       discoverVariablesExtension: awsclusterconfigvars-dv.cluster-api-runtime-extensions-nutanix
-      generateExtension: awsclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix
+      generateExtension: awsclusterv3configpatch-gp.cluster-api-runtime-extensions-nutanix
     name: cluster-config
   - external:
       discoverVariablesExtension: awsworkerconfigvars-dv.cluster-api-runtime-extensions-nutanix

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/docker-cluster-class.yaml

@@ -31,7 +31,7 @@ spec:
   patches:
   - external:
       discoverVariablesExtension: dockerclusterconfigvars-dv.cluster-api-runtime-extensions-nutanix
-      generateExtension: dockerclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix
+      generateExtension: dockerclusterv3configpatch-gp.cluster-api-runtime-extensions-nutanix
     name: cluster-config
   - external:
       discoverVariablesExtension: dockerworkerconfigvars-dv.cluster-api-runtime-extensions-nutanix

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/nutanix-cluster-class.yaml

@@ -67,7 +67,7 @@ spec:
   patches:
   - external:
       discoverVariablesExtension: nutanixclusterconfigvars-dv.cluster-api-runtime-extensions-nutanix
-      generateExtension: nutanixclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix
+      generateExtension: nutanixclusterv3configpatch-gp.cluster-api-runtime-extensions-nutanix
     name: cluster-config
   - external:
       discoverVariablesExtension: nutanixworkerconfigvars-dv.cluster-api-runtime-extensions-nutanix
@@ -132,7 +132,69 @@ spec:
           scheduler:
             extraArgs:
               tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
-        files: []
+        files:
+        - content: |-
+            apiVersion: v1
+            kind: Pod
+            metadata:
+              name: kube-vip
+              namespace: kube-system
+            spec:
+              containers:
+                - args:
+                    - manager
+                  env:
+                    - name: vip_arp
+                      value: "true"
+                    - name: port
+                      value: '{{ .Port }}'
+                    - name: vip_nodename
+                      valueFrom:
+                        fieldRef:
+                          fieldPath: spec.nodeName
+                    - name: vip_cidr
+                      value: "32"
+                    - name: dns_mode
+                      value: first
+                    - name: cp_enable
+                      value: "true"
+                    - name: cp_namespace
+                      value: kube-system
+                    - name: vip_leaderelection
+                      value: "true"
+                    - name: vip_leasename
+                      value: plndr-cp-lock
+                    - name: vip_leaseduration
+                      value: "15"
+                    - name: vip_renewdeadline
+                      value: "10"
+                    - name: vip_retryperiod
+                      value: "2"
+                    - name: address
+                      value: '{{ .Address }}'
+                    - name: prometheus_server
+                  image: ghcr.io/kube-vip/kube-vip:v0.8.9
+                  imagePullPolicy: IfNotPresent
+                  name: kube-vip
+                  resources: {}
+                  securityContext:
+                    capabilities:
+                      add:
+                        - NET_ADMIN
+                        - NET_RAW
+                  volumeMounts:
+                    - mountPath: /etc/kubernetes/admin.conf
+                      name: kubeconfig
+              hostAliases:
+                - hostnames:
+                    - kubernetes
+                  ip: 127.0.0.1
+              hostNetwork: true
+              volumes:
+                - hostPath:
+                    path: /etc/kubernetes/admin.conf
+                  name: kubeconfig
+          path: /etc/kubernetes/manifests/kube-vip.yaml
         initConfiguration:
           nodeRegistration:
             kubeletExtraArgs:

common/pkg/testutils/capitest/request/items.go

@@ -83,9 +83,19 @@ func NewKubeadmConfigTemplateRequest(
 	)
 }
 
-func NewKubeadmControlPlaneTemplateRequest(
+type KubeadmControlPlaneTemplateRequestItemBuilder struct {
+	files []bootstrapv1.File
+}
+
+func (b *KubeadmControlPlaneTemplateRequestItemBuilder) WithFiles(
+	files ...bootstrapv1.File,
+) *KubeadmControlPlaneTemplateRequestItemBuilder {
+	b.files = files
+	return b
+}
+
+func (b *KubeadmControlPlaneTemplateRequestItemBuilder) NewRequest(
 	uid types.UID,
-	name string,
 ) runtimehooksv1.GeneratePatchesRequestItem {
 	return NewRequestItem(
 		&controlplanev1.KubeadmControlPlaneTemplate{
@@ -94,7 +104,7 @@ func NewKubeadmControlPlaneTemplateRequest(
 				Kind:       "KubeadmControlPlaneTemplate",
 			},
 			ObjectMeta: metav1.ObjectMeta{
-				Name:      name,
+				Name:      kubeadmControlPlaneTemplateRequestObjectName,
 				Namespace: Namespace,
 			},
 			Spec: controlplanev1.KubeadmControlPlaneTemplateSpec{
@@ -107,6 +117,7 @@ func NewKubeadmControlPlaneTemplateRequest(
 							JoinConfiguration: &bootstrapv1.JoinConfiguration{
 								NodeRegistration: bootstrapv1.NodeRegistrationOptions{},
 							},
+							Files: b.files,
 						},
 					},
 				},
@@ -126,7 +137,8 @@ func NewKubeadmControlPlaneTemplateRequest(
 func NewKubeadmControlPlaneTemplateRequestItem(
 	uid types.UID,
 ) runtimehooksv1.GeneratePatchesRequestItem {
-	return NewKubeadmControlPlaneTemplateRequest(uid, kubeadmControlPlaneTemplateRequestObjectName)
+	builder := &KubeadmControlPlaneTemplateRequestItemBuilder{}
+	return builder.NewRequest(uid)
 }
 
 func NewCPDockerMachineTemplateRequestItem(

hack/addons/update-kube-vip-manifests.sh

@@ -13,10 +13,7 @@ if [ -z "${KUBE_VIP_VERSION:-}" ]; then
   exit 1
 fi
 
-ASSETS_DIR="$(mktemp -d -p "${TMPDIR:-/tmp}")"
-readonly ASSETS_DIR
-trap_add "rm -rf ${ASSETS_DIR}" EXIT
-
+readonly ASSETS_DIR="hack/examples/files"
 readonly FILE_NAME="kube-vip.yaml"
 
 # shellcheck disable=SC2016 # Single quotes are required for the gojq expression.
@@ -33,23 +30,9 @@ docker container run --rm ghcr.io/kube-vip/kube-vip:"${KUBE_VIP_VERSION}" \
   gojq --yaml-input --yaml-output \
     'del(.metadata.creationTimestamp, .status) |
      .spec.containers[].imagePullPolicy |= "IfNotPresent" |
-     (.spec.containers[0].env[] | select(.name == "port").value) |= "{{ `{{ .Port }}` }}" |
-     (.spec.containers[0].env[] | select(.name == "address").value) |= "{{ `{{ .Address }}` }}"
+     (.spec.containers[0].env[] | select(.name == "port").value) |= "{{ .Port }}" |
+     (.spec.containers[0].env[] | select(.name == "address").value) |= "{{ .Address }}"
     ' >"${ASSETS_DIR}/${FILE_NAME}"
 
-kubectl create configmap "{{ .Values.hooks.virtualIP.kubeVip.defaultTemplateConfigMap.name }}" --dry-run=client --output yaml \
-  --from-file "${ASSETS_DIR}/${FILE_NAME}" \
-  >"${ASSETS_DIR}/kube-vip-configmap.yaml"
-
-# add warning not to edit file directly
-cat <<EOF >"${GIT_REPO_ROOT}/charts/cluster-api-runtime-extensions-nutanix/templates/virtual-ip/kube-vip/manifests/kube-vip-configmap.yaml"
-$(cat "${GIT_REPO_ROOT}/hack/license-header.yaml.txt")
-
-#=================================================================
-#                 DO NOT EDIT THIS FILE
-#  IT HAS BEEN GENERATED BY /hack/addons/update-kube-vip-manifests.sh
-#=================================================================
-{{- if .Values.hooks.virtualIP.kubeVip.defaultTemplateConfigMap.create }}
-$(cat "${ASSETS_DIR}/kube-vip-configmap.yaml")
-{{- end -}}
-EOF
+# add 8 spaces to each line so that the kustomize template can be properly indented
+sed -i -e 's/^/        /' "${ASSETS_DIR}/${FILE_NAME}"

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl

@@ -46,16 +46,23 @@ patches:
     - op: "remove"
       path: "/spec/template/spec/kubeadmConfigSpec/clusterConfiguration/apiServer/certSANs"
 
-# Delete the kube-vip file.
-# Will be templated and added back in the handler if enabled.
+# Template the kube-vip file.
+# The handler will set the variables if needed, or remove it.
 - target:
     kind: KubeadmControlPlaneTemplate
   patch: |-
     - op: test
       path: /spec/template/spec/kubeadmConfigSpec/files/0/path
       value: "/etc/kubernetes/manifests/kube-vip.yaml"
     - op: "remove"
-      path: "/spec/template/spec/kubeadmConfigSpec/files/0"
+      path: "/spec/template/spec/kubeadmConfigSpec/files/0/owner"
+    - op: "replace"
+      path: "/spec/template/spec/kubeadmConfigSpec/files/0/path"
+      value: "/etc/kubernetes/manifests/kube-vip.yaml"
+    - op: "replace"
+      path: "/spec/template/spec/kubeadmConfigSpec/files/0/content"
+      value: |
+${KUBE_VIP_CONTENT}
 
 # Delete the kube-vip related pre and postKubeadmCommands.
 # Will be added back in the handler if enabled.

hack/examples/files/kube-vip.yaml

@@ -0,0 +1,60 @@
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: kube-vip
+          namespace: kube-system
+        spec:
+          containers:
+            - args:
+                - manager
+              env:
+                - name: vip_arp
+                  value: "true"
+                - name: port
+                  value: '{{ .Port }}'
+                - name: vip_nodename
+                  valueFrom:
+                    fieldRef:
+                      fieldPath: spec.nodeName
+                - name: vip_cidr
+                  value: "32"
+                - name: dns_mode
+                  value: first
+                - name: cp_enable
+                  value: "true"
+                - name: cp_namespace
+                  value: kube-system
+                - name: vip_leaderelection
+                  value: "true"
+                - name: vip_leasename
+                  value: plndr-cp-lock
+                - name: vip_leaseduration
+                  value: "15"
+                - name: vip_renewdeadline
+                  value: "10"
+                - name: vip_retryperiod
+                  value: "2"
+                - name: address
+                  value: '{{ .Address }}'
+                - name: prometheus_server
+              image: ghcr.io/kube-vip/kube-vip:v0.8.9
+              imagePullPolicy: IfNotPresent
+              name: kube-vip
+              resources: {}
+              securityContext:
+                capabilities:
+                  add:
+                    - NET_ADMIN
+                    - NET_RAW
+              volumeMounts:
+                - mountPath: /etc/kubernetes/admin.conf
+                  name: kubeconfig
+          hostAliases:
+            - hostnames:
+                - kubernetes
+              ip: 127.0.0.1
+          hostNetwork: true
+          volumes:
+            - hostPath:
+                path: /etc/kubernetes/admin.conf
+              name: kubeconfig

hack/examples/overlays/clusterclasses/aws/kustomization.yaml.tmpl

@@ -19,7 +19,7 @@ patches:
         value:
           - name: "cluster-config"
             external:
-              generateExtension: "awsclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix"
+              generateExtension: "awsclusterv3configpatch-gp.cluster-api-runtime-extensions-nutanix"
               discoverVariablesExtension: "awsclusterconfigvars-dv.cluster-api-runtime-extensions-nutanix"
           - name: "worker-config"
             external:

hack/examples/overlays/clusterclasses/docker/kustomization.yaml.tmpl

@@ -19,7 +19,7 @@ patches:
         value:
           - name: "cluster-config"
             external:
-              generateExtension: "dockerclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix"
+              generateExtension: "dockerclusterv3configpatch-gp.cluster-api-runtime-extensions-nutanix"
               discoverVariablesExtension: "dockerclusterconfigvars-dv.cluster-api-runtime-extensions-nutanix"
           - name: "worker-config"
             external: