Add hardened example

examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/README.md

@@ -0,0 +1,103 @@
+# Hardened Kubernetes Control Plane Template
+
+## Required Component: kubelet-csr-approver
+
+To meet the following CIS security benchmarks:
+
+- 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
+- 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
+
+You must install the Postfinance kubelet-csr-approver:
+
+```bash
+helm repo add kubelet-csr-approver https://postfinance.github.io/kubelet-csr-approver
+helm upgrade --install kubelet-csr-approver \
+  kubelet-csr-approver/kubelet-csr-approver \
+  -n kube-system \
+  --create-namespace \
+  --set maxExpirationSeconds=2592000 \
+  --set leaderElection=true \
+  --set bypassDnsResolution=true \
+  --set rbac.create=true
+```
+
+**Note**: If you choose not to install the kubelet-csr-approver, you must omit the flags related to the CIS benchmarks mentioned above from your configuration from the `cis-mitigations-cp-patch.yaml` file.
+
+## Directory Structure
+
+This directory contains the following files:
+
+- `harden.sh` - Automated script to simplify the hardening process (recommended method)
+- `cis-mitigations-cp-patch.yaml` - Patch file containing CIS hardening configurations
+- `kustomization.yaml` - Kustomization file that applies the patch and renames the template
+- `nkp-nutanix-<VERSION>.yaml` - The original KubeadmControlPlaneTemplate (generated during the hardening process via `./harden.sh`)
+
+The `harden.sh` script automates the following tasks:
+1. Lists available KubeadmControlPlaneTemplates
+2. Prompts for the NKP version
+3. Exports the original template
+4. Updates all version placeholders in configuration files
+5. Applies the kustomization to create the hardened template
+6. Provides guidance on patching the ClusterClass to use the hardened template
+
+## Applying the Hardening
+
+Simply run the hardening script and follow the prompts. Ensure that you have the `KUBECONFIG` environment variable set to the Management Cluster (or Self-Managed) before running it:
+
+```bash
+#export KUBECONFIG=<MANAGEMENT_CLUSTER_KUBECONFIG>
+./harden.sh
+```
+
+This script will guide you through the process, automatically generate the required files, and apply the kustomization.
+
+**Note**: For a fully hardened cluster, you should also apply hardening to the worker nodes by using the scripts in the `../worker` directory.
+
+## CIS Mitigations Applied
+
+The following CIS mitigations are applied to the Control Plane Nodes:
+
+### API Server
+
+- **1.2.15**: Disabled profiling for API server (`profiling: "false"`)
+- **1.2.21**: Enabled service account lookup (`service-account-lookup: "true"`)
+- **1.2.3, 1.2.9, 1.2.11, 1.2.14**: Added admission plugins:
+  - AlwaysPullImages: Enforces that images are always pulled prior to starting containers
+  - DenyServiceExternalIPs: Prevents services from using arbitrary external IPs
+  - EventRateLimit: Mitigates event flooding attacks
+  - NodeRestriction: Limits node access to specific APIs
+- **1.2.9**: Configured EventRateLimit with appropriate admission control config file
+- **1.2.5**: Set kubelet certificate authority (`kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt`)
+  - **Note**: This requires kubelet-csr-approver to be installed. If not installed, this flag should be omitted.
+
+### Controller Manager
+
+- **1.3.1**: Set terminated pod GC threshold to 10000 for better garbage collection
+- **1.3.2**: Disabled profiling (`profiling: "false"`)
+- **1.3.6**: Enabled RotateKubeletServerCertificate feature gate (`feature-gates: RotateKubeletServerCertificate=true`)
+  - **Note**: This requires kubelet-csr-approver to be installed. If not installed, this flag should be omitted.
+
+### Scheduler
+
+- **1.4.1**: Disabled profiling (`profiling: "false"`)
+
+### Kubelet Configuration (Both Init and Join)
+
+- **1.2.5, 1.3.6**: Enabled kubelet server certificate rotation (`rotate-server-certificates: "true"`)
+  - **Note**: This requires kubelet-csr-approver to be installed. If not installed, this flag should be omitted.
+
+### EventRateLimit Configuration
+
+- **1.2.9**: Created admission configuration files with detailed rate limits:
+  - Server-wide: 5000 QPS with 20000 burst
+  - Namespace: 500 QPS with 2000 burst (1000 cache size)
+  - User: 100 QPS with 400 burst (2000 cache size)
+  - SourceAndObject: 50 QPS with 100 burst (5000 cache size)
+
+### File Permissions
+
+- **4.1.1**: Set appropriate file permissions (0600) for sensitive files including:
+  - kubelet.service
+  - kubelet config.yaml
+  - 10-kubeadm.conf
+

examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/cis-mitigations-cp-patch.yaml

@@ -0,0 +1,70 @@
+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+kind: KubeadmControlPlaneTemplate
+metadata:
+  name: nkp-nutanix-<VERSION>
+  namespace: <NAMESPACE>
+spec:
+  template:
+    spec:
+      kubeadmConfigSpec:
+        clusterConfiguration:
+          apiServer:
+            extraArgs:
+              #1.2.15 Ensure that the --profiling argument is set to false
+              profiling: "false"
+              #1.2.21 Ensure that the --service-account-lookup argument is set to true
+              service-account-lookup: "true"
+              #1.2.3 Ensure that the DenyServiceExternalIPs is set
+              #1.2.9 Ensure that the admission control plugin EventRateLimit is set
+              #1.2.11 Ensure that the admission control plugin AlwaysPullImages is set
+              #1.2.14 Ensure that the admission control plugin NodeRestriction is set
+              enable-admission-plugins: AlwaysPullImages,EventRateLimit,DenyServiceExternalIPs,NodeRestriction
+              #1.2.9 Ensure that the admission control plugin EventRateLimit is set
+              admission-control-config-file: /etc/kubernetes/admission/admissionConfiguration.yaml
+              #1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
+              #This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
+              #helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
+              #If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
+              kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt
+            extraVolumes:
+              #1.2.9 Ensure that the admission control plugin EventRateLimit is set
+            - name: admission-config
+              hostPath: /etc/kubernetes/admission
+              mountPath: /etc/kubernetes/admission
+              readOnly: true
+              pathType: DirectoryOrCreate
+          controllerManager:
+            extraArgs:
+              #1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate
+              terminated-pod-gc-threshold: "10000"
+              #1.3.2 Ensure that the --profiling argument is set to false
+              profiling: "false"
+              #1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
+              #This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
+              #helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
+              #If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
+              feature-gates: RotateKubeletServerCertificate=true
+          scheduler:
+            extraArgs:
+              #1.4.1 Ensure that the --profiling argument is set to false
+              profiling: "false"
+
+        initConfiguration:
+          nodeRegistration:
+            kubeletExtraArgs:
+              #1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
+              #1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
+              #This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
+              #helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
+              #If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
+              rotate-server-certificates: "true"
+        joinConfiguration:
+          nodeRegistration:
+            kubeletExtraArgs:
+              #1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
+              #1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
+              #This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
+              #helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
+              #If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
+              rotate-server-certificates: "true"
+

examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/harden.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# Color codes
+CYAN='\033[0;36m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+RED='\033[0;31m'
+NC='\033[0m' # No Color
+
+echo -e "${CYAN}Listing all clusters in all namespaces...${NC}"
+
+kubectl get clusters -A
+
+echo ""
+echo -e "${YELLOW}Please enter the namespace of the cluster you wish to harden (press Enter to use 'default'):${NC}"
+read -p "> " NAMESPACE
+NAMESPACE=${NAMESPACE:-default}
+
+echo ""
+echo -e "${CYAN}Using namespace: $NAMESPACE${NC}"
+
+echo ""
+echo -e "${CYAN}Listing all the KubeadmControlPlaneTemplates in namespace $NAMESPACE...${NC}"
+kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io -n $NAMESPACE
+
+echo ""
+echo -e "${YELLOW}Please enter the NKP version from the list above (e.g., for NKP version nkp-nutanix-v2.14.0, enter v2.14.0):${NC}"
+read -p "> " VERSION
+
+echo ""
+# Validate version format
+if [[ ! $VERSION =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+  echo -e "${RED}Invalid version format. Please use the format v<major>.<minor>.<patch> (e.g., v2.14.1)${NC}"
+  exit 1
+fi
+
+echo -e "${CYAN}Using NKP version: $VERSION${NC}"
+
+echo ""
+# Clone the Latest KubeadmControlplaneTemplate
+echo -e "${CYAN}Cloning the template: nkp-nutanix-${VERSION} from namespace $NAMESPACE...${NC}"
+kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io nkp-nutanix-${VERSION} -n $NAMESPACE -o yaml > nkp-nutanix-${VERSION}.yaml
+
+echo ""
+# Replace <VERSION> with the actual version in all files
+echo -e "${CYAN}Replacing <VERSION> with ${VERSION} in all files...${NC}"
+sed -i "s/<VERSION>/${VERSION}/g" kustomization.yaml
+sed -i "s/<VERSION>/${VERSION}/g" cis-mitigations-cp-patch.yaml
+
+# Replace <NAMESPACE> with the actual namespace in all files
+echo -e "${CYAN}Replacing <NAMESPACE> with ${NAMESPACE} in all files...${NC}"
+sed -i "s/<NAMESPACE>/${NAMESPACE}/g" kustomization.yaml
+sed -i "s/<NAMESPACE>/${NAMESPACE}/g" cis-mitigations-cp-patch.yaml
+
+echo ""
+echo -e "${GREEN}Replacement complete!${NC}"
+echo -e "${GREEN}Files have been updated with version: ${VERSION} and namespace: ${NAMESPACE}${NC}"
+
+echo ""
+echo -e "${CYAN}Applying kustomization to create hardened control plane template...${NC}"
+kubectl apply -n $NAMESPACE -k .
+
+echo ""
+echo -e "${CYAN}You can now patch the ClusterClass to use the Hardened KubeadmControlPlaneTemplates${NC}"
+echo -e "${CYAN}Here are the available ClusterClass in namespace $NAMESPACE${NC}"
+
+kubectl get clusterclasses.cluster.x-k8s.io -n $NAMESPACE
+
+echo ""
+echo -e "${YELLOW}Run the below command after replacing the <CLUSTER_CLASS> with the ClusterClass in use from the list above${NC}"
+echo -e "kubectl patch clusterclass -n $NAMESPACE <CLUSTER_CLASS>  \\
+  --type merge -p='{\"spec\":{\"controlPlane\":{\"ref\":{\"name\":\"nkp-nutanix-${VERSION}-hardened\"}}}}'"

examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/kustomization.yaml

@@ -0,0 +1,125 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- nkp-nutanix-<VERSION>.yaml
+
+patches:
+- path: cis-mitigations-cp-patch.yaml
+- target:
+    group: controlplane.cluster.x-k8s.io
+    version: v1beta1
+    kind: KubeadmControlPlaneTemplate
+    #target <2.15.0
+    name: nkp-nutanix-v2.1[0-4].*
+    namespace: <NAMESPACE>
+  patch: |
+    #so that it works in v2.14.0. Make sure to remove after to 2.14.0
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/files
+      value: []
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/files/-
+      value:
+        path: /etc/kubernetes/admission/admissionConfiguration.yaml
+        permissions: "0600"
+        content: |
+          apiVersion: apiserver.config.k8s.io/v1
+          kind: AdmissionConfiguration
+          plugins:
+          - name: EventRateLimit
+            path: /etc/kubernetes/admission/eventRateLimit.yaml
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/files/-
+      value:
+        path: /etc/kubernetes/admission/eventRateLimit.yaml
+        permissions: "0600"
+        content: |
+          apiVersion: eventratelimit.admission.k8s.io/v1alpha1
+          kind: Configuration
+          limits:
+          - type: Server
+            qps: 5000
+            burst: 20000
+          - type: Namespace
+            qps: 500
+            burst: 2000
+            cacheSize: 1000
+          - type: User
+            qps: 100
+            burst: 400
+            cacheSize: 2000
+          - type: SourceAndObject
+            qps: 50
+            burst: 100
+            cacheSize: 5000
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'chmod 600 "$(systemctl show -P FragmentPath kubelet.service)"'
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'chmod 600 /var/lib/kubelet/config.yaml'
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'echo "serverTLSBootstrap: true" >> /var/lib/kubelet/config.yaml'
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'chmod 600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf'
+
+- target:
+    group: controlplane.cluster.x-k8s.io
+    version: v1beta1
+    kind: KubeadmControlPlaneTemplate
+    #target only NKP => v2.15.0
+    name: nkp-nutanix-v2.1[5-9].* 
+    namespace: <NAMESPACE>
+  patch: |
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/files/-
+      value:
+        path: /etc/kubernetes/admission/admissionConfiguration.yaml
+        permissions: "0600"
+        content: |
+          apiVersion: apiserver.config.k8s.io/v1
+          kind: AdmissionConfiguration
+          plugins:
+          - name: EventRateLimit
+            path: /etc/kubernetes/admission/eventRateLimit.yaml
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/files/-
+      value:
+        path: /etc/kubernetes/admission/eventRateLimit.yaml
+        permissions: "0600"
+        content: |
+          apiVersion: eventratelimit.admission.k8s.io/v1alpha1
+          kind: Configuration
+          limits:
+          - type: Server
+            qps: 5000
+            burst: 20000
+          - type: Namespace
+            qps: 500
+            burst: 2000
+            cacheSize: 1000
+          - type: User
+            qps: 100
+            burst: 400
+            cacheSize: 2000
+          - type: SourceAndObject
+            qps: 50
+            burst: 100
+            cacheSize: 5000
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'chmod 600 "$(systemctl show -P FragmentPath kubelet.service)"'
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'chmod 600 /var/lib/kubelet/config.yaml'
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'echo "serverTLSBootstrap: true" >> /var/lib/kubelet/config.yaml'
+    - op: add
+      path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
+      value: 'chmod 600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf'
+namePrefix: ""
+nameSuffix: "-hardened"