Skip to content

Commit de6d99a

Browse files
WinsonSouWinsonSou
committed
simplified hardening + support for 2.14.0 and 2.15.0
1 parent b63e5cf commit de6d99a

File tree

8 files changed

+284
-203
lines changed

8 files changed

+284
-203
lines changed

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/README.md‎

Lines changed: 19 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -21,80 +21,41 @@ helm upgrade --install kubelet-csr-approver \
2121
--set rbac.create=true
2222
```
2323

24-
**Note**: If you choose not to install the kubelet-csr-approver, you must omit the flags related to the CIS benchmarks mentioned above from your configuration.
24+
**Note**: If you choose not to install the kubelet-csr-approver, you must omit the flags related to the CIS benchmarks mentioned above from your configuration from the `cis-mitigations-cp-patch.yaml` file.
2525

26-
## Prerequisites
26+
## Directory Structure
2727

28-
Before applying the kustomization, you need to determine the available Nutanix-provided KubeadmControlPlaneTemplates (those with the "nkp-" prefix) and their versions:
28+
This directory contains the following files:
2929

30-
```bash
31-
# List all available KubeadmControlPlaneTemplates
32-
kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io
33-
34-
# Example output:
35-
# NAME AGE
36-
# nkp-nutanix-v2.14.0 4d
37-
# some-other-template 19h
38-
```
39-
40-
**Important**: You must use the Nutanix-provided template (starting with "nkp-") for these hardening configurations to work correctly.
41-
42-
Then export the original Nutanix KubeadmControlPlaneTemplate using the appropriate version:
43-
44-
```bash
45-
# Replace <VERSION> with your actual version (e.g., v2.14.0)
46-
kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io nkp-nutanix-<VERSION> -o yaml > nkp-nutanix-<VERSION>.yaml
47-
```
48-
49-
## Structure
50-
51-
- `nkp-nutanix-<VERSION>.yaml` - The original KubeadmControlPlaneTemplate (generated by user, replace <VERSION> with your actual version)
30+
- `harden.sh` - Automated script to simplify the hardening process (recommended method)
5231
- `cis-mitigations-cp-patch.yaml` - Patch file containing CIS hardening configurations
5332
- `kustomization.yaml` - Kustomization file that applies the patch and renames the template
33+
- `nkp-nutanix-<VERSION>.yaml` - The original KubeadmControlPlaneTemplate (generated during the hardening process via `./harden.sh`)
5434

55-
**Note**: You need to edit BOTH of these files with your actual version:
56-
57-
1. Edit the `cis-mitigations-cp-patch.yaml` file to replace `<VERSION>` with your actual version in the `metadata.name` field:
58-
59-
```yaml
60-
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
61-
kind: KubeadmControlPlaneTemplate
62-
metadata:
63-
name: nkp-nutanix-<VERSION> # <-- Replace <VERSION> with your actual version
64-
```
65-
66-
2. Edit the `kustomization.yaml` file to replace `<VERSION>` in the resources section:
67-
68-
```yaml
69-
resources:
70-
- nkp-nutanix-<VERSION>.yaml # <-- Replace <VERSION> with your actual version
71-
```
35+
The `harden.sh` script automates the following tasks:
36+
1. Lists available KubeadmControlPlaneTemplates
37+
2. Prompts for the NKP version
38+
3. Exports the original template
39+
4. Updates all version placeholders in configuration files
40+
5. Applies the kustomization to create the hardened template
41+
6. Provides guidance on patching the ClusterClass to use the hardened template
7242

73-
## Applying the Kustomization
43+
## Applying the Hardening
7444

75-
You can apply this kustomization directly with kubectl:
45+
Simply run the hardening script and follow the prompts. Ensure that you have the `KUBECONFIG` environment variable set to the Management Cluster (or Self-Managed) before running it:
7646

7747
```bash
78-
#Ensure you are in a directory for the kustomization.yaml, cis-mitigations-cp-patch.yaml and the nkp-nutanix-<VERSION>.yaml file
79-
kubectl apply -k .
48+
#export KUBECONFIG=<MANAGEMENT_CLUSTER_KUBECONFIG>
49+
./harden.sh
8050
```
8151

82-
## Verification
83-
84-
The kustomization will:
52+
This script will guide you through the process, automatically generate the required files, and apply the kustomization.
8553

86-
1. Take the user-generated KubeadmControlPlaneTemplate `nkp-nutanix-<VERSION>`
87-
2. Apply the CIS mitigations patch that adds:
88-
- API Server hardening (profiling disabled, service account lookup, admission plugins)
89-
- Controller Manager settings (terminated pod GC threshold, profiling disabled)
90-
- Scheduler settings (profiling disabled)
91-
- Required configuration files for EventRateLimit
92-
- Additional postKubeadmCommands for file permission hardening
93-
3. Rename it to `nkp-nutanix-<VERSION>-hardened`
54+
**Note**: For a fully hardened cluster, you should also apply hardening to the worker nodes by using the scripts in the `../worker` directory.
9455

9556
## CIS Mitigations Applied
9657

97-
The following CIS mitigations are applied:
58+
The following CIS mitigations are applied to the Control Plane Nodes:
9859

9960
### API Server
10061

@@ -140,20 +101,3 @@ The following CIS mitigations are applied:
140101
- kubelet config.yaml
141102
- 10-kubeadm.conf
142103

143-
## Applying to Cluster Class
144-
145-
After generating the hardened template, you need to update the cluster class to use it:
146-
147-
1. First, identify the cluster class name and version:
148-
149-
```bash
150-
kubectl get clusterclasses.cluster.x-k8s.io
151-
```
152-
153-
2. Patch the cluster class to use the hardened control plane template (replace `<VERSION>` and cluster class name as needed):
154-
155-
```bash
156-
kubectl patch clusterclass nkp-nutanix-<VERSION> \
157-
--type merge \
158-
-p='{"spec":{"controlPlane":{"ref":{"name":"nkp-nutanix-<VERSION>-hardened"}}}}'
159-
```

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/cis-mitigations-cp-patch.yaml‎

Lines changed: 2 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -47,38 +47,7 @@ spec:
4747
extraArgs:
4848
#1.4.1 Ensure that the --profiling argument is set to false
4949
profiling: "false"
50-
files:
51-
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
52-
- content: |
53-
apiVersion: apiserver.config.k8s.io/v1
54-
kind: AdmissionConfiguration
55-
plugins:
56-
- name: EventRateLimit
57-
path: /etc/kubernetes/admission/eventRateLimit.yaml
58-
path: /etc/kubernetes/admission/admissionConfiguration.yaml
59-
permissions: "0600"
60-
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
61-
- content: |
62-
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
63-
kind: Configuration
64-
limits:
65-
- type: Server
66-
qps: 5000
67-
burst: 20000
68-
- type: Namespace
69-
qps: 500
70-
burst: 2000
71-
cacheSize: 1000
72-
- type: User
73-
qps: 100
74-
burst: 400
75-
cacheSize: 2000
76-
- type: SourceAndObject
77-
qps: 50
78-
burst: 100
79-
cacheSize: 5000
80-
path: /etc/kubernetes/admission/eventRateLimit.yaml
81-
permissions: "0600"
50+
8251
initConfiguration:
8352
nodeRegistration:
8453
kubeletExtraArgs:
@@ -97,10 +66,4 @@ spec:
9766
#helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
9867
#If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
9968
rotate-server-certificates: "true"
100-
postKubeadmCommands:
101-
- echo export KUBECONFIG=/etc/kubernetes/admin.conf >> /root/.bashrc
102-
- echo "after kubeadm call" > /var/log/postkubeadm.log
103-
#4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive
104-
- chmod 600 "$(systemctl show -P FragmentPath kubelet.service)"
105-
- chmod 600 /var/lib/kubelet/config.yaml
106-
- chmod 600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
69+

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/harden.sh‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
3+
# List all the KubeadmControlPlaneTemplates to find the latest version
4+
echo "Listing all KubeadmControlPlaneTemplates:"
5+
kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io
6+
7+
# Prompt for the NKP version
8+
echo "Please enter the NKP version from the list above (e.g., for NKP version nkp-nutanix-v2.14.0, enter v2.14.0):"
9+
read -p "> " VERSION
10+
11+
# Validate version format
12+
if [[ ! $VERSION =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
13+
echo "Invalid version format. Please use the format v<major>.<minor>.<patch> (e.g., v2.14.1)"
14+
exit 1
15+
fi
16+
17+
echo "Using NKP version: $VERSION"
18+
19+
# Clone the Latest KubeadmControlplaneTemplate
20+
echo "Cloning the template: nkp-nutanix-${VERSION}"
21+
kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io nkp-nutanix-${VERSION} -o yaml > nkp-nutanix-${VERSION}.yaml
22+
23+
# Replace <VERSION> with the actual version in all files
24+
echo "Replacing <VERSION> with ${VERSION} in all files..."
25+
sed -i "s/<VERSION>/${VERSION}/g" kustomization.yaml
26+
sed -i "s/<VERSION>/${VERSION}/g" cis-mitigations-cp-patch.yaml
27+
28+
echo "Replacement complete!"
29+
echo "Files have been updated with version: ${VERSION}"
30+
31+
echo "Applying kustomization to create hardened control plane template..."
32+
kubectl apply -k .
33+
34+
echo "You can now patch the ClusterClass to use the Hardened KubeadmControlPlaneTemplates"
35+
echo "Here are the available ClusterClass"
36+
37+
kubectl get clusterclasses.cluster.x-k8s.io
38+
39+
echo "Run the below command after replacing the <CLUSTER_CLASS> with the ClusterClass in use from the list above"
40+
echo "kubectl patch clusterclass <CLUSTER_CLASS> \\
41+
--type merge -p='{\"spec\":{\"controlPlane\":{\"ref\":{\"name\":\"nkp-nutanix-${VERSION}-hardened\"}}}}'"

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/kustomization.yaml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,112 @@ resources:
66

77
patches:
88
- path: cis-mitigations-cp-patch.yaml
9+
- target:
10+
group: controlplane.cluster.x-k8s.io
11+
version: v1beta1
12+
kind: KubeadmControlPlaneTemplate
13+
#target <2.15.0
14+
name: nkp-nutanix-v2.1[0-4].*
15+
patch: |
16+
#so that it works in v2.14.0. Make sure to remove after to 2.14.0
17+
- op: add
18+
path: /spec/template/spec/kubeadmConfigSpec/files
19+
value: []
20+
- op: add
21+
path: /spec/template/spec/kubeadmConfigSpec/files/-
22+
value:
23+
path: /etc/kubernetes/admission/admissionConfiguration.yaml
24+
permissions: "0600"
25+
content: |
26+
apiVersion: apiserver.config.k8s.io/v1
27+
kind: AdmissionConfiguration
28+
plugins:
29+
- name: EventRateLimit
30+
path: /etc/kubernetes/admission/eventRateLimit.yaml
31+
- op: add
32+
path: /spec/template/spec/kubeadmConfigSpec/files/-
33+
value:
34+
path: /etc/kubernetes/admission/eventRateLimit.yaml
35+
permissions: "0600"
36+
content: |
37+
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
38+
kind: Configuration
39+
limits:
40+
- type: Server
41+
qps: 5000
42+
burst: 20000
43+
- type: Namespace
44+
qps: 500
45+
burst: 2000
46+
cacheSize: 1000
47+
- type: User
48+
qps: 100
49+
burst: 400
50+
cacheSize: 2000
51+
- type: SourceAndObject
52+
qps: 50
53+
burst: 100
54+
cacheSize: 5000
55+
- op: add
56+
path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
57+
value: 'chmod 600 "$(systemctl show -P FragmentPath kubelet.service)"'
58+
- op: add
59+
path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
60+
value: 'chmod 600 /var/lib/kubelet/config.yaml'
61+
- op: add
62+
path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
63+
value: 'chmod 600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf'
964
65+
- target:
66+
group: controlplane.cluster.x-k8s.io
67+
version: v1beta1
68+
kind: KubeadmControlPlaneTemplate
69+
#target only NKP => v2.15.0
70+
name: nkp-nutanix-v2.1[5-9].*
71+
patch: |
72+
- op: add
73+
path: /spec/template/spec/kubeadmConfigSpec/files/-
74+
value:
75+
path: /etc/kubernetes/admission/admissionConfiguration.yaml
76+
permissions: "0600"
77+
content: |
78+
apiVersion: apiserver.config.k8s.io/v1
79+
kind: AdmissionConfiguration
80+
plugins:
81+
- name: EventRateLimit
82+
path: /etc/kubernetes/admission/eventRateLimit.yaml
83+
- op: add
84+
path: /spec/template/spec/kubeadmConfigSpec/files/-
85+
value:
86+
path: /etc/kubernetes/admission/eventRateLimit.yaml
87+
permissions: "0600"
88+
content: |
89+
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
90+
kind: Configuration
91+
limits:
92+
- type: Server
93+
qps: 5000
94+
burst: 20000
95+
- type: Namespace
96+
qps: 500
97+
burst: 2000
98+
cacheSize: 1000
99+
- type: User
100+
qps: 100
101+
burst: 400
102+
cacheSize: 2000
103+
- type: SourceAndObject
104+
qps: 50
105+
burst: 100
106+
cacheSize: 5000
107+
- op: add
108+
path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
109+
value: 'chmod 600 "$(systemctl show -P FragmentPath kubelet.service)"'
110+
- op: add
111+
path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
112+
value: 'chmod 600 /var/lib/kubelet/config.yaml'
113+
- op: add
114+
path: /spec/template/spec/kubeadmConfigSpec/postKubeadmCommands/-
115+
value: 'chmod 600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf'
10116
namePrefix: ""
11117
nameSuffix: "-hardened"

0 commit comments

Comments
 (0)