Skip to content

Commit 6771cdb

Browse files
WinsonSouWinsonSou
committed
Included hardening steps
1 parent f2818d7 commit 6771cdb

File tree

6 files changed

+490
-0
lines changed

6 files changed

+490
-0
lines changed

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/README.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Hardened Kubernetes Control Plane Template
2+
3+
## Required Component: kubelet-csr-approver
4+
5+
To meet the following CIS security benchmarks:
6+
7+
- 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
8+
- 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
9+
10+
You must install the Postfinance kubelet-csr-approver:
11+
12+
```bash
13+
helm repo add kubelet-csr-approver https://postfinance.github.io/kubelet-csr-approver
14+
helm upgrade --install kubelet-csr-approver \
15+
kubelet-csr-approver/kubelet-csr-approver \
16+
-n kube-system \
17+
--create-namespace \
18+
--set maxExpirationSeconds=2592000 \
19+
--set leaderElection=true \
20+
--set bypassDnsResolution=true \
21+
--set rbac.create=true
22+
```
23+
24+
**Note**: If you choose not to install the kubelet-csr-approver, you must omit the flags related to the CIS benchmarks mentioned above from your configuration.
25+
26+
## Prerequisites
27+
28+
Before applying the kustomization, you need to determine the available Nutanix-provided KubeadmControlPlaneTemplates (those with the "nkp-" prefix) and their versions:
29+
30+
```bash
31+
# List all available KubeadmControlPlaneTemplates
32+
kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io
33+
34+
# Example output:
35+
# NAME AGE
36+
# nkp-nutanix-v2.14.0 4d
37+
# some-other-template 19h
38+
```
39+
40+
**Important**: You must use the Nutanix-provided template (starting with "nkp-") for these hardening configurations to work correctly.
41+
42+
Then export the original Nutanix KubeadmControlPlaneTemplate using the appropriate version:
43+
44+
```bash
45+
# Replace <VERSION> with your actual version (e.g., v2.14.0)
46+
kubectl get kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io nkp-nutanix-<VERSION> -o yaml > nkp-nutanix-<VERSION>.yaml
47+
```
48+
49+
## Structure
50+
51+
- `nkp-nutanix-<VERSION>.yaml` - The original KubeadmControlPlaneTemplate (generated by user, replace <VERSION> with your actual version)
52+
- `cis-mitigations-cp-patch.yaml` - Patch file containing CIS hardening configurations
53+
- `kustomization.yaml` - Kustomization file that applies the patch and renames the template
54+
55+
**Note**: You need to edit BOTH of these files with your actual version:
56+
57+
1. Edit the `cis-mitigations-cp-patch.yaml` file to replace `<VERSION>` with your actual version in the `metadata.name` field:
58+
59+
```yaml
60+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
61+
kind: KubeadmControlPlaneTemplate
62+
metadata:
63+
name: nkp-nutanix-<VERSION> # <-- Replace <VERSION> with your actual version
64+
```
65+
66+
2. Edit the `kustomization.yaml` file to replace `<VERSION>` in the resources section:
67+
68+
```yaml
69+
resources:
70+
- nkp-nutanix-<VERSION>.yaml # <-- Replace <VERSION> with your actual version
71+
```
72+
73+
## Applying the Kustomization
74+
75+
You can apply this kustomization directly with kubectl:
76+
77+
```bash
78+
#Ensure you are in a directory for the kustomization.yaml, cis-mitigations-cp-patch.yaml and the nkp-nutanix-<VERSION>.yaml file
79+
kubectl apply -k .
80+
```
81+
82+
## Verification
83+
84+
The kustomization will:
85+
86+
1. Take the user-generated KubeadmControlPlaneTemplate `nkp-nutanix-<VERSION>`
87+
2. Apply the CIS mitigations patch that adds:
88+
- API Server hardening (profiling disabled, service account lookup, admission plugins)
89+
- Controller Manager settings (terminated pod GC threshold, profiling disabled)
90+
- Scheduler settings (profiling disabled)
91+
- Required configuration files for EventRateLimit
92+
- Additional postKubeadmCommands for file permission hardening
93+
3. Rename it to `nkp-nutanix-<VERSION>-hardened`
94+
95+
## CIS Mitigations Applied
96+
97+
The following CIS mitigations are applied:
98+
99+
### API Server
100+
101+
- **1.2.15**: Disabled profiling for API server (`profiling: "false"`)
102+
- **1.2.21**: Enabled service account lookup (`service-account-lookup: "true"`)
103+
- **1.2.3, 1.2.9, 1.2.11, 1.2.14**: Added admission plugins:
104+
- AlwaysPullImages: Enforces that images are always pulled prior to starting containers
105+
- DenyServiceExternalIPs: Prevents services from using arbitrary external IPs
106+
- EventRateLimit: Mitigates event flooding attacks
107+
- NodeRestriction: Limits node access to specific APIs
108+
- **1.2.9**: Configured EventRateLimit with appropriate admission control config file
109+
- **1.2.5**: Set kubelet certificate authority (`kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt`)
110+
- **Note**: This requires kubelet-csr-approver to be installed. If not installed, this flag should be omitted.
111+
112+
### Controller Manager
113+
114+
- **1.3.1**: Set terminated pod GC threshold to 10000 for better garbage collection
115+
- **1.3.2**: Disabled profiling (`profiling: "false"`)
116+
- **1.3.6**: Enabled RotateKubeletServerCertificate feature gate (`feature-gates: RotateKubeletServerCertificate=true`)
117+
- **Note**: This requires kubelet-csr-approver to be installed. If not installed, this flag should be omitted.
118+
119+
### Scheduler
120+
121+
- **1.4.1**: Disabled profiling (`profiling: "false"`)
122+
123+
### Kubelet Configuration (Both Init and Join)
124+
125+
- **1.2.5, 1.3.6**: Enabled kubelet server certificate rotation (`rotate-server-certificates: "true"`)
126+
- **Note**: This requires kubelet-csr-approver to be installed. If not installed, this flag should be omitted.
127+
128+
### EventRateLimit Configuration
129+
130+
- **1.2.9**: Created admission configuration files with detailed rate limits:
131+
- Server-wide: 5000 QPS with 20000 burst
132+
- Namespace: 500 QPS with 2000 burst (1000 cache size)
133+
- User: 100 QPS with 400 burst (2000 cache size)
134+
- SourceAndObject: 50 QPS with 100 burst (5000 cache size)
135+
136+
### File Permissions
137+
138+
- **4.1.1**: Set appropriate file permissions (0600) for sensitive files including:
139+
- kubelet.service
140+
- kubelet config.yaml
141+
- 10-kubeadm.conf
142+
143+
## Applying to Cluster Class
144+
145+
After generating the hardened template, you need to update the cluster class to use it:
146+
147+
1. First, identify the cluster class name and version:
148+
149+
```bash
150+
kubectl get clusterclasses.cluster.x-k8s.io
151+
```
152+
153+
2. Patch the cluster class to use the hardened control plane template (replace `<VERSION>` and cluster class name as needed):
154+
155+
```bash
156+
kubectl patch clusterclass nkp-nutanix-<VERSION> \
157+
--type merge \
158+
-p='{"spec":{"controlPlane":{"ref":{"name":"nkp-nutanix-<VERSION>-hardened"}}}}'
159+
```

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/cis-mitigations-cp-patch.yaml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
2+
kind: KubeadmControlPlaneTemplate
3+
metadata:
4+
name: nkp-nutanix-<VERSION>
5+
spec:
6+
template:
7+
spec:
8+
kubeadmConfigSpec:
9+
clusterConfiguration:
10+
apiServer:
11+
extraArgs:
12+
#1.2.15 Ensure that the --profiling argument is set to false
13+
profiling: "false"
14+
#1.2.21 Ensure that the --service-account-lookup argument is set to true
15+
service-account-lookup: "true"
16+
#1.2.3 Ensure that the DenyServiceExternalIPs is set
17+
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
18+
#1.2.11 Ensure that the admission control plugin AlwaysPullImages is set
19+
#1.2.14 Ensure that the admission control plugin NodeRestriction is set
20+
enable-admission-plugins: AlwaysPullImages,EventRateLimit,DenyServiceExternalIPs,NodeRestriction
21+
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
22+
admission-control-config-file: /etc/kubernetes/admission/admissionConfiguration.yaml
23+
#1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
24+
#This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
25+
#helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
26+
#If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
27+
kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt
28+
extraVolumes:
29+
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
30+
- name: admission-config
31+
hostPath: /etc/kubernetes/admission
32+
mountPath: /etc/kubernetes/admission
33+
readOnly: true
34+
pathType: DirectoryOrCreate
35+
controllerManager:
36+
extraArgs:
37+
#1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate
38+
terminated-pod-gc-threshold: "10000"
39+
#1.3.2 Ensure that the --profiling argument is set to false
40+
profiling: "false"
41+
#1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
42+
#This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
43+
#helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
44+
#If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
45+
feature-gates: RotateKubeletServerCertificate=true
46+
scheduler:
47+
extraArgs:
48+
#1.4.1 Ensure that the --profiling argument is set to false
49+
profiling: "false"
50+
files:
51+
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
52+
- content: |
53+
apiVersion: apiserver.config.k8s.io/v1
54+
kind: AdmissionConfiguration
55+
plugins:
56+
- name: EventRateLimit
57+
path: /etc/kubernetes/admission/eventRateLimit.yaml
58+
path: /etc/kubernetes/admission/admissionConfiguration.yaml
59+
permissions: "0600"
60+
#1.2.9 Ensure that the admission control plugin EventRateLimit is set
61+
- content: |
62+
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
63+
kind: Configuration
64+
limits:
65+
- type: Server
66+
qps: 5000
67+
burst: 20000
68+
- type: Namespace
69+
qps: 500
70+
burst: 2000
71+
cacheSize: 1000
72+
- type: User
73+
qps: 100
74+
burst: 400
75+
cacheSize: 2000
76+
- type: SourceAndObject
77+
qps: 50
78+
burst: 100
79+
cacheSize: 5000
80+
path: /etc/kubernetes/admission/eventRateLimit.yaml
81+
permissions: "0600"
82+
initConfiguration:
83+
nodeRegistration:
84+
kubeletExtraArgs:
85+
#1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
86+
#1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
87+
#This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
88+
#helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
89+
#If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
90+
rotate-server-certificates: "true"
91+
joinConfiguration:
92+
nodeRegistration:
93+
kubeletExtraArgs:
94+
#1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate
95+
#1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true
96+
#This requires https://github.com/postfinance/kubelet-csr-approver to be installed. Install using the below helm command.
97+
#helm upgrade --install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system --create-namespace --set maxExpirationSeconds=2592000 --set leaderElection=true --set bypassDnsResolution=true --set rbac.create=true
98+
#If kubelet-csr-approver is not installed, ensure the below flag is omitted using a #
99+
rotate-server-certificates: "true"
100+
postKubeadmCommands:
101+
- echo export KUBECONFIG=/etc/kubernetes/admin.conf >> /root/.bashrc
102+
- echo "after kubeadm call" > /var/log/postkubeadm.log
103+
#4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive
104+
- chmod 600 "$(systemctl show -P FragmentPath kubelet.service)"
105+
- chmod 600 /var/lib/kubelet/config.yaml
106+
- chmod 600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

‎examples/capi-quick-start/nutanix-cluster-hardened-clusterclass/control-plane/kustomization.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
resources:
5+
- nkp-nutanix-<VERSION>.yaml
6+
7+
patches:
8+
- path: cis-mitigations-cp-patch.yaml
9+
10+
namePrefix: ""
11+
nameSuffix: "-hardened"

0 commit comments

Comments
 (0)