Skip to content

Commit 9006eb2

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #391 from meomnzak/feat/support-kubelet-config
feat: add kubeletConfiguration support to ibmNodeClass
2 parents 437e01d + d007be4 commit 9006eb2

File tree

21 files changed

+818
-292
lines changed

21 files changed

+818
-292
lines changed

‎docs/bootstrap-methods.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,78 @@ spec:
121121
122122
### Customization Options
123123
124+
### Kubelet Configuration Overrides
125+
126+
For VPC bootstrap, you can override a subset of kubelet configuration directly from the `IBMNodeClass`. These settings are rendered into the kubelet `config.yaml` on the node and validated by the CRD.
127+
128+
```yaml
129+
apiVersion: karpenter-ibm.sh/v1alpha1
130+
kind: IBMNodeClass
131+
metadata:
132+
name: vpc-bootstrap-kubelet
133+
spec:
134+
region: us-south
135+
zone: us-south-1
136+
vpc: "r006-a8efb117-fd5e-4f63-ae16-4fb9faafa4ff"
137+
image: ubuntu-24-04-amd64
138+
apiServerEndpoint: "https://10.240.0.1:6443"
139+
bootstrapMode: cloud-init
140+
securityGroups:
141+
- "r006-12345678-1234-1234-1234-123456789012"
142+
143+
kubelet:
144+
clusterDNS:
145+
- 10.96.0.10
146+
- 10.96.0.11
147+
148+
maxPods: 150
149+
podsPerCore: 10
150+
151+
kubeReserved:
152+
cpu: "200m"
153+
memory: "512Mi"
154+
155+
systemReserved:
156+
cpu: "100m"
157+
memory: "256Mi"
158+
159+
evictionHard:
160+
memory.available: "500Mi"
161+
162+
evictionSoft:
163+
memory.available: "1Gi"
164+
165+
evictionSoftGracePeriod:
166+
memory.available: "1m0s"
167+
168+
evictionMaxPodGracePeriod: 120
169+
170+
imageGCHighThresholdPercent: 85
171+
imageGCLowThresholdPercent: 70
172+
173+
cpuCFSQuota: true
174+
175+
```
176+
* **Reserved resources**
177+
* `systemReserved` and `kubeReserved` keys must be one of:
178+
`cpu`, `memory`, `ephemeral-storage`, `pid`.
179+
* Values must not be negative (strings starting with `-` are rejected).
180+
181+
* **Eviction settings**
182+
* `evictionHard`, `evictionSoft`, and `evictionSoftGracePeriod` may only use:
183+
`memory.available`, `nodefs.available`, `nodefs.inodesFree`,
184+
`imagefs.available`, `imagefs.inodesFree`, `pid.available`.
185+
* Every key in `evictionSoft` must exist in `evictionSoftGracePeriod`.
186+
* Every key in `evictionSoftGracePeriod` must exist in `evictionSoft`.
187+
188+
* **Image garbage collection**
189+
* `imageGCHighThresholdPercent` and `imageGCLowThresholdPercent` must be in
190+
the range `[0, 100]`.
191+
* If both are set, `imageGCLowThresholdPercent` must be **less than**
192+
`imageGCHighThresholdPercent`.
193+
194+
For a complete reference, see `examples/kubelet-configuration.yaml`.
195+
124196
#### **Custom User Data**
125197
```yaml
126198
spec:

‎examples/kubelet-configuration.yaml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
---
2+
# Example: IBMNodeClass with kubelet configuration overrides
3+
apiVersion: karpenter-ibm.sh/v1alpha1
4+
kind: IBMNodeClass
5+
metadata:
6+
name: kubelet-config-nodeclass
7+
spec:
8+
# Basic VPC-backed configuration
9+
region: us-south
10+
zone: us-south-1
11+
vpc: "r006-a8efb117-fd5e-4f63-ae16-4fb9faafa4ff"
12+
subnet: "0717-197e06f4-b500-426c-bc0f-900b215f996c"
13+
14+
apiServerEndpoint: "https://10.240.0.1:6443"
15+
bootstrapMode: cloud-init
16+
17+
securityGroups:
18+
- "r006-12345678-1234-1234-1234-123456789012"
19+
20+
image: "r006-dd3c20fa-71d3-4dc0-913f-2f097bf3e500"
21+
22+
sshKeys:
23+
- "r006-82091c89-68e4-4b3f-bd2b-4e63ca2f67da"
24+
25+
resourceGroup: "88427352321742ef8cfac50b0ee6cc26"
26+
27+
# Kubelet configuration (subset of upstream KubeletConfiguration)
28+
kubelet:
29+
# DNS for the cluster (overrides CLUSTER_DNS from bootstrap)
30+
clusterDNS:
31+
- 10.96.0.10
32+
33+
# Pod density settings
34+
maxPods: 150
35+
podsPerCore: 10
36+
37+
# Reserved resources for Kubernetes components
38+
kubeReserved:
39+
cpu: "200m"
40+
memory: "512Mi"
41+
42+
# Reserved resources for OS/system daemons
43+
systemReserved:
44+
cpu: "100m"
45+
memory: "256Mi"
46+
47+
# Eviction thresholds
48+
evictionHard:
49+
memory.available: "500Mi"
50+
51+
evictionSoft:
52+
memory.available: "1Gi"
53+
54+
evictionSoftGracePeriod:
55+
memory.available: 1m0s
56+
57+
evictionMaxPodGracePeriod: 120
58+
59+
# Image garbage collection thresholds
60+
# Validation enforces 0 <= low < high <= 100
61+
imageGCHighThresholdPercent: 85
62+
imageGCLowThresholdPercent: 70
63+
64+
# Enforce CPU CFS quota for pods that specify CPU limits
65+
cpuCFSQuota: true
66+
---
67+
apiVersion: karpenter.sh/v1
68+
kind: NodePool
69+
metadata:
70+
name: kubelet-config-nodepool
71+
spec:
72+
template:
73+
metadata:
74+
labels:
75+
app.kubernetes.io/managed-by: karpenter
76+
example: kubelet-config
77+
spec:
78+
nodeClassRef:
79+
apiVersion: karpenter-ibm.sh/v1alpha1
80+
kind: IBMNodeClass
81+
name: kubelet-config-nodeclass
82+
requirements:
83+
- key: "kubernetes.io/arch"
84+
operator: In
85+
values: ["amd64"]
86+
- key: "kubernetes.io/os"
87+
operator: In
88+
values: ["linux"]
89+
- key: "node.kubernetes.io/instance-type"
90+
operator: In
91+
values: ["bx2-4x16", "bx2-8x32"]
92+
disruption:
93+
consolidationPolicy: WhenEmpty
94+
consolidateAfter: 30s
95+
limits:
96+
cpu: "1000"

‎pkg/apis/v1alpha1/ibmnodeclass_types.go‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,77 @@ type BlockDeviceMapping struct {
253253
RootVolume bool `json:"rootVolume,omitempty"`
254254
}
255255

256+
// KubeletConfiguration defines args to be used when configuring kubelet on provisioned nodes.
257+
type KubeletConfiguration struct {
258+
// ClusterDNS is a list of IP addresses for the cluster DNS server.
259+
// +optional
260+
ClusterDNS []string `json:"clusterDNS,omitempty"`
261+
262+
// MaxPods is an override for the maximum number of pods that can run on a worker node instance.
263+
// +kubebuilder:validation:Minimum:=0
264+
// +optional
265+
MaxPods *int32 `json:"maxPods,omitempty"`
266+
267+
// PodsPerCore is an override for the number of pods that can run on a worker node
268+
// instance based on the number of cpu cores.
269+
// +kubebuilder:validation:Minimum:=0
270+
// +optional
271+
PodsPerCore *int32 `json:"podsPerCore,omitempty"`
272+
273+
// SystemReserved contains resources reserved for OS system daemons and kernel memory.
274+
// +kubebuilder:validation:XValidation:message="valid keys for systemReserved are ['cpu','memory','ephemeral-storage','pid']",rule="self.all(x, x=='cpu' || x=='memory' || x=='ephemeral-storage' || x=='pid')"
275+
// +kubebuilder:validation:XValidation:message="systemReserved value cannot be a negative resource quantity",rule="self.all(x, !self[x].startsWith('-'))"
276+
// +optional
277+
SystemReserved map[string]string `json:"systemReserved,omitempty"`
278+
279+
// KubeReserved contains resources reserved for Kubernetes system components.
280+
// +kubebuilder:validation:XValidation:message="valid keys for kubeReserved are ['cpu','memory','ephemeral-storage','pid']",rule="self.all(x, x=='cpu' || x=='memory' || x=='ephemeral-storage' || x=='pid')"
281+
// +kubebuilder:validation:XValidation:message="kubeReserved value cannot be a negative resource quantity",rule="self.all(x, !self[x].startsWith('-'))"
282+
// +optional
283+
KubeReserved map[string]string `json:"kubeReserved,omitempty"`
284+
285+
// EvictionHard is the map of signal names to quantities that define hard eviction thresholds
286+
// +kubebuilder:validation:XValidation:message="valid keys for evictionHard are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']",rule="self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'])"
287+
// +kubebuilder:validation:XValidation:message="evictionHard value cannot be a negative resource quantity",rule="self.all(x, !self[x].startsWith('-'))"
288+
// +optional
289+
EvictionHard map[string]string `json:"evictionHard,omitempty"`
290+
291+
// EvictionSoft is the map of signal names to quantities that define soft eviction thresholds
292+
// +kubebuilder:validation:XValidation:message="valid keys for evictionSoft are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']",rule="self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'])"
293+
// +kubebuilder:validation:XValidation:message="evictionSoft value cannot be a negative resource quantity",rule="self.all(x, !self[x].startsWith('-'))"
294+
// +optional
295+
EvictionSoft map[string]string `json:"evictionSoft,omitempty"`
296+
297+
// EvictionSoftGracePeriod is the map of signal names to quantities that define grace periods for each eviction signal
298+
// +kubebuilder:validation:XValidation:message="valid keys for evictionSoftGracePeriod are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']",rule="self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'])"
299+
// +optional
300+
EvictionSoftGracePeriod map[string]metav1.Duration `json:"evictionSoftGracePeriod,omitempty"`
301+
302+
// EvictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use when terminating pods in
303+
// response to soft eviction thresholds being met.
304+
// +kubebuilder:validation:Minimum:=0
305+
// +optional
306+
EvictionMaxPodGracePeriod *int32 `json:"evictionMaxPodGracePeriod,omitempty"`
307+
308+
// ImageGCHighThresholdPercent is the percent of disk usage after which image
309+
// garbage collection is always run.
310+
// +kubebuilder:validation:Minimum:=0
311+
// +kubebuilder:validation:Maximum:=100
312+
// +optional
313+
ImageGCHighThresholdPercent *int32 `json:"imageGCHighThresholdPercent,omitempty"`
314+
315+
// ImageGCLowThresholdPercent is the percent of disk usage before which image
316+
// garbage collection is never run.
317+
// +kubebuilder:validation:Minimum:=0
318+
// +kubebuilder:validation:Maximum:=100
319+
// +optional
320+
ImageGCLowThresholdPercent *int32 `json:"imageGCLowThresholdPercent,omitempty"`
321+
322+
// CPUCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits.
323+
// +optional
324+
CPUCFSQuota *bool `json:"cpuCFSQuota,omitempty"`
325+
}
326+
256327
// VolumeSpec defines IBM Cloud volume configuration
257328
type VolumeSpec struct {
258329
// Capacity is the volume size in gigabytes
@@ -507,6 +578,15 @@ type IBMNodeClassSpec struct {
507578
// +optional
508579
// +kubebuilder:validation:MaxItems=10
509580
BlockDeviceMappings []BlockDeviceMapping `json:"blockDeviceMappings,omitempty"`
581+
582+
// Kubelet defines args to be used when configuring kubelet on provisioned nodes.
583+
// They are a subset of the upstream types, recognizing not all options may be supported.
584+
// Wherever possible, the types and names should reflect the upstream kubelet types.
585+
// +optional
586+
// +kubebuilder:validation:XValidation:message="imageGCHighThresholdPercent must be greater than imageGCLowThresholdPercent",rule="has(self.imageGCHighThresholdPercent) && has(self.imageGCLowThresholdPercent) ? self.imageGCHighThresholdPercent > self.imageGCLowThresholdPercent : true"
587+
// +kubebuilder:validation:XValidation:message="evictionSoft key does not have a matching evictionSoftGracePeriod",rule="has(self.evictionSoft) ? self.evictionSoft.all(e, e in self.evictionSoftGracePeriod) : true"
588+
// +kubebuilder:validation:XValidation:message="evictionSoftGracePeriod key does not have a matching evictionSoft",rule="has(self.evictionSoftGracePeriod) ? self.evictionSoftGracePeriod.all(e, e in self.evictionSoft) : true"
589+
Kubelet *KubeletConfiguration `json:"kubelet,omitempty"`
510590
}
511591

512592
// IBMNodeClassStatus defines the observed state of IBMNodeClass

‎pkg/apis/v1alpha1/zz_generated.deepcopy.go‎

Lines changed: 90 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)