Trivial bottlerocket NodePool filters out all instance types #8535
Description
Description
We've observed this behavior with both the fips and non-fips bottlerocket AMIs.
We are operating in GovCloud us-gov-west-1.
Observed Behavior:
The EC2NodeClass finds the fips bottlerocket AMI as expected.
kubectl get ec2nodeclass worker-0 -o yaml
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: worker-0
spec:
amiFamily: Bottlerocket
amiSelectorTerms:
- name: bottlerocket-aws-k8s-1.30-fips-x86_64-*
blockDeviceMappings:
- deviceName: /dev/xvdb
ebs:
deleteOnTermination: true
encrypted: true
iops: 3000
throughput: 125
volumeSize: 100Gi
volumeType: gp3
- deviceName: /dev/xvda
ebs:
deleteOnTermination: true
encrypted: true
iops: 3000
throughput: 125
volumeSize: 4Gi
volumeType: gp3
rootVolume: true
kubelet:
cpuCFSQuota: true
evictionHard:
memory.available: 5%
nodefs.available: 10%
nodefs.inodesFree: 10%
evictionMaxPodGracePeriod: 60
evictionSoft:
memory.available: 500Mi
nodefs.available: 15%
nodefs.inodesFree: 15%
evictionSoftGracePeriod:
memory.available: 1m0s
nodefs.available: 1m30s
nodefs.inodesFree: 2m0s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
kubeReserved:
cpu: 200m
ephemeral-storage: 3Gi
memory: 100Mi
systemReserved:
cpu: 100m
ephemeral-storage: 1Gi
memory: 100Mi
metadataOptions:
httpEndpoint: enabled
httpProtocolIPv6: disabled
httpPutResponseHopLimit: 1
httpTokens: required
role: gs-284986962550-cluster-0-karpenter-instance-profile
securityGroupSelectorTerms:
- tags:
aws:eks:cluster-name: gs-X-cluster-0
subnetSelectorTerms:
- tags:
region: us-gov-west-1
visibility: private
tags:
nominal.io:environment: staging
nominal.io:kube_cluster_name: gs-X-cluster-0
userData: |
[settings.kubernetes]
cluster-name = "gs-X-cluster-0"
status:
amis:
- id: ami-018937f1eb9700738
name: bottlerocket-aws-k8s-1.30-fips-x86_64-v1.47.0-6154605b
requirements:
- key: kubernetes.io/arch
operator: In
values:
- amd64
conditions:
- lastTransitionTime: "2025-02-21T22:24:58Z"
message: ""
observedGeneration: 6
reason: AMIsReady
status: "True"
type: AMIsReady
- lastTransitionTime: "2025-02-21T22:24:58Z"
message: ""
observedGeneration: 6
reason: SubnetsReady
status: "True"
type: SubnetsReady
- lastTransitionTime: "2025-02-21T22:24:58Z"
message: ""
observedGeneration: 6
reason: SecurityGroupsReady
status: "True"
type: SecurityGroupsReady
- lastTransitionTime: "2025-02-21T22:24:58Z"
message: ""
observedGeneration: 6
reason: InstanceProfileReady
status: "True"
type: InstanceProfileReady
- lastTransitionTime: "2025-02-21T22:24:58Z"
message: ""
observedGeneration: 6
reason: ValidationSucceeded
status: "True"
type: ValidationSucceeded
- lastTransitionTime: "2025-09-26T00:10:29Z"
message: ""
observedGeneration: 6
reason: Ready
status: "True"
type: Ready
instanceProfile: gs-X-cluster-0_3296502207782327165
securityGroups:
- id: sg-07596ad3ede0ce142
name: eks-cluster-sg-gs-X-cluster-0-1824486891
subnets:
- id: subnet-08be51988159408b8
zone: us-gov-west-1a
zoneID: usgw1-az1
- id: subnet-01e159d6b68591292
zone: us-gov-west-1b
zoneID: usgw1-az2
- id: subnet-002988ac8209fd30f
zone: us-gov-west-1c
zoneID: usgw1-az3
NodePool spec.
kubectl get nodepool worker-bottlerocket -o yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: worker-0
spec:
disruption:
budgets:
- nodes: 10%
reasons:
- Underutilized
- nodes: 10%
reasons:
- Drifted
- Empty
consolidateAfter: 1h
consolidationPolicy: WhenEmptyOrUnderutilized
template:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
expireAfter: 720h
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: worker-0
requirements:
- key: kubernetes.io/os
operator: In
values:
- linux
terminationGracePeriod: 48h
status:
conditions:
- lastTransitionTime: "2025-09-26T00:17:29Z"
message: ""
observedGeneration: 7
reason: NodeClassReady
status: "True"
type: NodeClassReady
- lastTransitionTime: "2025-09-26T00:17:29Z"
message: ""
observedGeneration: 7
reason: ValidationSucceeded
status: "True"
type: ValidationSucceeded
- lastTransitionTime: "2025-09-26T00:17:29Z"
message: object is awaiting reconciliation
observedGeneration: 7
reason: AwaitingReconciliation
status: Unknown
type: NodeRegistrationHealthy
- lastTransitionTime: "2025-09-26T00:23:14Z"
message: ""
observedGeneration: 7
reason: Ready
status: "True"
type: Ready
resources:
cpu: "0"
ephemeral-storage: "0"
memory: "0"
nodes: "0"
pods: "0"
No instance types can be found.
{"level":"INFO","time":"2025-09-26T00:29:54.559Z","logger":"controller","message":"skipping, nodepool requirements filtered out all instance types","commit":"9458bb5","controller":"provisioner","namespace":"","name":"","reconcileID":"99e66691-7b14-4601-b0d1-4d7f169be096","NodePool":{"name":"worker-0"}}
Expected Behavior:
Instance types are found and provisioned for pending pods.
Reproduction Steps (Please include YAML):
Versions:
- Chart Version: 1.5.0
- Kubernetes Version (
kubectl version):
kubectl version
Client Version: v1.30.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.14-eks-b707fbb
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment