eks version upgrade to 1.34 caused flux reconcilation issues and flux-system namespace is completely broke.

[ravindrainduri]

Hi,

Recently i've upgraded my eks cluster version to 1.34 , after the upgrade karpenter has got some issues, while trouble shooting my karpenter which is in v0.29.2 not compatible with eks 1.34 , I tried to upgrade the crd's and a mismatch happened, which in turn caused flux crd mismatch and broke the entire flux.

Kindly let me know if there's a way to bring back flux and karpenter and make it work as expected in gitops style to latest eks 1.34 version.

Kindly suggest.

[stefanprodan]

Flux shouldn't be affected by any cluster addon upgrade failures. You should have the addons be deployed with dedicated Flux Kustomizations, this way you can always recover via Git. What is your repo structure and what exactly "caused flux crd mismatch" means? Please post the flux get all -A.

[ravindrainduri]

this is my output

and also i've updated my gotk-components.yaml to v2.7.5 is that an issue of this broken flux state and also this my flux-system pods

$ k get pods -n flux-system
NAME READY STATUS RESTARTS AGE
helm-controller-77648d7c85-dpbg8 1/1 Running 0 10d
image-automation-controller-74cb7f7565-7dp2p 0/1 CrashLoopBackOff 3634 (3m11s ago) 12d
image-reflector-controller-c7f6c7947-fngh2 0/1 CrashLoopBackOff 3635 (23s ago) 12d
kustomize-controller-8df48cb8c-2fmls 1/1 Running 0 10d
notification-controller-6d766f87cf-mx9c8 1/1 Running 0 10d
source-controller-6679d8bdb-27trf 1/1 Running 0 10d
and this is my description of image-automation-controller

$ k describe pods image-automation-controller-74cb7f7565-7dp2p -n flux-system
Name: image-automation-controller-74cb7f7565-7dp2p
Namespace: flux-system
Priority: 0
Service Account: image-automation-controller
Node: ip-172-16-13-103.ec2.internal/172.16.13.103
Start Time: Wed, 03 Dec 2025 09:24:15 +0530
Labels: app=image-automation-controller
app.kubernetes.io/component=image-automation-controller
app.kubernetes.io/instance=flux-system
app.kubernetes.io/part-of=flux
pod-template-hash=74cb7f7565
Annotations: prometheus.io/port: 8080
prometheus.io/scrape: true
Status: Running
IP: 172.16.4.214
IPs:
IP: 172.16.4.214
Controlled By: ReplicaSet/image-automation-controller-74cb7f7565
Containers:
manager:
Container ID: containerd://de31a32da8cab801b7a03e703c247be9692e8d2dda4dba013fc4cdcfb2a75d90
Image: ghcr.io/fluxcd/image-automation-controller:v1.0.4
Image ID: ghcr.io/fluxcd/image-automation-controller@sha256:f9383dccb80ec65e274648941af623ce74084d25026e14389111c14b630efece
Ports: 8080/TCP (http-prom), 9440/TCP (healthz)
Host Ports: 0/TCP (http-prom), 0/TCP (healthz)
SeccompProfile: RuntimeDefault
Args:
--events-addr=http://notification-controller.flux-system.svc.cluster.local./
--watch-all-namespaces
--log-level=info
--log-encoding=json
--enable-leader-election
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 16 Dec 2025 07:02:17 +0530
Finished: Tue, 16 Dec 2025 07:02:17 +0530
Ready: False
Restart Count: 3635
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 64Mi
Liveness: http-get http://:healthz/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:healthz/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
RUNTIME_NAMESPACE: flux-system (v1:metadata.namespace)
GOMEMLIMIT: 1073741824 (limits.memory)
Mounts:
/tmp from temp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l8lkc (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
temp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-l8lkc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning BackOff 4m20s (x91048 over 12d) kubelet Back-off restarting failed container manager in pod image-automation-controller-74cb7f7565-7dp2p_flux-system(ad5e8e01-8bb8-4498-bf6a-c69cf49c9027)
Normal Pulled 91s (x3635 over 12d) kubelet Container image "ghcr.io/fluxcd/image-automation-controller:v1.0.4" already present on machine

k get kustomizations -A
NAMESPACE NAME AGE READY STATUS
flux-system dependencies 228d False HelmRelease/flux-system/karpenter-crd dry-run failed: no matches for kind "HelmRelease" in version "helm.toolkit.fluxcd.io/v2"...
flux-system flux-system 11d False CustomResourceDefinition/imagepolicies.image.toolkit.fluxcd.io dry-run failed (Invalid): CustomResourceDefinition.apiextensions.k8s.io "imagepolicies.image.toolkit.fluxcd.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersions...
flux-system track-bps-batch-dev 228d False dependency 'flux-system/dependencies' is not ready
flux-system track-bps-dev 228d False dependency 'flux-system/dependencies' is not ready

can you pls check and help me further.

[stefanprodan]

You need to follow the upgrade guide: #5572