vmware images are using a removed argument for api-server cloud-provider=external

[macintoshme]

Image I'm using:
bottlerocket-v1.33.5-kubernetes-1-33-eks-17-amd64-a5ca614

What I expected to happen:
(EKSA) Update to go as expected, with all services starting

What actually happened:
Found node wouldn't become ready, looked at the pods and found an error in the logs:

macintoshme@werk  ~  k logs kube-apiserver-dvlp-workload-bdb7h        
Error: unknown flag: --cloud-provider

How to reproduce the problem:
I am very surprised no one else has run into this as it appears to affect all vmware 1.33 clusters:

Found what appears to be the offending lines:

bottlerocket/sources/shared-defaults/kubernetes-vmware.toml

Line 7 in 79b1ece

cloud-provider = "external"

bottlerocket/sources/shared-defaults/kubernetes-aws-external-cloud-provider.toml

Line 2 in 79b1ece

cloud-provider = "external"

This was removed in 1.31, but I think it wasn't an issue until they actually removed it from kubernetes later.

[bcressey]

Bottlerocket doesn't package the k8s control plane software like kube-apiserver and etcd, those are configured by EKS-A.

[bcressey]

On the Bottlerocket side, the cloud-provider field is only used as part of the kubelet-exec-start-conf template, which controls how kubelet starts.

It doesn't propagate upwards from there into pods, or at least it shouldn't!

[macintoshme]

You're thinking this may be an issue where kubelet is passing the cloud provider argument provided by the template and spawning the additional argument?

I do see in documentation where it says it shouldn't when set to empty.

[bcressey]

kubelet wouldn't and can't really do that.

It'd have to be a flow where the EKS-A cluster bootstrap code used the value of that Bottlerocket setting when configuring the apiserver static pod, or something like that.

It's definitely a bug on the EKS-A side.

[nclh-edwrodriguez]

Leaving this here for folks that might encounter this issue. Until an official fix comes out, the following workaround fixes the issue if you started an upgrade from 1.32 to 1.33. This mainly applies to clusters using bottlerocket OS and its the only way you will be able to change the api-server kubelet configuration before node bootstrapping.

Edit the kubeadmcontrolplanes object for your cluster.

k edit kubeadmcontrolplanes -n eksa-system

And remove the line with the part cloud-provider: external in the apiServer extraArgs section. After that you upgrade will kickoff a fresh control plane node without the removed cloud-provider flag.

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          audit-log-maxage: "30"
          audit-log-maxbackup: "10"
          audit-log-maxsize: "512"
          cloud-provider: external # Line to remove