vSphere-CSI with Bottlerocket broken since v0.23.0

[mariuszadamski-ca]

What happened:

eks-anywhere v0.23.0 introduced Bottlerocket v1.40.0 image with the new kernel 6.12 (for Kubernetes 1.33), which is currently missing support for the VMware Paravirtual SCSI device, needed for vSphere CSI to work.

The problem with the image and the kernel configuration has also been reported to bottlerocket and bottlerocket-kernel-kit

What you expected to happen:

vSphere-CSI should be able to work correctly on the Kubernetes cluster deployed with eks-anywhere.

How to reproduce it (as minimally and precisely as possible):

• Deploy a Kubernetes cluster with Bottlerocket on vSphere with eks-anywhere v0.23.0 or v0.23.1
• Deploy vSphere-CSI on the cluster
• Deploy a pod with a PVC and observe that the PV fails to attach to the pod

Alternatively, without the vSphere-CSI driver, simply attach a new hard disk to one of the nodes using VMware Paravirtual SCSI device (which the CSI driver would effectively do) and observe that it is not detected by the system.

Anything else we need to know?:

Environment:

• EKS Anywhere Release: v0.23.0
• EKS Distro Release: 1-33-5
• Bottlerocket image: bottlerocket-v1.33.1-eks-d-1-33-5-eks-a-101-amd64.ova
• VMware v7.0.3.01900

[jmorenoc-o]

Thanks for reporting the issue, @mariuszadamski-ca.

I’m experiencing the same problem after upgrading EKS Anywhere from v0.22.7 to v0.23.1 using the following template:
bottlerocket-v1.33.1-kubernetes-1-33-eks-8-amd64-39a2431.

After upgrading multiple clusters, none of my Pods can use the vSphere VMDK volumes anymore.