Panic on startup with `csi-proxy` version `v1.3.0`/`v2.0.0-alpha.2` in CSI Driver

[ConnorJC3]

What happened:

Hi, I'm a maintainer on the EBS CSI Driver. We recently tried to bump our version of the csi-proxy library from v1.2.1 to v1.3.0. Upon testing this bump, we find the new WMI library included causes our driver to repeatedly crash on startup with a panic:

$ kubectl -n kube-system logs -f ebs-csi-node-windows-lxfxq             
Defaulted container "ebs-plugin" out of: ebs-plugin, node-driver-registrar, liveness-probe
panic: couldn't initialize the WmiSessionManager: Invalid class string

goroutine 1 [running]:
github.com/microsoft/wmi/pkg/wmiinstance.NewWmiSessionManager()
        /go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/wmi/pkg/wmiinstance/WmiSessionManager.go:32 +0x6b
github.com/microsoft/wmi/pkg/base/session.init.0()
        /go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/wmi/pkg/base/session/session.go:28 +0x6d

What you expected to happen:

Upgrading the library shouldn't cause the driver to panic.

How to reproduce it:

Working on getting easy repro steps, but attempting to use the csi-proxy in a CSI Driver on a eksctl-created Windows cluster should cause the issue.

Anything else we need to know?:

Unfortunately, this comes from a panic here, which is in a function run via init() here. So there's a good chance we'll need to report this upstream.

Environment:

• CSI Driver version: N/A
• Kubernetes version (use kubectl version): 1.33 (doesn't seem to matter)
• OS (e.g. from /etc/os-release): Windows Server 2022 Core (Official EKS AMI ami-073ce66bebaa6242b in us-west-2)
• Kernel (e.g. uname -a): N/A
• Install tools: N/A
• Others: N/A

[mauriciopoppe]

Thanks for the report and sorry I missed it. I'm going to mark 1.3.0 as having a known issue and move the latest release back to 1.2.1 in the meantime.

cc @laozc

[laozc]

/assign
Hi @ConnorJC3 Could you share the setup details and the branch which you integrated the latest version?

[ConnorJC3]

Ok, sorry for the delay, got sucked into other high priority items. Doing some research on this:

• The issue affects v1.3.0 and v2.0.0-alpha.2
• However, in v1.3.0 the client package specifically is safe because it has its own go.mod: https://github.com/kubernetes-csi/csi-proxy/blob/v1.3.0/client/go.mod
• This probably means that many CSI drivers are safe, as a CSI driver can safely upgrade to if it meets BOTH of these criteria:
  • It does not use the v2 package (or at minimum stays on v2.0.0-alpha.1 and does not upgrade)
  • For v1, it only imports github.com/kubernetes-csi/csi-proxy/client/... and does NOT import anything from the pkg directory

In our case, we implement the v2 functionality. Normally, that functionality is unused and gated behind a CLI flag, but unfortunately this breaks us even when the functionality is disabled as the init() function runs no matter what and triggers the panic.

@laozc if it helps:
Here's a simplistic reproduction case that doesn't use CSI at all (see the README): https://github.com/ConnorJC3/csi-proxy-panic-repro/

Here's an example of it happening to a real CSI driver in CI: kubernetes-sigs/aws-ebs-csi-driver#2714

[laozc]

Hi @ConnorJC3 , was the EBS driver running in non Host Process Container mode?

It could be the reason why the driver could not initialize WMI because of the restricted environment.
Changes in both v1.3.0 and v2.0.0-alpha.2 branch requires direct access to WMI interface which can only be done via HPC when the program is running as a Job on Windows nodes.
In the old releases, the library initialisation is done via the .NET framework running in PowerShell process so the driver may run without extra permissions.

@mauriciopoppe I think you have a KEP for the initiative to move all CSI drivers to running as HPC on Windows node?

@ConnorJC3 Could you elaborate more on the details of the EBS driver design? Let me know if you need help on migrating the driver to HPC to align with other CSI drivers.

[mauriciopoppe]

Yeah, the KEP main tracker is kubernetes/enhancements#3636.