Windows nodes occasionally fail to join clusters

[nojnhuh]

Which jobs are flaky:

https://storage.googleapis.com/k8s-triage/index.html?pr=1&text=Timed%20out%20waiting%20for%20%5Cd%2B%20(nodes%20to%20be%20created%20for%20MachineDeployment%7Cready%20replicas%20for%20MachinePool)%20.*-win&job=cluster-api-provider-azure

Which tests are flaky:

Testgrid link:

Reason for failure (if possible):

Appears to be a kubeadm failure.

Anything else we need to know:

Looking at a couple of failures:

• CAPI/CAPZ status shows infra succeeded, bootstrap completed (edit: looks like this actually fails despite the BootstrapSucceeded condition being set on the AzureMachine)
• No logs available for failed Windows machines

/kind flake

[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[nojnhuh]

• CAPI/CAPZ status shows infra succeeded, bootstrap completed

Actually I was only looking at the BootstrapSucceeded and VMRunning conditions on the AzureMachines, but I see status.vmState is Failed. These show up in the activity logs after the VM create request returns HTTP 201:

    "Properties": {
        "entity": "/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourcegroups/capz-e2e-q40bhi/providers/Microsoft.Compute/virtualMachines/quick-sta-lvmn5",
        "eventCategory": "Administrative",
        "hierarchy": "d1aa7522-0959-442e-80ee-8c4f7fb4c184/46678f10-4bbb-447e-98e8-d2829589f2d8",
        "message": "Microsoft.Compute/virtualMachines/write",
        "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"OSProvisioningClientError\",\"message\":\"OS Provisioning for VM 'quick-sta-lvmn5' did not finish in the allotted time. However, the VM guest agent was detected running. This suggests the guest OS has not been properly prepared to be used as a VM image (with CreateOption=FromImage). To resolve this issue, either use the VHD as is with CreateOption=Attach or prepare it properly for use as an image:\\r\\n * Instructions for Windows: https://learn.microsoft.com/azure/virtual-machines/windows/prepare-for-upload-vhd-image\\r\\n * Instructions for Linux: https://learn.microsoft.com/azure/virtual-machines/linux/create-upload-generic \"}]}}"
    },

    "Properties": {
        "cause": "PlatformInitiated",
        "currentHealthStatus": "Available",
        "details": "The Virtual Machine provisioning has failed due to Guest operating system issues or incorrect user run scripts. You can attempt to either re-create this Virtual Machine or if this Virtual Machine is part of a virtual machine scale set, you can try reimaging it.",
        "previousHealthStatus": "Available",
        "title": "Provisioning failure",
        "type": "Downtime"
    },

[jsturtevant]

this looks like kubernetes-sigs/windows-testing#496, our upstream sig-windows tests we have the following mitigation: kubernetes-sigs/windows-testing#498

[nojnhuh]

/assign

[nojnhuh]

#5708 removes the Windows nodes from the HA control plane test, but that's also the only place where Windows VM-backed load balancer tests are run. The VMSS test will still create a load balancer backed by a Windows VMSS, but we should make sure in whatever reorganization we do that we bring back a Windows VM-based load balancer test.

[nojnhuh]

The low-hanging fruit here is done: #5708, kubernetes/test-infra#35008

I don't think I'll have time soon to follow up and reorganize the tests more. Some random ideas that could be mixed and matched or remixed:

• Move all tests running Windows nodes to the -optional presubmit job
• Split the tests currently running Windows nodes into "with Windows" and "without Windows"
• Create dedicated -flaky periodic and presubmit jobs where we could move tests using Windows nodes

/unassign