Make e2e node log collection more self-sufficient

[nojnhuh]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

Collecting logs from VMSS instances currently relies on CAPI having set nodeRefs for those instances in order to map those to the underlying Azure resources. This means that when bootstrapping a VMSS node fails and no nodeRef is produced on the MachinePool, CAPZ is unable to collect the logs that would help determine why that happened, as seen in this run.

These changes undercut CAPZ by relying on as little of its functionality as possible to determine how to reach nodes in order to collect logs. This should help us gather logs even when things fail (which is when we need them the most).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

TODOs:

• squashed commits
• includes documentation
• adds unit tests
• cherry-pick candidate

Release note:

NONE

[nojnhuh]

This PR is [WIP] while I make sure that VMSS instances that fail to bootstrap really do get logs now.

[nojnhuh]

logs!

Jun 18 05:38:36.422585 capz-e2e-d2pftj-vmss-mp-0000000 kubelet[1551]: E0618 05:38:36.422433    1551 run.go:72] "command failed" err="failed to parse kubelet flag: unknown flag: --youwillfailtobootstrappleaseandthankyou"

and on windows:

E0618 06:13:42.140420    2456 run.go:72] "command failed" err="failed to parse kubelet flag: unknown flag: --youwillfailtobootstrappleaseandthankyou"

Reverted these template changes.

/retitle Make e2e node log collection more self-sufficient

[nojnhuh]

I see the not-intentionally-broken run also successfully grabbed logs from the VMSS flex machine pool. I think that covers all the bases.

[nojnhuh]

/test pull-cluster-api-provider-azure-e2e-optional

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 52.83%. Comparing base (8aa9eca) to head (c29ee55).
Report is 23 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5710      +/-   ##
==========================================
- Coverage   52.84%   52.83%   -0.01%     
==========================================
  Files         278      278              
  Lines       29610    29610              
==========================================
- Hits        15647    15645       -2     
- Misses      13146    13148       +2     
  Partials      817      817              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nojnhuh]

/test pull-cluster-api-provider-azure-e2e-optional

[willie-yao]

Just left some comments after an initial review! Looks good overall, I might just need some clarification on the flow and maybe some code comments explaining the checks would be helpful.

[nojnhuh]

/hold for squash

[nojnhuh]

/hold for squash

(And also to double check the logs in CI to make sure I didn't break anything)

[willie-yao]

/retest

[willie-yao]

/lgtm
/assign @mboersma

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: a5c5986744c1a15939028eddc1453d0206911d7e

[nojnhuh]

/test pull-cluster-api-provider-azure-e2e-optional

[nojnhuh]

I broke VMSS flex log collection in the last commit but I think I fixed it now.

/test pull-cluster-api-provider-azure-e2e-optional

[nojnhuh]

Logs are looking good. Squashed!

/hold cancel

[willie-yao]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 10798ec52f9676a1512bcecc792393332f65cd7e

[mboersma]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment