Add clusterctl upgrade tests with AKS workload clusters

[nojnhuh]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This PR adds e2e tests that upgrade a CAPZ that's managing AKS workload clusters.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4361

Special notes for your reviewer:

I included some drive-by changes here in separate commits.

TODOs:

• squashed commits
• includes documentation
• adds unit tests
• cherry-pick candidate

Release note:

NONE

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 52.84%. Comparing base (73e4394) to head (0909161).
Report is 6 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5680   +/-   ##
=======================================
  Coverage   52.84%   52.84%           
=======================================
  Files         278      278           
  Lines       29610    29610           
=======================================
  Hits        15647    15647           
  Misses      13146    13146           
  Partials      817      817           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nojnhuh]

Hunting for flakes in the new tests 🔍

/test pull-cluster-api-provider-azure-apiversion-upgrade

[nojnhuh]

Hunting for flakes in the new tests 🔍

anotha one

/test pull-cluster-api-provider-azure-apiversion-upgrade

[nojnhuh]

More flake hunting now that #5560 is merged:

/test pull-cluster-api-provider-azure-apiversion-upgrade

[nojnhuh]

/test pull-cluster-api-provider-azure-apiversion-upgrade

[nojnhuh]

/test pull-cluster-api-provider-azure-apiversion-upgrade

#5690

[willie-yao]

/retest

[jackfrancis]

/lgtm
/approve

/hold in case prow doesn't wait for capi-e2e test

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: ecb25645f4cb24c14623528f27aeebdb3bc3680e

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@nojnhuh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-capi-e2e	0909161	link	false	/test pull-cluster-api-provider-azure-capi-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[nojnhuh]

These changes don't affect the tests in the capi-e2e job. One more run of the apiversion-upgrade job will help me sleep tonight.

/test pull-cluster-api-provider-azure-apiversion-upgrade

[nojnhuh]

No flakes in the new tests, and I'm reasonably confident the capi-e2e failures are unrelated.

/hold cancel