k8s community infra migration: CI tests must use workload identity

[jackfrancis]

As a requirement of the migration of Azure tests to the new Azure-sponsored community prow infra environment, we won't be able to leverage secrets (e.g., passwords) during CI runs.

azuredisk-csi-driver seems to use service principal + secrets for all of its E2E scenarios:

• azuredisk-csi-driver/test/utils/credentials/credentials.go

  Lines 123 to 132 in 71740c6

  	// If the tests are being run in Prow, credentials are not supplied through env vars. Instead, it is supplied
  	// through env var AZURE_CREDENTIALS. We need to convert it to AZURE_CREDENTIAL_FILE for sanity, integration and E2E tests
  	if testutil.IsRunningInProw() {
  	log.Println("Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE")
  	c, err := getCredentialsFromAzureCredentials(os.Getenv("AZURE_CREDENTIALS"))
  	if err != nil {
  	return nil, err
  	}
  	return parseAndExecuteTemplate(cloud, c.TenantID, c.SubscriptionID, c.ClientID, c.ClientSecret, resourceGroup, location, vmType)
  	}

Documentation suggests that the project has first class support for workload identity:

• https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/workload-identity.md

In order to continue running E2E tests after the community infra migration (deadline is 1 August) we'll need to update the test implementation to use workload identity instead of service principal secrets.

Here are the relevant large CAPZ PRs that did this work in CAPZ:

• CI: workload-identity native cluster-api-provider-azure#4765
• remove service principal references from CI cluster-api-provider-azure#4978

This is the new environment variable configuration that we expect to ship to prow jobs as part of the conversion:

• https://github.com/kubernetes/test-infra/blob/fdf57bdb742c8fc010412d83cf7d4967a370a4dd/config/jobs/kubernetes-sigs/cluster-api-provider-azure/cluster-api-provider-azure-presets.yaml#L17-L27

Note that the AZURE_CLIENT_ID reference above is the user-assigned ID.

[dtzar]

Related to #2410

[andyzhangx]

/assign @cvvz