CAPZ fails to apply new API version of ASO resources

[RoRoMaan]

/kind bug

What steps did you take and what happened:
We are trying to set up a new AKS cluster by using ManagedCluster with the apiVersion containerservice.azure.com/v1api20240901 & ManagedClustersAgentPool with the apiVersion containerservice.azure.com/v1api20240901.
CAPZ however cannot create these resources giving us the following error messages in the logs:

manager E0327 09:51:15.320962       1 controller.go:316] "Reconciler error" err="failed to run mutator: no kind \"ManagedCluster\" is registered for version \"containerservice.azure.com/v1api20240901\" in scheme \"pkg/runtime/scheme.go:100\"" controller="azureasomanagedcontrolplane" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AzureASOManagedControlPlane" AzureASOManagedControlPlane="my-namespace/my-awesome-cluster-vtqpv" namespace="my-namespace" name="my-awesome-cluster-vtqpv" reconcileID="c07c8cde-4152-4b9b-a352-d356daa99792"

manager E0327 09:51:15.369700       1 controller.go:316] "Reconciler error" err="failed to run mutator: error creating new containerservice.azure.com/v1api20240901, Kind=ManagedClustersAgentPool: no kind \"ManagedClustersAgentPool\" is registered for version \"containerservice.azure.com/v1api20240901\" in scheme \"pkg/runtime/scheme.go:100\"" controller="azureasomanagedcontrolplane" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AzureASOManagedControlPlane" AzureASOManagedControlPlane="my-namespace/my-awesome-cluster-vtqpv" namespace="my-namespace" name="my-awesome-cluster-vtqpv" reconcileID="c48ee890-0097-4297-b562-98150978c3fa"

What did you expect to happen:
CAPZ creates the k8s resources ManagedCluster & ManagedClustersAgentPool with the new apiVersion so that ASO can start to work it´s magic.

Anything else you would like to add:
We suspect that there might be missing references in the code, e.g.

• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/main.go#L30
• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/pkg/mutators/azureasomanagedcontrolplane.go#L25

There are references for the previous api version (v1api20231001) or the preview versions but not the api version v1api20240901

Disclaimer: we aren´t developers but we hope that those findings can help somehow

Environment:

• cluster-api-provider-azure version: v1.19.1
• Kubernetes version: (use kubectl version): v1.31.5
• OS (e.g. from /etc/os-release): Ubuntu 22.04.5 LTS

[jackfrancis]

Hi @RoRoMaan, are you attempting to create a new AKS cluster and have access to the native ASO definitions? If so, here's a reference for doing so. The AzureASOMangedCluster resource will look like this:

• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/v1.19.0/templates/test/ci/cluster-template-prow-aks-aso.yaml#L49-L67

And the AzureASOManagedControlPlane will look like this:

• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/v1.19.0/templates/test/ci/cluster-template-prow-aks-aso.yaml#L16-L47

And here's a reference example of a AzureASOManagedMachinePool:

• 20240901

The full reference for the v1api20240901 APIs are here (the latest 1.19.1 release of CAPZ includes support for v1api20240901 ):

• https://azure.github.io/azure-service-operator/reference/containerservice/v1api20240901/#ManagedCluster
• https://azure.github.io/azure-service-operator/reference/containerservice/v1api20240901/#ManagedClustersAgentPool

Let me know if you have any more questions!

cc @nojnhuh @nawazkh

[nojnhuh]

This is the only change we need to enable the new AKS API version with the ASO API (AzureAKSManagedControlPlane and friends):

diff --git a/main.go b/main.go
index 58b97a40e..343c0e8b0 100644
--- a/main.go
+++ b/main.go
@@ -30,6 +30,7 @@ import (
 	asocontainerservicev1api20231001 "github.com/Azure/azure-service-operator/v2/api/containerservice/v1api20231001"
 	asocontainerservicev1api20231102preview "github.com/Azure/azure-service-operator/v2/api/containerservice/v1api20231102preview"
 	asocontainerservicev1api20240402preview "github.com/Azure/azure-service-operator/v2/api/containerservice/v1api20240402preview"
+	asocontainerservicev1api20240901 "github.com/Azure/azure-service-operator/v2/api/containerservice/v1api20240901"
 	asokubernetesconfigurationv1 "github.com/Azure/azure-service-operator/v2/api/kubernetesconfiguration/v1api20230501"
 	asonetworkv1api20201101 "github.com/Azure/azure-service-operator/v2/api/network/v1api20201101"
 	asonetworkv1api20220701 "github.com/Azure/azure-service-operator/v2/api/network/v1api20220701"
@@ -92,6 +93,7 @@ func init() {
 	_ = asocontainerservicev1api20230315preview.AddToScheme(scheme)
 	_ = asocontainerservicev1api20231102preview.AddToScheme(scheme)
 	_ = asocontainerservicev1api20240402preview.AddToScheme(scheme)
+	_ = asocontainerservicev1api20240901.AddToScheme(scheme)
 	_ = asokubernetesconfigurationv1.AddToScheme(scheme)
 	// +kubebuilder:scaffold:scheme
 }

Any other mention of a particular API version is just for CAPZ's internal use and it doesn't tend to rely on bleeding-edge AKS features.

Usually we make these changes when we bump the ASO version, e.g. #4973

Once this issue is resolved then we shouldn't need a complete accounting of the AKS API versions like this: Azure/azure-service-operator#2791

[blackbaddl13]

We built the fix @nojnhuh suggested ourselves, this removes the errors but we can not confirm the full functionality yet.

[RoRoMaan]

Hi @jackfrancis thank you for your reply. We are using ClusterClasses which are patching the ASO native resources into AzureASOManaged....Templates which then create the AzureASOManaged... resources you are refering to.

Thank you @nojnhuh for the clarification. It seems like adding those references in the main.go indeed fixes the issue.
CAPZ does not complain about missing a scheme anymore, however I noticed something else confusing.

Here is a snippet of how our rendered AzureASOManagedControlPlane looks like:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: AzureASOManagedControlPlane
metadata:
  name: my-awesome-cluster-hr87q
  namespace: my-namespace
spec:
  resources:
   - apiVersion: containerservice.azure.com/v1api20240901
      kind: ManagedCluster
      metadata:
        name: my-awesome-cluster
        namespace: my-namespace
      spec:
        apiServerAccessProfile:
          authorizedIPRanges:
          - 1.2.3.4/32
        autoUpgradeProfile:
          nodeOSUpgradeChannel: Unmanaged
          upgradeChannel: none
        ...

We are aiming to use the new CRD version v1api20240901 of the api group containerservice.azure.com - so far so good.

When I check the created ManagedCluster object I see the following:

apiVersion: containerservice.azure.com/v1api20210501
kind: ManagedCluster
metadata:
  name: my-awesome-cluster
  namespace: my-namespace
spec:
  apiServerAccessProfile:
    authorizedIPRanges:
      - 1.2.3.4/32
  autoUpgradeProfile:
    nodeOSUpgradeChannel: Unmanaged
    upgradeChannel: none
  ...

The version changed to v1api20210501. The same seems to happen to ManagedClustersAgentPool.
I´m sure it has nothing to do with the issue above but maybe someone has an idea of what is happening here?
I´m not sure wether it is CAPZ or ASO doing this.

[nojnhuh]

That's obviously not ideal, but it is expected: #5168 (comment)

[mboersma]

@blackbaddl13 @RoRoMaan does the new documentation clarify this behavior enough, or is there more we should do?

[nojnhuh]

The changes to main.go still need to be added for the new API version to work. That's up for grabs for whoever wants to open a PR.

[RoRoMaan]

@mboersma this information helped to understand the issue, thank you!