fix(webhook): Update cluster webhook to validate failure domains #1226

thunderboltsid · 2025-07-22T13:33:06Z

Use XOR instead of OR for cluster and subnet presence check against failure domains on control plane and worker overrides.
Update CAPX version to latest beta to omit zero values for cluster and null values for subnet
Update patch unit tests as those are fragile (due to ordering) and there were changes in patches
Add kustomize patches for nutanix clusterclass template's NutanixMachineTemplate to strip out cluster and subnet
Improve verbose-mode logging for easier debugging in future

How has this been tested?

Run make dev.run-on-kind and see default NutanixMachineTemplates when management cluster is created and observe no placeholder cluster and subnet fields

- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    ...
    name: nutanix-quick-start-cp-nmt
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: ClusterClass
      name: nutanix-quick-start
      uid: 5b002494-0fab-41a8-a702-5cdf6f791b7c
  spec:
    template:
      spec:
        bootType: legacy
        image:
          name: ""
          type: name
        memorySize: 4Gi
        systemDiskSize: 40Gi
        vcpuSockets: 2
        vcpusPerSocket: 1
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    ...
    name: nutanix-quick-start-md-nmt
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: ClusterClass
      name: nutanix-quick-start
      uid: 5b002494-0fab-41a8-a702-5cdf6f791b7c
  spec:
    template:
      spec:
        bootType: legacy
        image:
          name: ""
          type: name
        memorySize: 4Gi
        systemDiskSize: 40Gi
        vcpuSockets: 2
        vcpusPerSocket: 1

Create a cluster with control plane and machine deployments with failure domains with cluster and subnet variables also set in controlPlane machineDetails and machineDeployment variables overrides.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: nkp-sid-caren
    cluster.x-k8s.io/provider: nutanix
  name: nkp-sid-caren
spec:
  ...
  topology:
    class: nutanix-quick-start
    controlPlane:
      metadata: {}
      replicas: 3
    variables:
    - name: clusterConfig
      value:
        ...
        controlPlane:
          nutanix:
            failureDomains:
            - fd-1
            - fd-2
            - fd-3
            machineDetails:
              bootType: uefi
              cluster:
                type: name
                name: ncn-dev-sandbox
              subnets:
              - type: name
                name: vlan173
              imageLookup:
                baseOS: rocky-9.6
                format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
              memorySize: 4Gi
              systemDiskSize: 40Gi
              vcpuSockets: 2
              vcpusPerSocket: 1
        ...
    - name: workerConfig
      value:
        nutanix:
          machineDetails:
            bootType: uefi
            cluster:
              name: ncn-dev-sandbox
              type: name
            imageLookup:
              baseOS: rocky-9.6
              format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
            memorySize: 4Gi
            subnets:
            - name: vlan173
              type: name
            systemDiskSize: 40Gi
            vcpuSockets: 2
            vcpusPerSocket: 1
    version: 1.33.1
    workers:
      machineDeployments:
      - class: default-worker
        failureDomain: fd-1
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: md-0
        variables:
          overrides:
          - name: workerConfig
            value:
              nutanix:
                machineDetails:
                  bootType: uefi
                  cluster:
                    type: name
                    name: ncn-dev-sandbox
                  subnets:
                  - type: name
                    name: vlan173
                  imageLookup:
                    baseOS: rocky-9.6
                    format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
                  memorySize: 8Gi
                  systemDiskSize: 80Gi
                  vcpuSockets: 8
                  vcpusPerSocket: 1

and expect failures from the webhook

$ k apply -f nkp-sid-caren-neg.yaml
...
Error from server (Forbidden): error when creating "nkp-sid-caren-neg.yaml": admission webhook "cluster-validator.caren.nutanix.com" denied the request: [spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.cluster: Forbidden: "cluster" must not be set when failureDomains are configured., spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.subnets: Forbidden: "subnets" must not be set when failureDomains are configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.cluster: Forbidden: "cluster" must not be set when failureDomain is configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.subnets: Forbidden: "subnets" must not be set when failureDomain is configured.]

Create a cluster without failure domains with a machine deployment with cluster and subnet not set

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: nkp-sid-caren-no-fd
    cluster.x-k8s.io/provider: nutanix
  name: nkp-sid-caren-no-fd
spec:
  ...
  topology:
    class: nutanix-quick-start
    controlPlane:
      metadata: {}
      replicas: 3
    variables:
    - name: clusterConfig
      value:
        ...
        controlPlane:
          nutanix:
            machineDetails:
              bootType: uefi
              imageLookup:
                baseOS: rocky-9.6
                format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
              memorySize: 4Gi
              systemDiskSize: 40Gi
              vcpuSockets: 2
              vcpusPerSocket: 1
        ...
    - name: workerConfig
      value:
        nutanix:
          machineDetails:
            bootType: uefi
            cluster:
              name: ncn-dev-sandbox
              type: name
            imageLookup:
              baseOS: rocky-9.6
              format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
            memorySize: 4Gi
            subnets:
            - name: vlan173
              type: name
            systemDiskSize: 40Gi
            vcpuSockets: 2
            vcpusPerSocket: 1
    version: 1.33.1
    workers:
      machineDeployments:
      - class: default-worker
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: md-0
        variables:
          overrides:
          - name: workerConfig
            value:
              nutanix:
                machineDetails:
                  bootType: uefi
                  imageLookup:
                    baseOS: rocky-9.6
                    format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
                  memorySize: 8Gi
                  cluster:
                    type: name
                    name: ncn-dev-sandbox
                  subnets:
                  - type: name
                    name: vlan173
                  systemDiskSize: 80Gi
                  vcpuSockets: 8
                  vcpusPerSocket: 1
      - class: default-worker
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: md-1
        variables:
          overrides:
          - name: workerConfig
            value:
              nutanix:
                machineDetails:
                  bootType: uefi
                  imageLookup:
                    baseOS: rocky-9.6
                    format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
                  memorySize: 8Gi
                  systemDiskSize: 80Gi
                  vcpuSockets: 8
                  vcpusPerSocket: 1

and expect failures from webhook

$ k apply -f no-fd-neg.yaml
...
Error from server (Forbidden): error when creating "no-fd-neg.yaml": admission webhook "cluster-validator.caren.nutanix.com" denied the request: [spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.cluster: Required value: "cluster" must be set when failureDomains are not configured., spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.subnets: Required value: "subnets" must be set when failureDomains are not configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.cluster: Required value: "cluster" must be set when failureDomain is not configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.subnets: Required value: "subnets" must be set when failureDomain is not configured.]

Create a cluster without failure domains with a machine deployment with cluster and subnet set

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  ...
  name: nkp-sid-caren-no-fd
spec:
  ...
  topology:
    class: nutanix-quick-start
    controlPlane:
      metadata: {}
      replicas: 3
    variables:
    - name: clusterConfig
      value:
        ...
        controlPlane:
          nutanix:
            machineDetails:
              bootType: uefi
              imageLookup:
                baseOS: rocky-9.6
                format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
              memorySize: 4Gi
              systemDiskSize: 40Gi
              vcpuSockets: 2
              vcpusPerSocket: 1
              cluster:
                type: name
                name: ncn-dev-sandbox
              subnets:
              - type: name
                name: vlan173
        ...
    - name: workerConfig
      value:
        nutanix:
          machineDetails:
            bootType: uefi
            cluster:
              name: ncn-dev-sandbox
              type: name
            imageLookup:
              baseOS: rocky-9.6
              format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
            memorySize: 4Gi
            subnets:
            - name: vlan173
              type: name
            systemDiskSize: 40Gi
            vcpuSockets: 2
            vcpusPerSocket: 1
    version: 1.33.1
    workers:
      machineDeployments:
      - class: default-worker
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: md-0
        variables:
          overrides:
          - name: workerConfig
            value:
              nutanix:
                machineDetails:
                  bootType: uefi
                  imageLookup:
                    baseOS: rocky-9.6
                    format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
                  memorySize: 8Gi
                  cluster:
                    type: name
                    name: ncn-dev-sandbox
                  subnets:
                  - type: name
                    name: vlan173
                  systemDiskSize: 80Gi
                  vcpuSockets: 8
                  vcpusPerSocket: 1
      - class: default-worker
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: md-1
        variables:
          overrides:
          - name: workerConfig
            value:
              nutanix:
                machineDetails:
                  bootType: uefi
                  imageLookup:
                    baseOS: rocky-9.6
                    format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
                  memorySize: 8Gi
                  cluster:
                    type: uuid
                    uuid: 00061f7f-44f7-19dc-3be1-7cc25586ee44
                  subnets:
                  - type: name
                    name: vlan173
                  systemDiskSize: 80Gi
                  vcpuSockets: 8
                  vcpusPerSocket: 1

Observe the generated NutanixMachineTemplate has cluster and subnets set correctly

apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    ...
    name: nkp-sid-caren-no-fd-7np7m
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: nkp-sid-caren-no-fd
  spec:
    template:
      spec:
        bootType: uefi
        cluster:
          name: ncn-dev-sandbox
          type: name
        imageLookup:
          baseOS: rocky-9.6
          format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
        memorySize: 4Gi
        subnet:
        - name: vlan173
          type: name
        systemDiskSize: 40Gi
        vcpuSockets: 2
        vcpusPerSocket: 1
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    ...
    name: nkp-sid-caren-no-fd-md-0-qnr58
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: nkp-sid-caren-no-fd
  spec:
    template:
      spec:
        bootType: uefi
        cluster:
          name: ncn-dev-sandbox
          type: name
        imageLookup:
          baseOS: rocky-9.6
          format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
        memorySize: 8Gi
        subnet:
        - name: vlan173
          type: name
        systemDiskSize: 80Gi
        vcpuSockets: 8
        vcpusPerSocket: 1
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    ...
    name: nkp-sid-caren-no-fd-md-1-59tlq
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: nkp-sid-caren-no-fd
  spec:
    template:
      spec:
        bootType: uefi
        cluster:
          type: uuid
          uuid: 00061f7f-44f7-19dc-3be1-7cc25586ee44
        imageLookup:
          baseOS: rocky-9.6
          format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
        memorySize: 8Gi
        subnet:
        - name: vlan173
          type: name
        systemDiskSize: 80Gi
        vcpuSockets: 8
        vcpusPerSocket: 1

Create a cluster with control plane and machine deployment on failure domains

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: nkp-sid-caren
    cluster.x-k8s.io/provider: nutanix
  name: nkp-sid-caren
spec:
  ...
  topology:
    class: nutanix-quick-start
    controlPlane:
      metadata: {}
      replicas: 3
    variables:
    - name: clusterConfig
      value:
        ...
        controlPlane:
          nutanix:
            failureDomains:
            - fd-1
            - fd-2
            - fd-3
            machineDetails:
              bootType: uefi
              imageLookup:
                baseOS: rocky-9.6
                format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
              memorySize: 4Gi
              systemDiskSize: 40Gi
              vcpuSockets: 2
              vcpusPerSocket: 1
        ...
    - name: workerConfig
      value:
        nutanix:
          machineDetails:
            bootType: uefi
            cluster:
              name: ncn-dev-sandbox
              type: name
            imageLookup:
              baseOS: rocky-9.6
              format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
            memorySize: 4Gi
            subnets:
            - name: vlan173
              type: name
            systemDiskSize: 40Gi
            vcpuSockets: 2
            vcpusPerSocket: 1
    version: 1.33.1
    workers:
      machineDeployments:
      - class: default-worker
        failureDomain: fd-1
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: md-0
        variables:
          overrides:
          - name: workerConfig
            value:
              nutanix:
                machineDetails:
                  bootType: uefi
                  imageLookup:
                    baseOS: rocky-9.6
                    format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
                  memorySize: 8Gi
                  systemDiskSize: 80Gi
                  vcpuSockets: 8
                  vcpusPerSocket: 1

Observe the generated NutanixMachineTemplate does not have cluster and subnet set

- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    annotations:
      cluster.x-k8s.io/cloned-from-groupkind: NutanixMachineTemplate.infrastructure.cluster.x-k8s.io
      cluster.x-k8s.io/cloned-from-name: nutanix-quick-start-cp-nmt
      meta.helm.sh/release-name: cluster-api-runtime-extensions-nutanix
      meta.helm.sh/release-namespace: default
    creationTimestamp: "2025-07-23T17:12:31Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
      cluster.x-k8s.io/cluster-name: nkp-sid-caren
      cluster.x-k8s.io/provider: nutanix
      topology.cluster.x-k8s.io/owned: ""
    name: nkp-sid-caren-kp4nc
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: nkp-sid-caren
      uid: edd422ad-d4d4-497a-9e6b-892f220252c6
    resourceVersion: "57785"
    uid: ff0a17f3-50a4-41bc-90b8-b1c0efc8311f
  spec:
    template:
      spec:
        bootType: uefi
        imageLookup:
          baseOS: rocky-9.6
          format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
        memorySize: 4Gi
        systemDiskSize: 40Gi
        vcpuSockets: 2
        vcpusPerSocket: 1
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: NutanixMachineTemplate
  metadata:
    annotations:
      cluster.x-k8s.io/cloned-from-groupkind: NutanixMachineTemplate.infrastructure.cluster.x-k8s.io
      cluster.x-k8s.io/cloned-from-name: nutanix-quick-start-md-nmt
      meta.helm.sh/release-name: cluster-api-runtime-extensions-nutanix
      meta.helm.sh/release-namespace: default
    creationTimestamp: "2025-07-23T17:12:31Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
      cluster.x-k8s.io/cluster-name: nkp-sid-caren
      cluster.x-k8s.io/provider: nutanix
      topology.cluster.x-k8s.io/deployment-name: md-0
      topology.cluster.x-k8s.io/owned: ""
    name: nkp-sid-caren-md-0-bfm6v
    namespace: default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: nkp-sid-caren
      uid: edd422ad-d4d4-497a-9e6b-892f220252c6
    resourceVersion: "57795"
    uid: a9ea9499-ad82-4082-b9cb-4c920940e06d
  spec:
    template:
      spec:
        bootType: uefi
        imageLookup:
          baseOS: rocky-9.6
          format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-*
        memorySize: 8Gi
        systemDiskSize: 80Gi
        vcpuSockets: 8
        vcpusPerSocket: 1

pkg/webhook/cluster/nutanix_validator.go

Use XOR instead of OR for cluster and subnet presence check against failure domains on control plane and worker overrides.

we shouldn't be injecting extraneous values: this is confusing for debugging behaviour.

this isn't an error e.g. clusters that don't have failure domain. remove the "error" wording from log as it's confusing when debugging.

remove subnet and cluster

dkoshkin

Thanks!

pkg/webhook/cluster/nutanix_validator_test.go

hack/third-party/capx/go.mod

…an (#1236) These changes ensure the resulting NutanixMachineTemplate patches also reflect the XOR between failureDomains and cluster/subnets. **How has this been tested?** Tested in conjunction with #1233 in #1226 * Run `make dev.run-on-kind` and see default NutanixMachineTemplates when management cluster is created and observe no placeholder cluster and subnet fields ```yaml - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nutanix-quick-start-cp-nmt namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: ClusterClass name: nutanix-quick-start spec: template: spec: bootType: legacy image: name: "" type: name memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nutanix-quick-start-md-nmt namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: ClusterClass name: nutanix-quick-start spec: template: spec: bootType: legacy image: name: "" type: name memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 ``` * Create a cluster without failure domains with a machine deployment with cluster and subnet set ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: ... name: nkp-sid-caren-no-fd spec: ... topology: class: nutanix-quick-start controlPlane: metadata: {} replicas: 3 variables: - name: clusterConfig value: ... controlPlane: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 ... - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnets: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 version: 1.33.1 workers: machineDeployments: - class: default-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-0 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 - class: default-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-1 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi cluster: type: uuid uuid: 00061f7f-44f7-19dc-3be1-7cc25586ee44 subnets: - type: name name: vlan173 systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` Observe the generated NutanixMachineTemplate has cluster and subnets set correctly ```yaml apiVersion: v1 items: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-no-fd-7np7m namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren-no-fd spec: template: spec: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnet: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-no-fd-md-0-qnr58 namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren-no-fd spec: template: spec: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi subnet: - name: vlan173 type: name systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-no-fd-md-1-59tlq namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren-no-fd spec: template: spec: bootType: uefi cluster: type: uuid uuid: 00061f7f-44f7-19dc-3be1-7cc25586ee44 imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi subnet: - name: vlan173 type: name systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` * Create a cluster with control plane and machine deployment on failure domains ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: labels: cluster.x-k8s.io/cluster-name: nkp-sid-caren cluster.x-k8s.io/provider: nutanix name: nkp-sid-caren spec: ... topology: class: nutanix-quick-start controlPlane: metadata: {} replicas: 3 variables: - name: clusterConfig value: ... controlPlane: nutanix: failureDomains: - fd-1 - fd-2 - fd-3 machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 ... - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnets: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 version: 1.33.1 workers: machineDeployments: - class: default-worker failureDomain: fd-1 metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-0 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` Observe the generated NutanixMachineTemplate does not have cluster and subnet set ```yaml - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-kp4nc namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren spec: template: spec: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-md-0-bfm6v namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren spec: template: spec: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ```

These changes ensure the resulting NutanixMachineTemplate patches also reflect the XOR between failureDomains and cluster/subnets. **How has this been tested?** Tested in conjunction with #1233 in #1226 * Run `make dev.run-on-kind` and see default NutanixMachineTemplates when management cluster is created and observe no placeholder cluster and subnet fields ```yaml - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nutanix-quick-start-cp-nmt namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: ClusterClass name: nutanix-quick-start spec: template: spec: bootType: legacy image: name: "" type: name memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nutanix-quick-start-md-nmt namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: ClusterClass name: nutanix-quick-start spec: template: spec: bootType: legacy image: name: "" type: name memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 ``` * Create a cluster without failure domains with a machine deployment with cluster and subnet set ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: ... name: nkp-sid-caren-no-fd spec: ... topology: class: nutanix-quick-start controlPlane: metadata: {} replicas: 3 variables: - name: clusterConfig value: ... controlPlane: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 ... - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnets: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 version: 1.33.1 workers: machineDeployments: - class: default-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-0 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 - class: default-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-1 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi cluster: type: uuid uuid: 00061f7f-44f7-19dc-3be1-7cc25586ee44 subnets: - type: name name: vlan173 systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` Observe the generated NutanixMachineTemplate has cluster and subnets set correctly ```yaml apiVersion: v1 items: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-no-fd-7np7m namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren-no-fd spec: template: spec: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnet: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-no-fd-md-0-qnr58 namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren-no-fd spec: template: spec: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi subnet: - name: vlan173 type: name systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-no-fd-md-1-59tlq namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren-no-fd spec: template: spec: bootType: uefi cluster: type: uuid uuid: 00061f7f-44f7-19dc-3be1-7cc25586ee44 imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi subnet: - name: vlan173 type: name systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` * Create a cluster with control plane and machine deployment on failure domains ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: labels: cluster.x-k8s.io/cluster-name: nkp-sid-caren cluster.x-k8s.io/provider: nutanix name: nkp-sid-caren spec: ... topology: class: nutanix-quick-start controlPlane: metadata: {} replicas: 3 variables: - name: clusterConfig value: ... controlPlane: nutanix: failureDomains: - fd-1 - fd-2 - fd-3 machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 ... - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnets: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 version: 1.33.1 workers: machineDeployments: - class: default-worker failureDomain: fd-1 metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-0 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` Observe the generated NutanixMachineTemplate does not have cluster and subnet set ```yaml - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-kp4nc namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren spec: template: spec: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: NutanixMachineTemplate metadata: ... name: nkp-sid-caren-md-0-bfm6v namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: nkp-sid-caren spec: template: spec: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ```

Use XOR instead of OR for cluster and subnet presence check against failure domains on control plane and worker overrides. **How has this been tested?** Tested in conjunction with #1236 in #1226 * Create a cluster with control plane and machine deployments with failure domains with cluster and subnet variables also set in controlPlane machineDetails and machineDeployment variables overrides. ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: labels: cluster.x-k8s.io/cluster-name: nkp-sid-caren cluster.x-k8s.io/provider: nutanix name: nkp-sid-caren spec: ... topology: class: nutanix-quick-start controlPlane: metadata: {} replicas: 3 variables: - name: clusterConfig value: ... controlPlane: nutanix: failureDomains: - fd-1 - fd-2 - fd-3 machineDetails: bootType: uefi cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 ... - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnets: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 version: 1.33.1 workers: machineDeployments: - class: default-worker failureDomain: fd-1 metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-0 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` and expect failures from the webhook ```sh $ k apply -f nkp-sid-caren-neg.yaml ... Error from server (Forbidden): error when creating "nkp-sid-caren-neg.yaml": admission webhook "cluster-validator.caren.nutanix.com" denied the request: [spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.cluster: Forbidden: "cluster" must not be set when failureDomains are configured., spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.subnets: Forbidden: "subnets" must not be set when failureDomains are configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.cluster: Forbidden: "cluster" must not be set when failureDomain is configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.subnets: Forbidden: "subnets" must not be set when failureDomain is configured.] ``` * Create a cluster without failure domains with a machine deployment with cluster and subnet not set ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: labels: cluster.x-k8s.io/cluster-name: nkp-sid-caren-no-fd cluster.x-k8s.io/provider: nutanix name: nkp-sid-caren-no-fd spec: ... topology: class: nutanix-quick-start controlPlane: metadata: {} replicas: 3 variables: - name: clusterConfig value: ... controlPlane: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 ... - name: workerConfig value: nutanix: machineDetails: bootType: uefi cluster: name: ncn-dev-sandbox type: name imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 4Gi subnets: - name: vlan173 type: name systemDiskSize: 40Gi vcpuSockets: 2 vcpusPerSocket: 1 version: 1.33.1 workers: machineDeployments: - class: default-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-0 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi cluster: type: name name: ncn-dev-sandbox subnets: - type: name name: vlan173 systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 - class: default-worker metadata: annotations: cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" name: md-1 variables: overrides: - name: workerConfig value: nutanix: machineDetails: bootType: uefi imageLookup: baseOS: rocky-9.6 format: nkp-{{.BaseOS}}-release-{{.K8sVersion}}-* memorySize: 8Gi systemDiskSize: 80Gi vcpuSockets: 8 vcpusPerSocket: 1 ``` and expect failures from webhook ```sh $ k apply -f no-fd-neg.yaml ... Error from server (Forbidden): error when creating "no-fd-neg.yaml": admission webhook "cluster-validator.caren.nutanix.com" denied the request: [spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.cluster: Required value: "cluster" must be set when failureDomains are not configured., spec.topology.variables.clusterConfig.value.controlPlane.nutanix.machineDetails.subnets: Required value: "subnets" must be set when failureDomains are not configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.cluster: Required value: "cluster" must be set when failureDomain is not configured., spec.topology.workers.machineDeployments.variables.overrides.workerConfig.value.nutanix.machineDetails.subnets: Required value: "subnets" must be set when failureDomain is not configured.] ```

github-actions bot added the fix label Jul 22, 2025

thunderboltsid requested review from dkoshkin, dlipovetsky, jimmidyson and yanhua121 July 22, 2025 13:36

github-actions bot added fix and removed fix labels Jul 22, 2025

yanhua121 reviewed Jul 22, 2025

View reviewed changes

pkg/webhook/cluster/nutanix_validator.go Outdated Show resolved Hide resolved

thunderboltsid requested a review from yanhua121 July 22, 2025 14:30

dkoshkin reviewed Jul 22, 2025

View reviewed changes

pkg/webhook/cluster/nutanix_validator.go Show resolved Hide resolved

pkg/webhook/cluster/nutanix_validator.go Outdated Show resolved Hide resolved

thunderboltsid added the do not merge label Jul 22, 2025

thunderboltsid requested a review from dkoshkin July 23, 2025 10:20

thunderboltsid force-pushed the issue/cluster-fd-xor-low-churn branch 2 times, most recently from 049d109 to 165c195 Compare July 23, 2025 13:01

thunderboltsid added 2 commits July 23, 2025 16:25

fix(webhook): Update cluster webhook to validate failure domains

ccf868d

Use XOR instead of OR for cluster and subnet presence check against failure domains on control plane and worker overrides.

fix(handler): remove extraneous injection for nutanix

8782a87

we shouldn't be injecting extraneous values: this is confusing for debugging behaviour.

thunderboltsid force-pushed the issue/cluster-fd-xor-low-churn branch from 165c195 to a57543b Compare July 23, 2025 14:26

thunderboltsid added 5 commits July 23, 2025 18:31

fix(deps): Update CAPX to v1.7.0-beta.4

1401f1f

fix(log): instrument logging for mutation/meta.go

a8f39d8

fix(log): update failure domain config not found log

65d5a41

this isn't an error e.g. clusters that don't have failure domain. remove the "error" wording from log as it's confusing when debugging.

fix(test): update the patches for machinedetails test

62c7c37

fix(clusterclass): update NutanixMachineTemplate in clusterclass

f4bf850

remove subnet and cluster

thunderboltsid force-pushed the issue/cluster-fd-xor-low-churn branch from f369ad5 to f4bf850 Compare July 23, 2025 16:32

github-actions bot added fix and removed fix labels Jul 23, 2025

thunderboltsid removed the do not merge label Jul 23, 2025

github-actions bot added fix and removed fix labels Jul 23, 2025

dkoshkin approved these changes Jul 23, 2025

View reviewed changes

pkg/webhook/cluster/nutanix_validator_test.go Show resolved Hide resolved

pkg/webhook/cluster/nutanix_validator_test.go Show resolved Hide resolved

github-actions bot added fix and removed fix labels Jul 23, 2025

dkoshkin reviewed Jul 23, 2025

View reviewed changes

hack/third-party/capx/go.mod Show resolved Hide resolved

thunderboltsid added the do not merge label Jul 23, 2025

thunderboltsid closed this Jul 23, 2025

This was referenced Jul 23, 2025

fix(patches): Ensure resulting NutanixMachineTemplate patches are clean #1236

Merged

fix(webhook): Update cluster webhook to validate failure domains #1233

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(webhook): Update cluster webhook to validate failure domains #1226

fix(webhook): Update cluster webhook to validate failure domains #1226

Uh oh!

thunderboltsid commented Jul 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkoshkin left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix(webhook): Update cluster webhook to validate failure domains #1226

fix(webhook): Update cluster webhook to validate failure domains #1226

Uh oh!

Conversation

thunderboltsid commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkoshkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thunderboltsid commented Jul 22, 2025 •

edited

Loading