diff --git a/docs/content/customization/aws/placement-group-nfd.md b/docs/content/customization/aws/placement-group-nfd.md new file mode 100644 index 000000000..9d8514701 --- /dev/null +++ b/docs/content/customization/aws/placement-group-nfd.md @@ -0,0 +1,201 @@ ++++ +title = "AWS Placement Group Node Feature Discovery" ++++ + +The AWS placement group NFD (Node Feature Discovery) customization automatically discovers and labels nodes with their placement group information, enabling workload scheduling based on placement group characteristics. + +This customization will be available when the +[provider-specific cluster configuration patch]({{< ref "..">}}) is included in the `ClusterClass`. + +## What is Placement Group NFD? + +Placement Group NFD automatically discovers the placement group information for each node and creates node labels that can be used for workload scheduling. This enables: + +- **Workload Affinity**: Schedule pods on nodes within the same placement group for low latency +- **Fault Isolation**: Schedule critical workloads on nodes in different placement groups +- **Resource Optimization**: Use placement group labels for advanced scheduling strategies + +## How it Works + +The NFD customization: + +1. **Deploys a Discovery Script**: Automatically installs a script on each node that queries AWS metadata +2. **Queries AWS Metadata**: Uses EC2 instance metadata to discover placement group information +3. **Creates Node Labels**: Generates Kubernetes node labels with placement group details +4. **Updates Continuously**: Refreshes labels as nodes are added or moved + +## Generated Node Labels + +The NFD customization creates the following node labels: + +| Label | Description | Example | +|-------|-------------|---------| +| `feature.node.kubernetes.io/aws-placement-group` | The name of the placement group | `my-cluster-pg` | +| `feature.node.kubernetes.io/partition` | The partition number (for partition placement groups) | `0`, `1`, `2` | + +## Configuration + +The placement group NFD customization is automatically enabled when a placement group is configured. No additional configuration is required. + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: +spec: + topology: + variables: + - name: clusterConfig + value: + controlPlane: + aws: + placementGroup: + name: "control-plane-pg" + - name: workerConfig + value: + aws: + placementGroup: + name: "worker-pg" +``` + +## Usage Examples + +### Workload Affinity + +Schedule pods on nodes within the same placement group for low latency: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: high-performance-app +spec: + replicas: 3 + selector: + matchLabels: + app: high-performance-app + template: + metadata: + labels: + app: high-performance-app + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: feature.node.kubernetes.io/aws-placement-group + operator: In + values: ["worker-pg"] + containers: + - name: app + image: my-app:latest +``` + +### Fault Isolation + +Distribute critical workloads across different placement groups: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: critical-app +spec: + replicas: 6 + selector: + matchLabels: + app: critical-app + template: + metadata: + labels: + app: critical-app + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: ["critical-app"] + topologyKey: feature.node.kubernetes.io/aws-placement-group + containers: + - name: app + image: critical-app:latest +``` + +### Partition-Aware Scheduling + +For partition placement groups, schedule workloads on specific partitions: + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: distributed-database +spec: + replicas: 3 + selector: + matchLabels: + app: distributed-database + template: + metadata: + labels: + app: distributed-database + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: feature.node.kubernetes.io/partition + operator: In + values: ["0", "1", "2"] + containers: + - name: database + image: my-database:latest +``` + +## Verification + +You can verify that the NFD labels are working by checking the node labels: + +```bash +# Check all nodes and their placement group labels +kubectl get nodes --show-labels | grep placement-group + +# Check specific node labels +kubectl describe node | grep placement-group + +# Check partition labels +kubectl get nodes --show-labels | grep partition +``` + +## Troubleshooting + +### Check NFD Script Status + +Verify that the discovery script is running: + +```bash +# Check if the script exists on nodes +kubectl debug node/ -it --image=busybox -- chroot /host ls -la /etc/kubernetes/node-feature-discovery/source.d/ + +# Check script execution +kubectl debug node/ -it --image=busybox -- chroot /host cat /etc/kubernetes/node-feature-discovery/features.d/placementgroup +``` + +## Integration with Other Features + +Placement Group NFD works seamlessly with: + +- **Pod Affinity/Anti-Affinity**: Use placement group labels for advanced scheduling +- **Topology Spread Constraints**: Distribute workloads across placement groups + +## Security Considerations + +- The discovery script queries AWS instance metadata (IMDSv2) +- No additional IAM permissions are required beyond standard node permissions +- Labels are automatically managed and do not require manual intervention +- The script runs with appropriate permissions and security context diff --git a/docs/content/customization/aws/placement-group.md b/docs/content/customization/aws/placement-group.md new file mode 100644 index 000000000..4e4cb16fc --- /dev/null +++ b/docs/content/customization/aws/placement-group.md @@ -0,0 +1,138 @@ ++++ +title = "AWS Placement Group" ++++ + +The AWS placement group customization allows the user to specify placement groups for control-plane +and worker machines to control their placement strategy within AWS. + +This customization will be available when the +[provider-specific cluster configuration patch]({{< ref "..">}}) is included in the `ClusterClass`. + +## What are Placement Groups? + +AWS placement groups are logical groupings of instances within a single Availability Zone that influence how instances are placed on underlying hardware. They are useful for: + +- **Cluster Placement Groups**: For applications that benefit from low network latency, high network throughput, or both +- **Partition Placement Groups**: For large distributed and replicated workloads, such as HDFS, HBase, and Cassandra +- **Spread Placement Groups**: For applications that have a small number of critical instances that should be kept separate + +## Configuration + +The placement group configuration supports the following field: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | The name of the placement group (1-255 characters) | + +## Examples + +### Control Plane and Worker Placement Groups + +To specify placement groups for both control plane and worker machines: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: +spec: + topology: + variables: + - name: clusterConfig + value: + controlPlane: + aws: + placementGroup: + name: "control-plane-pg" + - name: workerConfig + value: + aws: + placementGroup: + name: "worker-pg" +``` + +### Control Plane Only + +To specify placement group only for control plane machines: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: +spec: + topology: + variables: + - name: clusterConfig + value: + controlPlane: + aws: + placementGroup: + name: "control-plane-pg" +``` + +### MachineDeployment Overrides + +You can customize individual MachineDeployments by using the overrides field: + +```yaml +spec: + topology: + # ... + workers: + machineDeployments: + - class: default-worker + name: md-0 + variables: + overrides: + - name: workerConfig + value: + aws: + placementGroup: + name: "special-worker-pg" +``` + +## Resulting CAPA Configuration + +Applying the placement group configuration will result in the following value being set: + +- control-plane `AWSMachineTemplate`: + + - ```yaml + spec: + template: + spec: + placementGroupName: control-plane-pg + ``` + +- worker `AWSMachineTemplate`: + + - ```yaml + spec: + template: + spec: + placementGroupName: worker-pg + ``` + +## Best Practices + +1. **Placement Group Types**: Choose the appropriate placement group type based on your workload: + - **Cluster**: For applications requiring low latency and high throughput + - **Partition**: For large distributed workloads that need fault isolation + - **Spread**: For critical instances that need maximum availability + +2. **Naming Convention**: Use descriptive names that indicate the purpose and type of the placement group + +3. **Availability Zone**: Placement groups are constrained to a single Availability Zone, so plan your cluster topology accordingly + +4. **Instance Types**: Some instance types have restrictions on placement groups (e.g., some bare metal instances) + +5. **Capacity Planning**: Consider the placement group capacity limits when designing your cluster + +## Important Notes + +- Placement groups must be created in AWS before they can be referenced +- Placement groups are constrained to a single Availability Zone +- You cannot move an existing instance into a placement group +- Some instance types cannot be launched in placement groups +- Placement groups have capacity limits that vary by type and instance family diff --git a/docs/content/customization/eks/placement-group-nfd.md b/docs/content/customization/eks/placement-group-nfd.md new file mode 100644 index 000000000..fcb390070 --- /dev/null +++ b/docs/content/customization/eks/placement-group-nfd.md @@ -0,0 +1,195 @@ ++++ +title = "EKS Placement Group Node Feature Discovery" ++++ + +The EKS placement group NFD (Node Feature Discovery) customization automatically discovers and labels EKS worker nodes with their placement group information, enabling workload scheduling based on placement group characteristics. + +This customization will be available when the +[provider-specific cluster configuration patch]({{< ref "..">}}) is included in the `ClusterClass`. + +## What is Placement Group NFD? + +Placement Group NFD automatically discovers the placement group information for each EKS worker node and creates node labels that can be used for workload scheduling. This enables: + +- **Workload Affinity**: Schedule pods on nodes within the same placement group for low latency +- **Fault Isolation**: Schedule critical workloads on nodes in different placement groups +- **Resource Optimization**: Use placement group labels for advanced scheduling strategies + +## How it Works + +The NFD customization: + +1. **Deploys a Discovery Script**: Automatically installs a script on each EKS worker node that queries AWS metadata +2. **Queries AWS Metadata**: Uses EC2 instance metadata to discover placement group information +3. **Creates Node Labels**: Generates Kubernetes node labels with placement group details +4. **Updates Continuously**: Refreshes labels as nodes are added or moved + +## Generated Node Labels + +The NFD customization creates the following node labels: + +| Label | Description | Example | +|-------|-------------|---------| +| `feature.node.kubernetes.io/aws-placement-group` | The name of the placement group | `my-eks-worker-pg` | +| `feature.node.kubernetes.io/partition` | The partition number (for partition placement groups) | `0`, `1`, `2` | + +## Configuration + +The placement group NFD customization is automatically enabled when a placement group is configured for EKS workers. No additional configuration is required. + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: +spec: + topology: + variables: + - name: workerConfig + value: + eks: + placementGroup: + name: "eks-worker-pg" +``` + +## Usage Examples + +### Workload Affinity + +Schedule pods on nodes within the same placement group for low latency: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: high-performance-app +spec: + replicas: 3 + selector: + matchLabels: + app: high-performance-app + template: + metadata: + labels: + app: high-performance-app + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: feature.node.kubernetes.io/aws-placement-group + operator: In + values: ["eks-worker-pg"] + containers: + - name: app + image: my-app:latest +``` + +### Fault Isolation + +Distribute critical workloads across different placement groups: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: critical-app +spec: + replicas: 6 + selector: + matchLabels: + app: critical-app + template: + metadata: + labels: + app: critical-app + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: ["critical-app"] + topologyKey: feature.node.kubernetes.io/aws-placement-group + containers: + - name: app + image: critical-app:latest +``` + +### Partition-Aware Scheduling + +For partition placement groups, schedule workloads on specific partitions: + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: distributed-database +spec: + replicas: 3 + selector: + matchLabels: + app: distributed-database + template: + metadata: + labels: + app: distributed-database + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: feature.node.kubernetes.io/partition + operator: In + values: ["0", "1", "2"] + containers: + - name: database + image: my-database:latest +``` + +## Verification + +You can verify that the NFD labels are working by checking the node labels: + +```bash +# Check all nodes and their placement group labels +kubectl get nodes --show-labels | grep placement-group + +# Check specific node labels +kubectl describe node | grep placement-group + +# Check partition labels +kubectl get nodes --show-labels | grep partition +``` + +## Troubleshooting + +### Check NFD Script Status + +Verify that the discovery script is running: + +```bash +# Check if the script exists on nodes +kubectl debug node/ -it --image=busybox -- chroot /host ls -la /etc/kubernetes/node-feature-discovery/source.d/ + +# Check script execution +kubectl debug node/ -it --image=busybox -- chroot /host cat /etc/kubernetes/node-feature-discovery/features.d/placementgroup +``` + +## Integration with Other Features + +Placement Group NFD works seamlessly with: + +- **Pod Affinity/Anti-Affinity**: Use placement group labels for advanced scheduling +- **Topology Spread Constraints**: Distribute workloads across placement groups + +## Security Considerations + +- The discovery script queries AWS instance metadata (IMDSv2) +- No additional IAM permissions are required beyond standard EKS node permissions +- Labels are automatically managed and do not require manual intervention +- The script runs with appropriate permissions and security context diff --git a/docs/content/customization/eks/placement-group.md b/docs/content/customization/eks/placement-group.md new file mode 100644 index 000000000..4f2da859a --- /dev/null +++ b/docs/content/customization/eks/placement-group.md @@ -0,0 +1,123 @@ ++++ +title = "EKS Placement Group" ++++ + +The EKS placement group customization allows the user to specify placement groups for EKS worker nodes to control their placement strategy within AWS. + +This customization will be available when the +[provider-specific cluster configuration patch]({{< ref "..">}}) is included in the `ClusterClass`. + +## What are Placement Groups? + +AWS placement groups are logical groupings of instances within a single Availability Zone that influence how instances are placed on underlying hardware. They are useful for: + +- **Cluster Placement Groups**: For applications that benefit from low network latency, high network throughput, or both +- **Partition Placement Groups**: For large distributed and replicated workloads, such as HDFS, HBase, and Cassandra +- **Spread Placement Groups**: For applications that have a small number of critical instances that should be kept separate + +## Configuration + +The placement group configuration supports the following field: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | The name of the placement group (1-255 characters) | + +## Examples + +### EKS Worker Placement Groups + +To specify placement groups for EKS worker nodes: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: +spec: + topology: + variables: + - name: workerConfig + value: + eks: + placementGroup: + name: "eks-worker-pg" +``` + +### Multiple Node Groups with Different Placement Groups + +You can configure different placement groups for different node groups: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: +spec: + topology: + variables: + - name: workerConfig + value: + eks: + placementGroup: + name: "general-worker-pg" + workers: + machineDeployments: + - class: high-performance-worker + name: md-0 + variables: + overrides: + - name: workerConfig + value: + eks: + placementGroup: + name: "high-performance-pg" + - class: general-worker + name: md-1 + variables: + overrides: + - name: workerConfig + value: + eks: + placementGroup: + name: "general-worker-pg" +``` + +## Resulting EKS Configuration + +Applying the placement group configuration will result in the following value being set in the EKS worker node configuration: + +- worker `AWSMachineTemplate`: + + - ```yaml + spec: + template: + spec: + placementGroupName: worker-pg + ``` + +## Best Practices + +1. **Placement Group Types**: Choose the appropriate placement group type based on your workload: + - **Cluster**: For applications requiring low latency and high throughput + - **Partition**: For large distributed workloads that need fault isolation + - **Spread**: For critical instances that need maximum availability + +2. **Naming Convention**: Use descriptive names that indicate the purpose and type of the placement group + +3. **Availability Zone**: Placement groups are constrained to a single Availability Zone, so plan your cluster topology accordingly + +4. **Instance Types**: Some instance types have restrictions on placement groups (e.g., some bare metal instances) + +5. **Capacity Planning**: Consider the placement group capacity limits when designing your cluster + +6. **EKS Node Groups**: Consider using different placement groups for different node groups based on workload requirements + +## Important Notes + +- Placement groups must be created in AWS before they can be referenced +- Placement groups are constrained to a single Availability Zone +- You cannot move an existing instance into a placement group +- Some instance types cannot be launched in placement groups +- Placement groups have capacity limits that vary by type and instance family +- EKS managed node groups support placement groups for enhanced networking performance diff --git a/pkg/handlers/aws/mutation/metapatch_handler.go b/pkg/handlers/aws/mutation/metapatch_handler.go index 6c3145ed0..c3868f448 100644 --- a/pkg/handlers/aws/mutation/metapatch_handler.go +++ b/pkg/handlers/aws/mutation/metapatch_handler.go @@ -16,6 +16,7 @@ import ( "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/instancetype" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/network" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/placementgroup" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/placementgroupnfd" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/region" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/securitygroups" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/tags" @@ -39,6 +40,7 @@ func MetaPatchHandler(mgr manager.Manager) handlers.Named { securitygroups.NewControlPlanePatch(), volumes.NewControlPlanePatch(), placementgroup.NewControlPlanePatch(), + placementgroupnfd.NewControlPlanePatch(), } patchHandlers = append(patchHandlers, genericmutation.MetaMutators(mgr)...) patchHandlers = append(patchHandlers, genericmutation.ControlPlaneMetaMutators()...) @@ -60,6 +62,7 @@ func MetaWorkerPatchHandler(mgr manager.Manager) handlers.Named { securitygroups.NewWorkerPatch(), volumes.NewWorkerPatch(), placementgroup.NewWorkerPatch(), + placementgroupnfd.NewWorkerPatch(), } patchHandlers = append(patchHandlers, genericmutation.WorkerMetaMutators()...) diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/embedded/placementgroup_discovery.sh b/pkg/handlers/aws/mutation/placementgroupnfd/embedded/placementgroup_discovery.sh new file mode 100755 index 000000000..e2a3d705c --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/embedded/placementgroup_discovery.sh @@ -0,0 +1,37 @@ +#!/bin/bash +set -euo pipefail + +# This script is used to discover the placement group for the node. +# The script will be placed in /etc/kubernetes/node-feature-discovery/source.d/ +# The script will be executed by the node feature discovery's local feature hook. +PLACEMENT_GROUP_FEATURE_DIR="/etc/kubernetes/node-feature-discovery/features.d" +PLACEMENT_GROUP_FEATURE_FILE="${PLACEMENT_GROUP_FEATURE_DIR}/placementgroup" +# Fetch IMDSv2 token +TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \ + -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") + +# Get placement info with HTTP status check +PARTITION_RESPONSE=$(curl -s -w "%{http_code}" -H "X-aws-ec2-metadata-token: $TOKEN" \ + http://169.254.169.254/latest/meta-data/placement/partition-number) + +PG_RESPONSE=$(curl -s -w "%{http_code}" -H "X-aws-ec2-metadata-token: $TOKEN" \ + http://169.254.169.254/latest/meta-data/placement/group-name) + +# Extract HTTP status codes and content +PARTITION_HTTP_CODE="${PARTITION_RESPONSE: -3}" # last 3 characters are the HTTP status code +PARTITION_CONTENT="${PARTITION_RESPONSE%???}" # remove the last 3 characters to get the content + +mkdir -p "${PLACEMENT_GROUP_FEATURE_DIR}" +touch "${PLACEMENT_GROUP_FEATURE_FILE}" + +# Only print features if HTTP 200 response +if [ "$PARTITION_HTTP_CODE" = "200" ] && [ -n "$PARTITION_CONTENT" ]; then + echo "feature.node.kubernetes.io/partition=${PARTITION_CONTENT}" >>"${PLACEMENT_GROUP_FEATURE_FILE}" +fi + +PG_HTTP_CODE="${PG_RESPONSE: -3}" # last 3 characters are the HTTP status code +PG_CONTENT="${PG_RESPONSE%???}" # remove the last 3 characters to get the content + +if [ "$PG_HTTP_CODE" = "200" ] && [ -n "$PG_CONTENT" ]; then + echo "feature.node.kubernetes.io/aws-placement-group=${PG_CONTENT}" >>"${PLACEMENT_GROUP_FEATURE_FILE}" +fi diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/inject_controlplane.go b/pkg/handlers/aws/mutation/placementgroupnfd/inject_controlplane.go new file mode 100644 index 000000000..3edae2a9b --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/inject_controlplane.go @@ -0,0 +1,100 @@ +// Copyright 2024 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + "context" + + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + cabpkv1 "sigs.k8s.io/cluster-api/bootstrap/kubeadm/api/v1beta1" + controlplanev1 "sigs.k8s.io/cluster-api/controlplane/kubeadm/api/v1beta1" + runtimehooksv1 "sigs.k8s.io/cluster-api/exp/runtime/hooks/api/v1alpha1" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/handlers/mutation" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/patches" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/patches/selectors" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/variables" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/placementgroup" +) + +type controlPlanePatchHandler struct { + variableName string + variableFieldPath []string +} + +func NewControlPlanePatch() *controlPlanePatchHandler { + return &controlPlanePatchHandler{ + variableName: v1alpha1.ClusterConfigVariableName, + variableFieldPath: []string{ + v1alpha1.ControlPlaneConfigVariableName, + v1alpha1.AWSVariableName, + placementgroup.VariableName, + }, + } +} + +func (h *controlPlanePatchHandler) Mutate( + ctx context.Context, + obj *unstructured.Unstructured, + vars map[string]apiextensionsv1.JSON, + holderRef runtimehooksv1.HolderReference, + _ client.ObjectKey, + _ mutation.ClusterGetter, +) error { + log := ctrl.LoggerFrom(ctx).WithValues( + "holderRef", holderRef, + ) + + placementGroupVar, err := variables.Get[v1alpha1.PlacementGroup]( + vars, + h.variableName, + h.variableFieldPath..., + ) + if err != nil { + if variables.IsNotFoundError(err) { + log.V(5).Info("placement group variable for AWS controlplane not defined.") + return nil + } + return err + } + + log = log.WithValues( + "variableName", + h.variableName, + "variableFieldPath", + h.variableFieldPath, + "variableValue", + placementGroupVar, + ) + + return patches.MutateIfApplicable( + obj, + vars, + &holderRef, + selectors.ControlPlane(), log, + func(obj *controlplanev1.KubeadmControlPlaneTemplate) error { + log.WithValues( + "patchedObjectKind", obj.GetObjectKind().GroupVersionKind().String(), + "patchedObjectName", client.ObjectKeyFromObject(obj), + ).Info("setting placement group for local node feature discovery in AWS controlplane KubeadmControlPlaneTemplate") + obj.Spec.Template.Spec.KubeadmConfigSpec.Files = append( + obj.Spec.Template.Spec.KubeadmConfigSpec.Files, + cabpkv1.File{ + Path: PlacementGroupDiscoveryScriptFileOnRemote, + Content: string(PlacementgroupDiscoveryScript), + Permissions: "0700", + }, + ) + obj.Spec.Template.Spec.KubeadmConfigSpec.PreKubeadmCommands = append( + obj.Spec.Template.Spec.KubeadmConfigSpec.PreKubeadmCommands, + PlacementGroupDiscoveryScriptFileOnRemote, + ) + return nil + }, + ) +} diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/inject_controlplane_test.go b/pkg/handlers/aws/mutation/placementgroupnfd/inject_controlplane_test.go new file mode 100644 index 000000000..17adc09e5 --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/inject_controlplane_test.go @@ -0,0 +1,74 @@ +// Copyright 2025 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + . "github.com/onsi/ginkgo/v2" + "github.com/onsi/gomega" + runtimehooksv1 "sigs.k8s.io/cluster-api/exp/runtime/hooks/api/v1alpha1" + + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/handlers/mutation" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/testutils/capitest" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/testutils/capitest/request" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/test/helpers" +) + +var _ = Describe("Generate AWS Placement Group NFD patches for ControlPlane", func() { + patchGenerator := func() mutation.GeneratePatches { + return mutation.NewMetaGeneratePatchesHandler( + "", + helpers.TestEnv.Client, + NewControlPlanePatch(), + ).(mutation.GeneratePatches) + } + + testDefs := []capitest.PatchTestDef{ + { + Name: "unset variable", + }, + { + Name: "placement group set for control plane", + Vars: []runtimehooksv1.Variable{ + capitest.VariableWithValue( + v1alpha1.ClusterConfigVariableName, + v1alpha1.PlacementGroup{ + Name: "test-placement-group", + }, + v1alpha1.ControlPlaneConfigVariableName, + v1alpha1.AWSVariableName, + "placementGroup", + ), + }, + RequestItem: request.NewKubeadmControlPlaneTemplateRequestItem("1234"), + ExpectedPatchMatchers: []capitest.JSONPatchMatcher{ + { + Operation: "add", + Path: "/spec/template/spec/kubeadmConfigSpec/files", + ValueMatcher: gomega.ContainElement(gomega.HaveKeyWithValue( + "path", "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh", + )), + }, + { + Operation: "add", + Path: "/spec/template/spec/kubeadmConfigSpec/preKubeadmCommands", + ValueMatcher: gomega.ContainElement( + "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh", + ), + }, + }, + }, + } + + // create test node for each case + for _, tt := range testDefs { + It(tt.Name, func() { + capitest.AssertGeneratePatches( + GinkgoT(), + patchGenerator, + &tt, + ) + }) + } +}) diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/inject_suite_test.go b/pkg/handlers/aws/mutation/placementgroupnfd/inject_suite_test.go new file mode 100644 index 000000000..f851bf0a7 --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/inject_suite_test.go @@ -0,0 +1,16 @@ +// Copyright 2025 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestPlacementGroupNFDPatch(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "AWS Placement Group NFD mutator suite") +} diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/inject_worker.go b/pkg/handlers/aws/mutation/placementgroupnfd/inject_worker.go new file mode 100644 index 000000000..0f9f859a7 --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/inject_worker.go @@ -0,0 +1,95 @@ +// Copyright 2024 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + "context" + + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + cabpkv1 "sigs.k8s.io/cluster-api/bootstrap/kubeadm/api/v1beta1" + runtimehooksv1 "sigs.k8s.io/cluster-api/exp/runtime/hooks/api/v1alpha1" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/handlers/mutation" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/patches" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/patches/selectors" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/variables" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/placementgroup" +) + +type workerPatchHandler struct { + variableName string + variableFieldPath []string +} + +func NewWorkerPatch() *workerPatchHandler { + return &workerPatchHandler{ + variableName: v1alpha1.WorkerConfigVariableName, + variableFieldPath: []string{ + v1alpha1.AWSVariableName, + placementgroup.VariableName, + }, + } +} + +func (h *workerPatchHandler) Mutate( + ctx context.Context, + obj *unstructured.Unstructured, + vars map[string]apiextensionsv1.JSON, + holderRef runtimehooksv1.HolderReference, + _ client.ObjectKey, + _ mutation.ClusterGetter, +) error { + log := ctrl.LoggerFrom(ctx).WithValues( + "holderRef", holderRef, + ) + + placementGroupVar, err := variables.Get[v1alpha1.PlacementGroup]( + vars, + h.variableName, + h.variableFieldPath..., + ) + if err != nil { + if variables.IsNotFoundError(err) { + log.V(5).Info("placement group variable for AWS worker not defined.") + return nil + } + return err + } + + log = log.WithValues( + "variableName", + h.variableName, + "variableFieldPath", + h.variableFieldPath, + "variableValue", + placementGroupVar, + ) + + return patches.MutateIfApplicable( + obj, + vars, + &holderRef, + selectors.WorkersKubeadmConfigTemplateSelector(), log, + func(obj *cabpkv1.KubeadmConfigTemplate) error { + log.WithValues( + "patchedObjectKind", obj.GetObjectKind().GroupVersionKind().String(), + "patchedObjectName", client.ObjectKeyFromObject(obj), + ).Info("setting placement group for local node feature discovery in AWS workers KubeadmConfig template") + obj.Spec.Template.Spec.Files = append(obj.Spec.Template.Spec.Files, cabpkv1.File{ + Path: PlacementGroupDiscoveryScriptFileOnRemote, + Content: string(PlacementgroupDiscoveryScript), + Permissions: "0700", + }) + obj.Spec.Template.Spec.PreKubeadmCommands = append( + obj.Spec.Template.Spec.PreKubeadmCommands, + PlacementGroupDiscoveryScriptFileOnRemote, + ) + return nil + }, + ) +} diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/inject_worker_test.go b/pkg/handlers/aws/mutation/placementgroupnfd/inject_worker_test.go new file mode 100644 index 000000000..13980562b --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/inject_worker_test.go @@ -0,0 +1,77 @@ +// Copyright 2025 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + . "github.com/onsi/ginkgo/v2" + "github.com/onsi/gomega" + runtimehooksv1 "sigs.k8s.io/cluster-api/exp/runtime/hooks/api/v1alpha1" + + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/handlers/mutation" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/testutils/capitest" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/testutils/capitest/request" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/test/helpers" +) + +var _ = Describe("Generate AWS Placement Group NFD patches for Worker", func() { + patchGenerator := func() mutation.GeneratePatches { + return mutation.NewMetaGeneratePatchesHandler("", helpers.TestEnv.Client, NewWorkerPatch()).(mutation.GeneratePatches) + } + + testDefs := []capitest.PatchTestDef{ + { + Name: "unset variable", + }, + { + Name: "placement group set for workers", + Vars: []runtimehooksv1.Variable{ + capitest.VariableWithValue( + v1alpha1.WorkerConfigVariableName, + v1alpha1.PlacementGroup{ + Name: "test-placement-group", + }, + v1alpha1.AWSVariableName, + "placementGroup", + ), + capitest.VariableWithValue( + runtimehooksv1.BuiltinsName, + map[string]any{ + "machineDeployment": map[string]any{ + "class": "a-worker", + }, + }, + ), + }, + RequestItem: request.NewKubeadmConfigTemplateRequestItem("1234"), + ExpectedPatchMatchers: []capitest.JSONPatchMatcher{ + { + Operation: "add", + Path: "/spec/template/spec/files", + ValueMatcher: gomega.ContainElement(gomega.HaveKeyWithValue( + "path", "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh", + )), + }, + { + Operation: "add", + Path: "/spec/template/spec/preKubeadmCommands", + ValueMatcher: gomega.ContainElement( + "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh", + ), + }, + }, + }, + } + + // create test node for each case + for _, tt := range testDefs { + It(tt.Name, func() { + capitest.AssertGeneratePatches( + GinkgoT(), + patchGenerator, + &tt, + ) + }) + } +}) diff --git a/pkg/handlers/aws/mutation/placementgroupnfd/nfd.go b/pkg/handlers/aws/mutation/placementgroupnfd/nfd.go new file mode 100644 index 000000000..ca355c3c3 --- /dev/null +++ b/pkg/handlers/aws/mutation/placementgroupnfd/nfd.go @@ -0,0 +1,15 @@ +// Copyright 2024 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + _ "embed" +) + +var ( + //go:embed embedded/placementgroup_discovery.sh + PlacementgroupDiscoveryScript []byte + //nolint:lll // this is a constant with long file path + PlacementGroupDiscoveryScriptFileOnRemote = "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh" +) diff --git a/pkg/handlers/eks/mutation/metapatch_handler.go b/pkg/handlers/eks/mutation/metapatch_handler.go index 1461dcc62..94e87df9e 100644 --- a/pkg/handlers/eks/mutation/metapatch_handler.go +++ b/pkg/handlers/eks/mutation/metapatch_handler.go @@ -14,6 +14,7 @@ import ( "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/instancetype" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/network" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/placementgroup" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/placementgroupnfd" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/region" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/securitygroups" "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/tags" @@ -46,6 +47,7 @@ func MetaWorkerPatchHandler(mgr manager.Manager) handlers.Named { securitygroups.NewWorkerPatch(), volumes.NewWorkerPatch(), placementgroup.NewWorkerPatch(), + placementgroupnfd.NewWorkerPatch(), tags.NewWorkerPatch(), } patchHandlers = append(patchHandlers, workerMetaMutators()...) diff --git a/pkg/handlers/eks/mutation/placementgroupnfd/inject_suite_test.go b/pkg/handlers/eks/mutation/placementgroupnfd/inject_suite_test.go new file mode 100644 index 000000000..5e04ebfb2 --- /dev/null +++ b/pkg/handlers/eks/mutation/placementgroupnfd/inject_suite_test.go @@ -0,0 +1,16 @@ +// Copyright 2025 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestPlacementGroupNFDPatch(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "EKS Placement Group NFD mutator suite") +} diff --git a/pkg/handlers/eks/mutation/placementgroupnfd/inject_worker.go b/pkg/handlers/eks/mutation/placementgroupnfd/inject_worker.go new file mode 100644 index 000000000..13afd6be9 --- /dev/null +++ b/pkg/handlers/eks/mutation/placementgroupnfd/inject_worker.go @@ -0,0 +1,96 @@ +// Copyright 2024 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + "context" + + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + runtimehooksv1 "sigs.k8s.io/cluster-api/exp/runtime/hooks/api/v1alpha1" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + eksbootstrapv1 "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/external/sigs.k8s.io/cluster-api-provider-aws/v2/bootstrap/eks/api/v1beta2" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/handlers/mutation" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/patches" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/patches/selectors" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/variables" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/placementgroup" + awsplacementgroupnfd "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/aws/mutation/placementgroupnfd" +) + +type workerPatchHandler struct { + variableName string + variableFieldPath []string +} + +func NewWorkerPatch() *workerPatchHandler { + return &workerPatchHandler{ + variableName: v1alpha1.WorkerConfigVariableName, + variableFieldPath: []string{ + v1alpha1.EKSVariableName, + placementgroup.VariableName, + }, + } +} + +func (h *workerPatchHandler) Mutate( + ctx context.Context, + obj *unstructured.Unstructured, + vars map[string]apiextensionsv1.JSON, + holderRef runtimehooksv1.HolderReference, + _ client.ObjectKey, + _ mutation.ClusterGetter, +) error { + log := ctrl.LoggerFrom(ctx).WithValues( + "holderRef", holderRef, + ) + + placementGroupVar, err := variables.Get[v1alpha1.PlacementGroup]( + vars, + h.variableName, + h.variableFieldPath..., + ) + if err != nil { + if variables.IsNotFoundError(err) { + log.V(5).Info("placement group variable for EKS worker not defined.") + return nil + } + return err + } + + log = log.WithValues( + "variableName", + h.variableName, + "variableFieldPath", + h.variableFieldPath, + "variableValue", + placementGroupVar, + ) + + return patches.MutateIfApplicable( + obj, + vars, + &holderRef, + selectors.WorkersConfigTemplateSelector(eksbootstrapv1.GroupVersion.String(), "NodeadmConfigTemplate"), log, + func(obj *eksbootstrapv1.NodeadmConfigTemplate) error { + log.WithValues( + "patchedObjectKind", obj.GetObjectKind().GroupVersionKind().String(), + "patchedObjectName", client.ObjectKeyFromObject(obj), + ).Info("setting placement group for local node feature discovery in EKS workers NodeadmConfigTemplate") + obj.Spec.Template.Spec.Files = append(obj.Spec.Template.Spec.Files, eksbootstrapv1.File{ + Path: awsplacementgroupnfd.PlacementGroupDiscoveryScriptFileOnRemote, + Content: string(awsplacementgroupnfd.PlacementgroupDiscoveryScript), + Permissions: "0700", + }) + obj.Spec.Template.Spec.PreNodeadmCommands = append( + obj.Spec.Template.Spec.PreNodeadmCommands, + awsplacementgroupnfd.PlacementGroupDiscoveryScriptFileOnRemote, + ) + return nil + }, + ) +} diff --git a/pkg/handlers/eks/mutation/placementgroupnfd/inject_worker_test.go b/pkg/handlers/eks/mutation/placementgroupnfd/inject_worker_test.go new file mode 100644 index 000000000..af3056ddb --- /dev/null +++ b/pkg/handlers/eks/mutation/placementgroupnfd/inject_worker_test.go @@ -0,0 +1,81 @@ +// Copyright 2025 Nutanix. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package placementgroupnfd + +import ( + . "github.com/onsi/ginkgo/v2" + "github.com/onsi/gomega" + runtimehooksv1 "sigs.k8s.io/cluster-api/exp/runtime/hooks/api/v1alpha1" + + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/api/v1alpha1" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/capi/clustertopology/handlers/mutation" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/common/pkg/testutils/capitest" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pkg/handlers/eks/mutation/testutils" + "github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/test/helpers" +) + +var _ = Describe("Generate EKS Placement Group NFD patches for Worker", func() { + patchGenerator := func() mutation.GeneratePatches { + return mutation.NewMetaGeneratePatchesHandler( + "", + helpers.TestEnv.Client, + NewWorkerPatch(), + ).(mutation.GeneratePatches) + } + + testDefs := []capitest.PatchTestDef{ + { + Name: "unset variable", + }, + { + Name: "placement group set for EKS workers", + Vars: []runtimehooksv1.Variable{ + capitest.VariableWithValue( + v1alpha1.WorkerConfigVariableName, + v1alpha1.PlacementGroup{ + Name: "test-placement-group", + }, + v1alpha1.EKSVariableName, + "placementGroup", + ), + capitest.VariableWithValue( + runtimehooksv1.BuiltinsName, + map[string]any{ + "machineDeployment": map[string]any{ + "class": "a-worker", + }, + }, + ), + }, + RequestItem: testutils.NewNodeadmConfigTemplateRequestItem("1234"), + ExpectedPatchMatchers: []capitest.JSONPatchMatcher{ + { + Operation: "add", + Path: "/spec/template/spec/files", + ValueMatcher: gomega.ContainElement(gomega.HaveKeyWithValue( + "path", "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh", + )), + }, + { + Operation: "add", + Path: "/spec/template/spec/preNodeadmCommands", + ValueMatcher: gomega.ContainElement( + "/etc/kubernetes/node-feature-discovery/source.d/placementgroup_discovery.sh", + ), + }, + }, + }, + } + + // create test node for each case + for _, tt := range testDefs { + It(tt.Name, func() { + capitest.AssertGeneratePatches( + GinkgoT(), + patchGenerator, + &tt, + ) + }) + } +})