node-efa: extend doc for Bottlerocket

Signed-off-by: Yutong Sun <[email protected]>

Author: ytsssun

latest/ug/ml/node-efa.adoc

@@ -124,17 +124,114 @@ If you don't have an existing cluster, you can run the following command to crea
 eksctl create cluster -f efa-cluster.yaml
 ----
 +
-NOTE: Because the instance type used in this example has GPUs, `eksctl` automatically installs the NVIDIA Kubernetes device plugin on each instance for you.
-. Deploy the EFA Kubernetes device plugin.
+NOTE: Because the instance type used in this example has GPUs, `eksctl` automatically installs the NVIDIA Kubernetes device plugin on each instance for you. When `efaEnabled: true` is set in your configuration, `eksctl` also automatically deploys the EFA device plugin on your nodes.
+
+[#efa-bottlerocket]
+=== Using Bottlerocket with EFA
+
+Bottlerocket AMI version 1.28.0 and later include official support for EFA. To use Bottlerocket for EFA-enabled nodes, specify `amiFamily: Bottlerocket` in your configuration. If you need to use a custom AMI ID, you must use standard `nodeGroups` instead of `managedNodeGroups`.
+
+Here's an example configuration:
+
+[source,yaml,subs="verbatim,attributes"]
+----
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+metadata:
+  name: my-efa-bottlerocket-cluster
+  region: region-code
+  version: "1.XX"
+
+iam:
+  withOIDC: true
+
+availabilityZones: ["us-west-2a", "us-west-2c"]  
+
+nodeGroups:
+  - name: my-efa-bottlerocket-ng
+    instanceType: p5.48xlarge
+    minSize: 1
+    desiredCapacity: 2
+    maxSize: 3
+    availabilityZones: ["us-west-2a"]
+    volumeSize: 300
+    privateNetworking: true
+    efaEnabled: true
+    amiFamily: Bottlerocket
+    bottlerocket:
+      enableAdminContainer: true
+      settings:
+        kernel:
+          sysctl:
+            "vm.nr_hugepages": "3000"  # Configures 3000 * 2Mi = 6000Mi hugepages
+----
+
+The `vm.nr_hugepages` sysctl setting above configures the number of 2Mi hugepages. In this example, 3000 means 3000 * 2Mi = 6000Mi of hugepages.
+
+[#verify-efa-device-plugin]
+=== Verify EFA Device Plugin Installation
+
+When you create a node group with `efaEnabled: true`, eksctl automatically deploys the EFA Kubernetes device plugin for you. You can verify that the device plugin is installed and functioning correctly:
+
+. Check the DaemonSet status:
++
+[source,bash,subs="verbatim,attributes"]
+----
+kubectl get daemonsets -n kube-system
+----
 +
-The EFA Kubernetes device plugin detects and advertises EFA interfaces as allocatable resources to Kubernetes. An application can consume the extended resource type `vpc.amazonaws.com/efa` in a Pod request spec just like CPU and memory. For more information, see https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#consuming-extended-resources[Consuming extended resources] in the Kubernetes documentation. Once requested, the plugin automatically assigns and mounts an EFA interface to the Pod. Using the device plugin simplifies EFA setup and does not require a Pod to run in privileged mode.
+Sample output:
 +
 [source,bash,subs="verbatim,attributes"]
 ----
-helm repo add eks https://aws.github.io/eks-charts
-helm install aws-efa-k8s-device-plugin --namespace kube-system eks/aws-efa-k8s-device-plugin
+NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
+aws-efa-k8s-device-plugin-daemonset   2         2         2       2            2           <none>          6m16s
+...
 ----
++
+Here, the EFA device plugin DaemonSet is running on two nodes. Both are READY and AVAILABLE.
 
+. Next, verify the pods created by the DaemonSet:
++
+[source,bash,subs="verbatim,attributes"]
+----
+kubectl get pods -n kube-system -l name=aws-efa-k8s-device-plugin
+----
++
+Sample output:
++
+[source,bash,subs="verbatim,attributes"]
+----
+NAME                                        READY   STATUS    RESTARTS   AGE
+aws-efa-k8s-device-plugin-daemonset-d68bs   1/1     Running   0          6m16s
+aws-efa-k8s-device-plugin-daemonset-w4l8t   1/1     Running   0          6m16s
+----
++
+The EFA device plugin pods are in a Running state, confirming that the plugin is successfully deployed and operational.
+
+. Verify resource registration:
++
+You can confirm that the `vpc.amazonaws.com/efa` resource is registered with the kubelet by describing the nodes:
++
+[source,bash,subs="verbatim,attributes"]
+----
+kubectl describe nodes
+----
++
+If the EFA resource is properly registered, you will see it listed under the node's Capacity and Allocatable resources. For example:
++
+[source,bash,subs="verbatim,attributes"]
+----
+Capacity:
+  ...
+  vpc.amazonaws.com/efa:  4
+Allocatable:
+  ...
+  vpc.amazonaws.com/efa:  4
+----
++
+This output confirms that the node recognizes the EFA resource, making it available for pods that request it.
 
 [#efa-application]
 == (Optional) Test the performance of the EFA
@@ -305,4 +402,4 @@ View the log for the `nccl-tests-launcher` Pod. Replace [.replaceable]`nbql9` wi
 kubectl logs -f nccl-tests-launcher-nbql9
 ----
 
-If the test completed successfully, you can deploy your applications that use the Nvidia Collective Communication Library.
+If the test completed successfully, you can deploy your applications that use the Nvidia Collective Communication Library.