Skip to content

Commit 6a80516

Browse files
joekrjoekr
authored
docs: Add how to create gpu workload cluster docs (#170)
1 parent 57ca292 commit 6a80516

File tree

2 files changed

+342
-0
lines changed

2 files changed

+342
-0
lines changed

‎docs/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
- [Provision a management cluster with OKE](./gs/mgmt/mgmt-oke.md)
1919
- [Install Cluster API for Oracle Cloud Infrastructure](./gs/install-cluster-api.md)
2020
- [Create Workload Cluster](./gs/create-workload-cluster.md)
21+
- [Create GPU Workload Cluster](./gs/create-gpu-workload-cluster.md)
2122
- [Create Workload Templates](./gs/create-workload-templates.md)
2223
- [Using externally managed infrastructure](./gs/externally-managed-cluster-infrastructure.md)
2324
- [Install Oracle Cloud Infrastructure Cloud Controller Manager](./gs/install-oci-ccm.md)

‎docs/src/gs/create-gpu-workload-cluster.md‎

Lines changed: 341 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,341 @@
1+
# Create a GPU workload cluster
2+
3+
## Accessing GPU Shapes
4+
5+
Some shapes are limited to specific regions and specific Availability Domains (AD).
6+
In order to make sure the workload cluster comes up check the region and AD for
7+
shape availability.
8+
9+
### Check shape availability
10+
11+
Make sure the [OCI CLI][install-oci-cli] is installed. Then set the AD information if using
12+
muti-AD regions.
13+
14+
> NOTE: Use the [OCI Regions and Availability Domains][regions] page to figure out which
15+
regions have multiple ADs.
16+
17+
```bash
18+
oci iam availability-domain list --compartment-id=<your compartment> --region=<region>
19+
```
20+
21+
Using the AD `name` from the output start searching for GPU shape availability.
22+
23+
```bash
24+
oci compute shape list --compartment-id=<your compartment> --profile=DEFAULT --region=us-ashburn-1 --availability-domain=<your AD ID> | grep GPU
25+
26+
"shape-name": "BM.GPU3.8"
27+
"shape-name": "BM.GPU4.8"
28+
"shape-name": "VM.GPU3.1"
29+
"shape": "VM.GPU2.1"
30+
```
31+
32+
> NOTE: If the output is empty then the compartment for that region/AD doesn't have GPU shapes.
33+
If you are unable to locate any shapes you may need to submit a
34+
[service limit increase request][compute-service-limit]
35+
36+
## Create a new GPU workload cluster an Ubuntu custom image
37+
38+
> NOTE: Nvidia GPU drivers aren't supported for Oracle Linux at this time. Ubuntu is currently
39+
the only supported OS.
40+
41+
When launching a multi-AD region shapes are likely be limited to a specific AD (example: `US-ASHBURN-AD-2`).
42+
To make sure the cluster comes up without issue specifically target just that AD for the GPU worker nodes.
43+
To do that modify the released version of the `cluster-template-failure-domain-spread.yaml` template.
44+
45+
Download the [latest `cluster-template-failure-domain-spread.yaml`][releases] file and save it as
46+
`cluster-template-gpu.yaml`.
47+
48+
Make sure the modified template has only the `MachineDeployment` section(s) where there is GPU
49+
availability and remove all the others. See [the full example file][example-yaml-file]
50+
that targets only AD 2 (OCI calls them Availability Domains while Cluster-API calls them Failure Domains).
51+
52+
### Virtual instances
53+
54+
The following command will create a workload cluster comprising a single
55+
control plane node and single GPU worker node using the default values as specified in the preceding
56+
[Workload Cluster Parameters][workload-cluster-parameters] table:
57+
58+
> NOTE: The `OCI_NODE_MACHINE_TYPE_OCPUS` must match the OPCU count of the GPU shape.
59+
See the [Compute Shapes][compute-shapes] page to get the OCPU count for the specific shape.
60+
61+
```bash
62+
OCI_COMPARTMENT_ID=<compartment-id> \
63+
OCI_IMAGE_ID=<ubuntu-custom-image-id> \
64+
OCI_SSH_KEY=<ssh-key> \
65+
NODE_MACHINE_COUNT=1 \
66+
OCI_NODE_MACHINE_TYPE=VM.GPU3.1 \
67+
OCI_NODE_MACHINE_TYPE_OCPUS=6 \
68+
OCI_CONTROL_PLANE_MACHINE_TYPE_OCPUS=1 \
69+
OCI_CONTROL_PLANE_MACHINE_TYPE=VM.Standard3.Flex \
70+
CONTROL_PLANE_MACHINE_COUNT=1 \
71+
OCI_SHAPE_MEMORY_IN_GBS= \
72+
KUBERNETES_VERSION=v1.24.4 \
73+
clusterctl generate cluster <cluster-name> \
74+
--target-namespace default \
75+
--from cluster-template-gpu.yaml | kubectl apply -f -
76+
```
77+
78+
### Bare metal instances
79+
80+
The following command uses the `OCI_CONTROL_PLANE_MACHINE_TYPE` and `OCI_NODE_MACHINE_TYPE`
81+
parameters to specify bare metal shapes instead of using CAPOCI's default virtual
82+
instance shape. The `OCI_CONTROL_PLANE_PV_TRANSIT_ENCRYPTION` and `OCI_NODE_PV_TRANSIT_ENCRYPTION`
83+
parameters disable encryption of data in flight between the bare metal instance and the block storage resources.
84+
85+
> NOTE: The `OCI_NODE_MACHINE_TYPE_OCPUS` must match the OPCU count of the GPU shape.
86+
See the [Compute Shapes][compute-shapes] page to get the OCPU count for the specific shape.
87+
88+
```bash
89+
OCI_COMPARTMENT_ID=<compartment-id> \
90+
OCI_IMAGE_ID=<ubuntu-custom-image-id> \
91+
OCI_SSH_KEY=<ssh-key> \
92+
OCI_NODE_MACHINE_TYPE=BM.GPU3.8 \
93+
OCI_NODE_MACHINE_TYPE_OCPUS=52 \
94+
OCI_NODE_PV_TRANSIT_ENCRYPTION=false \
95+
OCI_CONTROL_PLANE_MACHINE_TYPE=VM.Standard3.Flex \
96+
CONTROL_PLANE_MACHINE_COUNT=1 \
97+
OCI_SHAPE_MEMORY_IN_GBS= \
98+
KUBERNETES_VERSION=v1.24.4 \
99+
clusterctl generate cluster <cluster-name> \
100+
--target-namespace default \
101+
--from cluster-template-gpu.yaml | kubectl apply -f -
102+
```
103+
104+
### Access workload cluster Kubeconfig
105+
106+
Execute the following command to list all the workload clusters present:
107+
108+
```bash
109+
kubectl get clusters -A
110+
```
111+
112+
Execute the following command to access the kubeconfig of a workload cluster:
113+
114+
```bash
115+
clusterctl get kubeconfig <cluster-name> -n default > <cluster-name>.kubeconfig
116+
```
117+
118+
### Install a CNI Provider, OCI Cloud Controller Manager and CSI in a self-provisioned cluster
119+
120+
To provision the CNI and Cloud Controller Manager follow the [Install a CNI Provider][install-a-cni-provider]
121+
and the [Install OCI Cloud Controller Manager][install-ccm] sections.
122+
123+
### Install Nvidia GPU Operator
124+
125+
Setup the worker instances to use the GPUs install the [Nvidia GPU Operator][nvidia-overview].
126+
127+
For the most up-to-date install instructions see the [official install instructions][nvidia-install-gpu-operator]. They
128+
layout how to install the [Helm tool][helm-install] and how to setup the Nvidia helm repo.
129+
130+
With Helm setup you can now install the GPU-Operator
131+
132+
```bash
133+
helm install --wait --generate-name \
134+
-n gpu-operator --create-namespace \
135+
nvidia/gpu-operator
136+
```
137+
138+
The pods will take a while to come up but you can check the status:
139+
140+
```bash
141+
kubectl --<cluster-name>.kubeconf get pods -n gpu-operator
142+
```
143+
144+
### Test GPU on worker node
145+
146+
Once all of the GPU-Operator pods are `running` or `completed` deploy the test pod:
147+
148+
```bash
149+
cat <<EOF | kubectl --kubeconfig=<cluster-name>.kubeconf apply -f -
150+
apiVersion: v1
151+
kind: Pod
152+
metadata:
153+
name: cuda-vector-add
154+
spec:
155+
restartPolicy: OnFailure
156+
containers:
157+
- name: cuda-vector-add
158+
# https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
159+
image: "registry.k8s.io/cuda-vector-add:v0.1"
160+
resources:
161+
limits:
162+
nvidia.com/gpu: 1 # requesting 1 GPU
163+
EOF
164+
```
165+
166+
Then check the output logs of the `cuda-vector-add` test pod:
167+
168+
```bash
169+
kubectl --kubeconfig=<cluster-name>.kubeconf logs cuda-vector-add -n default
170+
171+
[Vector addition of 50000 elements]
172+
Copy input data from the host memory to the CUDA device
173+
CUDA kernel launch with 196 blocks of 256 threads
174+
Copy output data from the CUDA device to the host memory
175+
Test PASSED
176+
Done
177+
```
178+
179+
## Example yaml file
180+
181+
This is an example file using a modified version of `cluster-template-failure-domain-spread.yaml`
182+
to target AD 2 (example: `US-ASHBURN-AD-2`).
183+
184+
```yaml
185+
apiVersion: cluster.x-k8s.io/v1beta1
186+
kind: Cluster
187+
metadata:
188+
labels:
189+
cluster.x-k8s.io/cluster-name: "${CLUSTER_NAME}"
190+
name: "${CLUSTER_NAME}"
191+
namespace: "${NAMESPACE}"
192+
spec:
193+
clusterNetwork:
194+
pods:
195+
cidrBlocks:
196+
- ${POD_CIDR:="192.168.0.0/16"}
197+
serviceDomain: ${SERVICE_DOMAIN:="cluster.local"}
198+
services:
199+
cidrBlocks:
200+
- ${SERVICE_CIDR:="10.128.0.0/12"}
201+
infrastructureRef:
202+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
203+
kind: OCICluster
204+
name: "${CLUSTER_NAME}"
205+
namespace: "${NAMESPACE}"
206+
controlPlaneRef:
207+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
208+
kind: KubeadmControlPlane
209+
name: "${CLUSTER_NAME}-control-plane"
210+
namespace: "${NAMESPACE}"
211+
---
212+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
213+
kind: OCICluster
214+
metadata:
215+
labels:
216+
cluster.x-k8s.io/cluster-name: "${CLUSTER_NAME}"
217+
name: "${CLUSTER_NAME}"
218+
spec:
219+
compartmentId: "${OCI_COMPARTMENT_ID}"
220+
---
221+
kind: KubeadmControlPlane
222+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
223+
metadata:
224+
name: "${CLUSTER_NAME}-control-plane"
225+
namespace: "${NAMESPACE}"
226+
spec:
227+
version: "${KUBERNETES_VERSION}"
228+
replicas: ${CONTROL_PLANE_MACHINE_COUNT}
229+
machineTemplate:
230+
infrastructureRef:
231+
kind: OCIMachineTemplate
232+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
233+
name: "${CLUSTER_NAME}-control-plane"
234+
namespace: "${NAMESPACE}"
235+
kubeadmConfigSpec:
236+
clusterConfiguration:
237+
kubernetesVersion: ${KUBERNETES_VERSION}
238+
apiServer:
239+
certSANs: [localhost, 127.0.0.1]
240+
dns: {}
241+
etcd: {}
242+
networking: {}
243+
scheduler: {}
244+
initConfiguration:
245+
nodeRegistration:
246+
criSocket: /var/run/containerd/containerd.sock
247+
kubeletExtraArgs:
248+
cloud-provider: external
249+
provider-id: oci://{{ ds["id"] }}
250+
joinConfiguration:
251+
discovery: {}
252+
nodeRegistration:
253+
criSocket: /var/run/containerd/containerd.sock
254+
kubeletExtraArgs:
255+
cloud-provider: external
256+
provider-id: oci://{{ ds["id"] }}
257+
---
258+
kind: OCIMachineTemplate
259+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
260+
metadata:
261+
name: "${CLUSTER_NAME}-control-plane"
262+
spec:
263+
template:
264+
spec:
265+
imageId: "${OCI_IMAGE_ID}"
266+
compartmentId: "${OCI_COMPARTMENT_ID}"
267+
shape: "${OCI_CONTROL_PLANE_MACHINE_TYPE=VM.Standard.E4.Flex}"
268+
shapeConfig:
269+
ocpus: "${OCI_CONTROL_PLANE_MACHINE_TYPE_OCPUS=1}"
270+
metadata:
271+
ssh_authorized_keys: "${OCI_SSH_KEY}"
272+
isPvEncryptionInTransitEnabled: ${OCI_CONTROL_PLANE_PV_TRANSIT_ENCRYPTION=true}
273+
---
274+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
275+
kind: OCIMachineTemplate
276+
metadata:
277+
name: "${CLUSTER_NAME}-md"
278+
spec:
279+
template:
280+
spec:
281+
imageId: "${OCI_IMAGE_ID}"
282+
compartmentId: "${OCI_COMPARTMENT_ID}"
283+
shape: "${OCI_NODE_MACHINE_TYPE=VM.Standard.E4.Flex}"
284+
shapeConfig:
285+
ocpus: "${OCI_NODE_MACHINE_TYPE_OCPUS=1}"
286+
metadata:
287+
ssh_authorized_keys: "${OCI_SSH_KEY}"
288+
isPvEncryptionInTransitEnabled: ${OCI_NODE_PV_TRANSIT_ENCRYPTION=true}
289+
---
290+
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha4
291+
kind: KubeadmConfigTemplate
292+
metadata:
293+
name: "${CLUSTER_NAME}-md"
294+
spec:
295+
template:
296+
spec:
297+
joinConfiguration:
298+
nodeRegistration:
299+
kubeletExtraArgs:
300+
cloud-provider: external
301+
provider-id: oci://{{ ds["id"] }}
302+
---
303+
apiVersion: cluster.x-k8s.io/v1beta1
304+
kind: MachineDeployment
305+
metadata:
306+
name: "${CLUSTER_NAME}-fd-2-md-0"
307+
spec:
308+
clusterName: "${CLUSTER_NAME}"
309+
replicas: ${NODE_MACHINE_COUNT}
310+
selector:
311+
matchLabels:
312+
template:
313+
spec:
314+
clusterName: "${CLUSTER_NAME}"
315+
version: "${KUBERNETES_VERSION}"
316+
bootstrap:
317+
configRef:
318+
name: "${CLUSTER_NAME}-md"
319+
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
320+
kind: KubeadmConfigTemplate
321+
infrastructureRef:
322+
name: "${CLUSTER_NAME}-md"
323+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
324+
kind: OCIMachineTemplate
325+
# Cluster-API calls them Failure Domains while OCI calls them Availability Domains
326+
# In the example this would be targeting US-ASHBURN-AD-2
327+
failureDomain: "2"
328+
```
329+
330+
[workload-cluster-parameters]: ../gs/create-workload-cluster.md#workload-cluster-parameters
331+
[install-a-cni-provider]: ../gs/create-workload-cluster.md#install-a-cni-provider
332+
[install-ccm]: ../gs/create-workload-cluster.md#install-oci-cloud-controller-manager-and-csi-in-a-self-provisioned-cluster
333+
[install-oci-cli]: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm
334+
[regions]: https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm
335+
[compute-service-limit]: https://docs.oracle.com/en-us/iaas/Content/General/Concepts/servicelimits.htm#computelimits
336+
[example-yaml-file]: #example-yaml-file
337+
[nvidia-overview]: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/overview.html
338+
[nvidia-install-gpu-operator]: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-nvidia-gpu-operator
339+
[helm-install]: https://helm.sh/docs/intro/install/
340+
[compute-shapes]: https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm
341+
[releases]: https://github.com/oracle/cluster-api-provider-oci/releases

0 commit comments

Comments
 (0)