Skip to content

Commit 47fd880

Browse files
joekrjoekr
authored
Update the machineHealthCheck and advance docs (#267)
Making the MHC a bit more clear which steps to run on specific clusters. This should help make it a bit more clear for users. Adding the setup heterogeneous cluster section to advanced. This should allow users to see how to setup a mix workload windows/linux cluster.
1 parent 92c873a commit 47fd880

File tree

2 files changed

+154
-9
lines changed

2 files changed

+154
-9
lines changed

‎docs/src/gs/advanced.md‎

Lines changed: 127 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,130 @@ go into error state, and the following error will show up in the CAPOCI pod logs
1515

1616
`OCI authentication credentials could not be retrieved from pod or cluster level,please install Cluster API Provider for OCI with OCI authentication credentials or set Cluster Identity in the OCICluster`
1717

18-
[cluster-identity]: ./multi-tenancy.md
18+
## Setup heterogeneous cluster
19+
20+
> This section assumes you have [setup a Windows workload cluster][windows-cluster].
21+
22+
To add Linux nodes to the existing Windows workload cluster use the following YAML as a guide to provision
23+
just the new Linux machines.
24+
25+
Create a file and call it `cluster-template-windows-calico-heterogeneous.yaml`. Then add the following:
26+
27+
```yaml
28+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
29+
kind: OCIMachineTemplate
30+
metadata:
31+
name: "${CLUSTER_NAME}-md-0"
32+
spec:
33+
template:
34+
spec:
35+
imageId: "${OCI_IMAGE_ID}"
36+
compartmentId: "${OCI_COMPARTMENT_ID}"
37+
shape: "${OCI_NODE_MACHINE_TYPE=VM.Standard.E4.Flex}"
38+
shapeConfig:
39+
ocpus: "${OCI_NODE_MACHINE_TYPE_OCPUS=1}"
40+
metadata:
41+
ssh_authorized_keys: "${OCI_SSH_KEY}"
42+
isPvEncryptionInTransitEnabled: ${OCI_NODE_PV_TRANSIT_ENCRYPTION=true}
43+
---
44+
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha4
45+
kind: KubeadmConfigTemplate
46+
metadata:
47+
name: "${CLUSTER_NAME}-md-0"
48+
spec:
49+
template:
50+
spec:
51+
joinConfiguration:
52+
nodeRegistration:
53+
kubeletExtraArgs:
54+
cloud-provider: external
55+
provider-id: oci://{{ ds["id"] }}
56+
---
57+
apiVersion: cluster.x-k8s.io/v1beta1
58+
kind: MachineDeployment
59+
metadata:
60+
name: "${CLUSTER_NAME}-md-0"
61+
spec:
62+
clusterName: "${CLUSTER_NAME}"
63+
replicas: ${NODE_MACHINE_COUNT}
64+
selector:
65+
matchLabels:
66+
template:
67+
spec:
68+
clusterName: "${CLUSTER_NAME}"
69+
version: "${KUBERNETES_VERSION}"
70+
bootstrap:
71+
configRef:
72+
name: "${CLUSTER_NAME}-md-0"
73+
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
74+
kind: KubeadmConfigTemplate
75+
infrastructureRef:
76+
name: "${CLUSTER_NAME}-md-0"
77+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
78+
kind: OCIMachineTemplate
79+
```
80+
81+
Then apply the template
82+
```bash
83+
OCI_IMAGE_ID=<your new linux image OCID> \
84+
OCI_NODE_IMAGE_ID=<your new linux image OCID> \
85+
OCI_COMPARTMENT_ID=<your compartment> \
86+
NODE_MACHINE_COUNT=2 \
87+
OCI_NODE_MACHINE_TYPE=<shape> \
88+
OCI_NODE_MACHINE_TYPE_OCPUS=4 \
89+
OCI_SSH_KEY="<your public ssh key>" \
90+
clusterctl generate cluster <cluster-name> --kubernetes-version <kubernetes-version> \
91+
--target-namespace default \
92+
--from cluster-template-windows-calico-heterogeneous.yaml | kubectl apply -f -
93+
```
94+
95+
After a few minutes the instances will come up and the CNI will be installed.
96+
97+
### Node constraints
98+
99+
All future deployments make sure to setup node constraints using something like [`nodeselctor`](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector). Example:
100+
101+
| Windows | Linux |
102+
| ----------- | ----------- |
103+
| ```nodeSelector: kubernetes.io/os: windows``` | ```nodeSelector:kubernetes.io/os: linux``` |
104+
105+
<br/>
106+
<details>
107+
<summary>nodeSelector examples - click to expand</summary>
108+
109+
Linux nginx deployment example:
110+
```bash
111+
apiVersion: apps/v1
112+
kind: Deployment
113+
metadata:
114+
name: my-nginx-linux
115+
spec:
116+
selector:
117+
matchLabels:
118+
run: my-nginx-linux
119+
replicas: 2
120+
template:
121+
metadata:
122+
labels:
123+
run: my-nginx-linux
124+
spec:
125+
nodeSelector:
126+
kubernetes.io/os: linux
127+
containers:
128+
- args:
129+
- /bin/sh
130+
- -c
131+
- sleep 3600
132+
name: nginx
133+
image: nginx:latest
134+
```
135+
136+
For a Windows deployment example see the [Kubernetes Getting Started: Deploying a Windows workload][windows-kubernetes-deployment] documentation
137+
138+
</details>
139+
140+
Without doing this it is possible that the Kubernetes scheduler will try to deploy your Windows pods onto a Linux worker, or vice versa.
141+
142+
[cluster-identity]: ./multi-tenancy.md
143+
[windows-cluster]: ./create-windows-workload-cluster.md
144+
[windows-kubernetes-deployment]: https://kubernetes.io/docs/concepts/windows/user-guide/#getting-started-deploying-a-windows-workload

‎docs/src/gs/create-mhc-workload-cluster.md‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,15 @@ This will move the machines into a `Ready` state.
2020
Another approach is to install MHC after the cluster is up and healthy (aka Day-2 Operation). This can prevent
2121
machine remediation while setting up the cluster.
2222

23+
Adding the MHC to either control-plane or machine is a multistep process. The steps are run on specific clusters
24+
(e.g. management cluster, workload cluster):
25+
1. Update the spec for future instances (management cluster)
26+
2. Add label to existing nodes (workload cluster)
27+
3. Add the MHC (management cluster)
28+
2329
### Add control-plane MHC
2430

31+
#### Update control plane spec
2532
We need to add the `controlplane.remediation` label to the `KubeadmControlPlane`.
2633

2734
Create a file named `control-plane-patch.yaml` that has this content:
@@ -33,13 +40,18 @@ spec:
3340
controlplane.remediation: ""
3441
```
3542
36-
Then run `kubectl patch KubeadmControlPlane <your-cluster-name>-control-plane --patch-file control-plane-patch.yaml --type=merge`.
43+
Then on the management cluster run
44+
`kubectl patch KubeadmControlPlane <your-cluster-name>-control-plane --patch-file control-plane-patch.yaml --type=merge`.
45+
46+
#### Add label to existing nodes
3747

38-
Then add the new label to any existing control-plane node(s)
48+
Then on the workload cluster add the new label to any existing control-plane node(s)
3949
`kubectl label node <control-plane-name> controlplane.remediation=""`. This will prevent the `KubeadmControlPlane` provisioning
4050
new nodes once the MHC is deployed.
4151

42-
Create a file named `control-plane-mhc.yaml` that has this content:
52+
#### Add the MHC
53+
54+
Finally, create a file named `control-plane-mhc.yaml` that has this content:
4355
```yaml
4456
apiVersion: cluster.x-k8s.io/v1beta1
4557
kind: MachineHealthCheck
@@ -61,12 +73,14 @@ spec:
6173
timeout: 300s
6274
```
6375

64-
Then run `kubectl apply -f control-plane-mhc.yaml`.
76+
Then on the management cluster run `kubectl apply -f control-plane-mhc.yaml`.
6577

6678
Then run `kubectl get machinehealthchecks` to check your MachineHealthCheck sees the expected machines.
6779

6880
### Add machine MHC
6981

82+
#### Update machine spec
83+
7084
We need to add the `machine.remediation` label to the `MachineDeployment`.
7185

7286
Create a file named `machine-patch.yaml` that has this content:
@@ -78,13 +92,18 @@ spec:
7892
machine.remediation: ""
7993
```
8094

81-
Then run `kubectl patch MachineDeployment oci-cluster-stage-md-0 --patch-file machine-patch.yaml --type=merge`.
95+
Then on the management cluster run
96+
`kubectl patch MachineDeployment oci-cluster-stage-md-0 --patch-file machine-patch.yaml --type=merge`.
97+
98+
#### Add label to existing nodes
8299

83-
Then add the new label to any existing control-plane node(s)
100+
Then on the workload cluster add the new label to any existing control-plane node(s)
84101
`kubectl label node <machine-name> machine.remediation=""`. This will prevent the `MachineDeployment` provisioning
85102
new nodes once the MHC is deployed.
86103

87-
Create a file named `machine-mhc.yaml` that has this content:
104+
#### Add the MHC
105+
106+
Finally, create a file named `machine-mhc.yaml` that has this content:
88107
```yaml
89108
apiVersion: cluster.x-k8s.io/v1beta1
90109
kind: MachineHealthCheck
@@ -106,7 +125,7 @@ spec:
106125
timeout: 300s
107126
```
108127

109-
Then run `kubectl apply -f machine-mhc.yaml`.
128+
Then on the management cluster run `kubectl apply -f machine-mhc.yaml`.
110129

111130
Then run `kubectl get machinehealthchecks` to check your MachineHealthCheck sees the expected machines.
112131

0 commit comments

Comments
 (0)