Skip to content

Commit 930fa86

Browse files
bergerhofferbergerhoffer
authored
Merge pull request #23827 from bergerhoffer/OSDOCS-1225
OSDOCS-1225: Adding custom scheduler docs
2 parents 5d4c8ff + 52466d2 commit 930fa86

File tree

4 files changed

+296
-0
lines changed

4 files changed

+296
-0
lines changed

‎_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1680,6 +1680,8 @@ Topics:
16801680
# File: nodes-scheduler-node-projects
16811681
# - Name: Keeping your cluster balanced using the descheduler
16821682
# File: nodes-scheduler-descheduler
1683+
- Name: Running a custom scheduler
1684+
File: nodes-custom-scheduler
16831685
- Name: Evicting pods using the descheduler
16841686
File: nodes-descheduler
16851687
- Name: Using Jobs and DaemonSets

‎modules/nodes-custom-scheduler-deploying.adoc‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * nodes/scheduling/nodes-custom-scheduler.adoc
4+
5+
[id="nodes-custom-scheduler-deploying_{context}"]
6+
= Deploying a custom scheduler
7+
8+
To include a custom scheduler in your cluster, include the image for a custom scheduler in a deployment.
9+
10+
.Prerequisites
11+
12+
* You have access to the cluster as a user with the `cluster-admin` role.
13+
* You have a scheduler binary.
14+
+
15+
[NOTE]
16+
====
17+
Information on how to create a scheduler binary is outside the scope of this document. For an example, see link:https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers[Configure Multiple Schedulers] in the Kubernetes documentation. Note that the actual functionality of your custom scheduler is not supported by Red Hat.
18+
====
19+
* You have created an image containing the scheduler binary and pushed it to a registry.
20+
21+
.Procedure
22+
23+
. Create a file that contains the deployment resources for the custom scheduler:
24+
+
25+
.Example `custom-scheduler.yaml` file
26+
[source,yaml]
27+
----
28+
apiVersion: v1
29+
kind: ServiceAccount
30+
metadata:
31+
name: custom-scheduler
32+
namespace: kube-system <1>
33+
---
34+
apiVersion: rbac.authorization.k8s.io/v1
35+
kind: ClusterRoleBinding
36+
metadata:
37+
name: custom-scheduler-as-kube-scheduler
38+
subjects:
39+
- kind: ServiceAccount
40+
name: custom-scheduler
41+
namespace: kube-system <1>
42+
roleRef:
43+
kind: ClusterRole
44+
name: system:kube-scheduler
45+
apiGroup: rbac.authorization.k8s.io
46+
---
47+
apiVersion: apps/v1
48+
kind: Deployment
49+
metadata:
50+
labels:
51+
component: scheduler
52+
tier: control-plane
53+
name: custom-scheduler
54+
namespace: kube-system <1>
55+
spec:
56+
selector:
57+
matchLabels:
58+
component: scheduler
59+
tier: control-plane
60+
replicas: 1
61+
template:
62+
metadata:
63+
labels:
64+
component: scheduler
65+
tier: control-plane
66+
version: second
67+
spec:
68+
serviceAccountName: custom-scheduler
69+
containers:
70+
- command:
71+
- /usr/local/bin/kube-scheduler
72+
- --address=0.0.0.0
73+
- --leader-elect=false
74+
- --scheduler-name=custom-scheduler <2>
75+
image: "<namespace>/<image_name>:<tag>" <3>
76+
livenessProbe:
77+
httpGet:
78+
path: /healthz
79+
port: 10251
80+
initialDelaySeconds: 15
81+
name: kube-second-scheduler
82+
readinessProbe:
83+
httpGet:
84+
path: /healthz
85+
port: 10251
86+
resources:
87+
requests:
88+
cpu: '0.1'
89+
securityContext:
90+
privileged: false
91+
volumeMounts: []
92+
hostNetwork: false
93+
hostPID: false
94+
volumes: []
95+
----
96+
<1> This procedure uses the `kube-system` namespace, but you can use the namespace of your choosing.
97+
<2> The command for your custom scheduler might require different arguments. For example, you can pass configuration as a mounted volume using the `--config` argument.
98+
<3> Specify the container image that you created for the custom scheduler.
99+
100+
. Create the deployment resources in the cluster:
101+
+
102+
[source,terminal]
103+
----
104+
$ oc create -f custom-scheduler.yaml
105+
----
106+
107+
.Verification
108+
109+
* Verify that the scheduler pod is running:
110+
+
111+
[source,terminal]
112+
----
113+
$ oc get pods -n kube-system
114+
----
115+
+
116+
The custom scheduler pod is listed as `Running`:
117+
+
118+
[source,terminal]
119+
----
120+
NAME READY STATUS RESTARTS AGE
121+
custom-scheduler-6cd7c4b8bc-854zb 1/1 Running 0 2m
122+
----

‎modules/nodes-custom-scheduler-pods.adoc‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * nodes/scheduling/nodes-custom-scheduler.adoc
4+
5+
[id="nodes-custom-scheduler-deploying-pods_{context}"]
6+
= Deploying pods using a custom scheduler
7+
8+
After the custom scheduler is deployed in your cluster, you can configure pods to use that scheduler instead of the default scheduler.
9+
10+
[NOTE]
11+
====
12+
Each scheduler has a separate view of resources in a cluster. For that reason, each scheduler should operate over its own set of nodes.
13+
14+
If two or more schedulers operate on the same node, they might intervene with each other and schedule more pods on the same node than there are available resources for. Pods might get rejected due to insufficient resources in this case.
15+
====
16+
17+
.Prerequisites
18+
19+
* You have access to the cluster as a user with the `cluster-admin` role.
20+
* The custom scheduler has been deployed in the cluster.
21+
22+
.Procedure
23+
24+
. If your cluster uses role-based access control (RBAC), add the custom scheduler name to the `system:kube-scheduler` cluster role.
25+
26+
.. Edit the `system:kube-scheduler` cluster role:
27+
+
28+
[source,terminal]
29+
----
30+
$ oc edit clusterrole system:kube-scheduler
31+
----
32+
33+
.. Add the name of the custom scheduler to the `resourceNames` lists for the `leases` and `endpoints` resources:
34+
+
35+
[source,yaml]
36+
----
37+
apiVersion: rbac.authorization.k8s.io/v1
38+
kind: ClusterRole
39+
metadata:
40+
annotations:
41+
rbac.authorization.kubernetes.io/autoupdate: "true"
42+
creationTimestamp: "2021-07-07T10:19:14Z"
43+
labels:
44+
kubernetes.io/bootstrapping: rbac-defaults
45+
name: system:kube-scheduler
46+
resourceVersion: "125"
47+
uid: 53896c70-b332-420a-b2a4-f72c822313f2
48+
rules:
49+
...
50+
- apiGroups:
51+
- coordination.k8s.io
52+
resources:
53+
- leases
54+
verbs:
55+
- create
56+
- apiGroups:
57+
- coordination.k8s.io
58+
resourceNames:
59+
- kube-scheduler
60+
- custom-scheduler <1>
61+
resources:
62+
- leases
63+
verbs:
64+
- get
65+
- update
66+
- apiGroups:
67+
- ""
68+
resources:
69+
- endpoints
70+
verbs:
71+
- create
72+
- apiGroups:
73+
- ""
74+
resourceNames:
75+
- kube-scheduler
76+
- custom-scheduler <1>
77+
resources:
78+
- endpoints
79+
verbs:
80+
- get
81+
- update
82+
...
83+
----
84+
<1> This example uses `custom-scheduler` as the custom scheduler name.
85+
86+
. Create a `Pod` configuration and specify the name of the custom scheduler in the `schedulerName` parameter:
87+
+
88+
.Example `custom-scheduler-example.yaml` file
89+
[source,yaml]
90+
----
91+
apiVersion: v1
92+
kind: Pod
93+
metadata:
94+
name: custom-scheduler-example
95+
labels:
96+
name: custom-scheduler-example
97+
spec:
98+
schedulerName: custom-scheduler <1>
99+
containers:
100+
- name: pod-with-second-annotation-container
101+
image: docker.io/ocpqe/hello-pod
102+
----
103+
<1> The name of the custom scheduler to use, which is `custom-scheduler` in this example. When no scheduler name is supplied, the pod is automatically scheduled using the default scheduler.
104+
105+
. Create the pod:
106+
+
107+
[source,terminal]
108+
----
109+
$ oc create -f custom-scheduler-example.yaml
110+
----
111+
112+
.Verification
113+
114+
. Enter the following command to check that the pod was created:
115+
+
116+
[source,terminal]
117+
----
118+
$ oc get pod custom-scheduler-example
119+
----
120+
+
121+
The `custom-scheduler-example` pod is listed in the output:
122+
+
123+
[source,terminal]
124+
----
125+
NAME READY STATUS RESTARTS AGE
126+
custom-scheduler-example 1/1 Running 0 4m
127+
----
128+
129+
. Enter the following command to check that the custom scheduler has scheduled the pod:
130+
+
131+
[source,terminal]
132+
----
133+
$ oc describe pod custom-scheduler-example
134+
----
135+
+
136+
The scheduler, `custom-scheduler`, is listed as shown in the following truncated output:
137+
+
138+
[source,terminal]
139+
----
140+
Events:
141+
Type Reason Age From Message
142+
---- ------ ---- ---- -------
143+
Normal Scheduled <unknown> custom-scheduler Successfully assigned default/custom-scheduler-example to <node_name>
144+
----

‎nodes/scheduling/nodes-custom-scheduler.adoc‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
[id="nodes-custom-scheduler"]
2+
= Running a custom scheduler
3+
include::modules/common-attributes.adoc[]
4+
:context: nodes-custom-scheduler
5+
6+
toc::[]
7+
8+
You can run multiple custom schedulers alongside the default scheduler and configure which scheduler to use for each pod.
9+
10+
[IMPORTANT]
11+
====
12+
It is supported to use a custom scheduler with {product-title}, but Red Hat does not directly support the functionality of the custom scheduler.
13+
14+
For information on how to configure the default scheduler, see xref:../../nodes/scheduling/nodes-scheduler-default.adoc#nodes-scheduler-default[Configuring the default scheduler to control pod placement].
15+
====
16+
17+
To schedule a given pod using a specific scheduler, xref:../../nodes/scheduling/nodes-custom-scheduler.adoc#nodes-custom-scheduler-deploying-pods_nodes-custom-scheduler[specify the name of the scheduler in that `Pod` specification].
18+
19+
// Deploying a custom scheduler
20+
include::modules/nodes-custom-scheduler-deploying.adoc[leveloffset=+1]
21+
22+
// Deploying pods using a custom scheduler
23+
include::modules/nodes-custom-scheduler-pods.adoc[leveloffset=+1]
24+
25+
[id="additional-resources_nodes-custom-scheduler"]
26+
== Additional resources
27+
28+
* xref:../../openshift_images/create-images.adoc#images-create-guidelines_create-images[Learning container best practices]

0 commit comments

Comments
 (0)