Skip to content

Commit 3aae0ff

Browse files
mburke5678mburke5678
authored
Merge pull request #26647 from mburke5678/BZ-1859050-follow-up
Reorganizing node selector topic
2 parents abc34b0 + d1a355c commit 3aae0ff

File tree

5 files changed

+244
-252
lines changed

5 files changed

+244
-252
lines changed

‎modules/nodes-scheduler-node-selectors-about.adoc‎

Lines changed: 213 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -2,48 +2,225 @@
22
//
33
// * nodes/nodes-scheduler-node-selector.adoc
44

5-
[id="nodes-scheduler-node-selector-about_{context}"]
6-
= Understanding node selectors
5+
[id="nodes-scheduler-node-selectors-about_{context}"]
6+
= About node selectors
77

8-
Using _node selectors_, you can ensure that pods are only placed onto nodes with specific labels. As a cluster administrator, you can
9-
use the Pod Node Constraints admission controller to set a policy that prevents users without the `pods/binding` permission
10-
from using node selectors to schedule pods.
8+
You can use node selectors on pods and labels on nodes to control where the pod is scheduled. With node selectors, {product-title} schedules the pods on nodes that contain matching labels.
119

12-
The `nodeSelectorLabelBlacklist` admission controller field gives you control over the labels that certain roles can specify in a pod configuration's
13-
`nodeSelector` field. Users, service accounts, and groups that have the
14-
`pods/binding` permission role can specify any node selector. Those without the
15-
`pods/binding` permission are prohibited from setting a `nodeSelector` for any
16-
label that appears in `nodeSelectorLabelBlacklist`.
10+
You can use a node selector to place specific pods on specific nodes, cluster-wide node selectors to place new pods on specific nodes anywhere in the cluster, and project node selectors to place new pods in a project on specific nodes.
1711

18-
For example, an {product-title} cluster might consist of five data
19-
centers spread across two regions. In the U.S., `us-east`, `us-central`, and
20-
`us-west`; and in the Asia-Pacific region (APAC), `apac-east` and `apac-west`.
21-
Each node in each geographical region is labeled accordingly. For example,
22-
`region: us-east`.
12+
For example, as a cluster administrator, you can create an infrastructure where application developers can deploy pods only onto the nodes closest to their geographical location by including a node selector in every pod they create. In this example, the cluster consists of five data centers spread across two regions. In the U.S., label the nodes as `us-east`, `us-central`, or `us-west`. In the Asia-Pacific region (APAC), label the nodes as `apac-east` or `apac-west`. The developers can add a node selector to the pods they create to ensure the pods get scheduled on those nodes.
2313

14+
A pod is not scheduled if the `Pod` object contains a node selector, but no node has a matching label.
15+
16+
[IMPORTANT]
17+
====
18+
If you are using node selectors and node affinity in the same pod configuration, the following rules control pod placement onto nodes:
19+
20+
* If you configure both `nodeSelector` and `nodeAffinity`, both conditions must be satisfied for the pod to be scheduled onto a candidate node.
21+
22+
* If you specify multiple `nodeSelectorTerms` associated with `nodeAffinity` types, then the pod can be scheduled onto a node if one of the `nodeSelectorTerms` is satisfied.
23+
24+
* If you specify multiple `matchExpressions` associated with `nodeSelectorTerms`, then the pod can be scheduled onto a node only if all `matchExpressions` are satisfied.
25+
====
26+
27+
Node selectors on specific pods and nodes::
28+
+
29+
You can control which node a specific pod is scheduled on by using node selectors and labels.
30+
+
31+
To use node selectors and labels, first label the node to avoid pods being descheduled, then add the node selector to the pod.
32+
+
33+
[NOTE]
34+
====
35+
You cannot add a node selector directly to an existing scheduled pod. You must label the object that controls the pod, such as deployment config.
36+
====
37+
+
38+
For example, the following `Node` object has the `region: east` label:
39+
+
40+
.Sample `Node` object with a label
41+
[source,yaml]
42+
----
43+
kind: Node
44+
apiVersion: v1
45+
metadata:
46+
name: ip-10-0-131-14.ec2.internal
47+
selfLink: /api/v1/nodes/ip-10-0-131-14.ec2.internal
48+
uid: 7bc2580a-8b8e-11e9-8e01-021ab4174c74
49+
resourceVersion: '478704'
50+
creationTimestamp: '2019-06-10T14:46:08Z'
51+
labels:
52+
beta.kubernetes.io/os: linux
53+
failure-domain.beta.kubernetes.io/zone: us-east-1a
54+
node.openshift.io/os_version: '4.5'
55+
node-role.kubernetes.io/worker: ''
56+
failure-domain.beta.kubernetes.io/region: us-east-1
57+
node.openshift.io/os_id: rhcos
58+
beta.kubernetes.io/instance-type: m4.large
59+
kubernetes.io/hostname: ip-10-0-131-14
60+
beta.kubernetes.io/arch: amd64
61+
region: east <1>
62+
----
63+
<1> Label to match the pod node selector.
64+
+
65+
A pod has the `type: user-node,region: east` node selector:
66+
+
67+
.Sample `Pod` object with node selectors
68+
[source,yaml]
69+
----
70+
apiVersion: v1
71+
kind: Pod
72+
73+
....
74+
75+
spec:
76+
nodeSelector: <1>
77+
region: east
78+
type: user-node
79+
----
80+
<1> Node selectors to match the node label.
81+
+
82+
When you create the pod using the example pod spec, it can be scheduled on the example node.
83+
84+
Default cluster-wide node selectors::
85+
+
86+
With default cluster-wide node selectors, when you create a pod in that cluster, {product-title} adds the default node selectors to the pod and schedules
87+
the pod on nodes with matching labels.
88+
+
89+
For example, the following `Scheduler` object has the default cluster-wide `region=east` and `type=user-node` node selectors:
90+
+
91+
.Example Scheduler Operator Custom Resource
92+
[source,yaml]
93+
----
94+
apiVersion: config.openshift.io/v1
95+
kind: Scheduler
96+
metadata:
97+
name: cluster
98+
...
99+
100+
spec:
101+
defaultNodeSelector: type=user-node,region=east
102+
...
103+
104+
----
105+
+
106+
A node in that cluster has the `type=user-node,region=east` labels:
107+
+
108+
.Example `Node` object
109+
[source,yaml]
110+
----
111+
apiVersion: v1
112+
kind: Node
113+
metadata:
114+
name: ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4
115+
...
116+
labels:
117+
region: east
118+
type: user-node
119+
...
120+
121+
----
122+
+
123+
.Example `Pod` object with a node selector
124+
[source,terminal]
125+
----
126+
apiVersion: v1
127+
kind: Pod
128+
...
129+
130+
spec:
131+
nodeSelector:
132+
region: east
133+
...
134+
135+
----
136+
+
137+
When you create the pod using the example pod spec in the example cluster, the pod is created with the cluster-wide node selector and is scheduled on the labeled node:
138+
+
139+
[source,terminal]
140+
.Example pod list with the pod on the labeled node
141+
----
142+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
143+
pod-s1 1/1 Running 0 20s 10.131.2.6 ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4 <none> <none>
144+
----
145+
+
24146
[NOTE]
25147
====
26-
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
27-
See Updating Labels on Nodes for details on assigning labels.
28-
endif::openshift-enterprise,openshift-webscale,openshift-origin[]
29-
ifdef::openshift-dedicated[]
30-
(request changes by opening a support case on the
31-
https://access.redhat.com/support/[Red Hat Customer Portal])
32-
endif::openshift-dedicated[]
148+
If the project where you create the pod has a project node selector, that selector takes preference over a cluster-wide node selector. Your pod is not created or scheduled if the pod does not have the project node selector.
33149
====
34150

35-
As a cluster administrator, you can create an infrastructure where application
36-
developers should be deploying pods only onto the nodes closest to their
37-
geographical location. You can create a node selector, grouping the U.S. data centers into `superregion: us` and the APAC
38-
data centers into `superregion: apac`.
39-
40-
To maintain an even loading of resources per data center, you can add the
41-
desired `region` to the `nodeSelectorLabelBlacklist` section of a master
42-
configuration. Then, whenever a developer located in the U.S. creates a pod, it
43-
is deployed onto a node in one of the regions with the `superregion: us` label.
44-
If the developer tries to target a specific region for their pod (for example,
45-
`region: us-east`), they receive an error. If they try again, without the
46-
node selector on their pod, it can still be deployed onto the region they tried
47-
to target, because `superregion: us` is set as the project-level node selector,
48-
and nodes labeled `region: us-east` are also labeled `superregion: us`.
151+
Project node selectors::
152+
+
153+
With project node selectors, when you create a pod in this project, {product-title} adds the node selectors to the pod and schedules the pods on a node with matching labels. If there is a cluster-wide default node selector, a project node selector takes preference.
154+
+
155+
For example, the following project has the `region=east` node selector:
156+
+
157+
.Example `Namespace` object
158+
[source,yaml]
159+
----
160+
apiVersion: v1
161+
kind: Namespace
162+
metadata:
163+
name: east-region
164+
annotations:
165+
openshift.io/node-selector: "region=east"
166+
...
167+
168+
----
169+
+
170+
The following node has the `type=user-node,region=east` labels:
171+
+
172+
.Example `Node` object
173+
[source,yaml]
174+
----
175+
apiVersion: v1
176+
kind: Node
177+
metadata:
178+
name: ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4
179+
...
180+
labels:
181+
region: east
182+
type: user-node
183+
...
184+
185+
----
186+
+
187+
When you create the pod using the example pod spec in this example project, the pod is created with the project node selectors and is scheduled on the labeled node:
188+
+
189+
.Example Pod object
190+
[source,yaml]
191+
----
192+
apiVersion: v1
193+
kind: Pod
194+
metadata:
195+
namespace: east-region
196+
...
197+
spec:
198+
nodeSelector:
199+
region: east
200+
type: user-node
201+
...
202+
----
203+
+
204+
[source,terminal]
205+
.Example pod list with the pod on the labeled node
206+
----
207+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
208+
pod-s1 1/1 Running 0 20s 10.131.2.6 ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4 <none> <none>
209+
----
210+
+
211+
A pod in the project is not created or scheduled if the pod contains different node selectors. For example, if you deploy the following pod into the example project, it is not be created:
212+
+
213+
.Example Pod object with an invalid node selector
214+
[source,yaml]
215+
----
216+
apiVersion: v1
217+
kind: Pod
218+
...
219+
220+
spec:
221+
nodeSelector:
222+
region: west
223+
224+
....
225+
----
49226

0 commit comments

Comments
 (0)