RHDEVDOCS-4967-update-nodeselector-example-for-moving-monitoring-components

Author: bburt-rh

modules/monitoring-moving-monitoring-components-to-different-nodes.adoc

@@ -6,30 +6,42 @@
 [id="moving-monitoring-components-to-different-nodes_{context}"]
 = Moving monitoring components to different nodes
 
-You can move any of the monitoring stack components to specific nodes.
+To specify the nodes in your cluster on which monitoring stack components will run, configure the `nodeSelector` constraint in the component's `ConfigMap` object to match labels assigned to the nodes.
+
+[NOTE]
+====
+You cannot add a node selector constraint directly to an existing scheduled pod.
+====
 
 .Prerequisites
 
 * *If you are configuring core {product-title} monitoring components*:
 ** You have access to the cluster as a user with the `cluster-admin` role.
 ** You have created the `cluster-monitoring-config` `ConfigMap` object.
 * *If you are configuring components that monitor user-defined projects*:
-** You have access to the cluster as a user with the `cluster-admin` role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
+** You have access to the cluster as a user with the `cluster-admin` role or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
 ** You have created the `user-workload-monitoring-config` `ConfigMap` object.
 * You have installed the OpenShift CLI (`oc`).
 
 .Procedure
 
+. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:
++
+[source,terminal]
+----
+$ oc label nodes <node-name> <node-label>
+----
 . Edit the `ConfigMap` object:
 ** *To move a component that monitors core {product-title} projects*:
+
 .. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
 +
 [source,terminal]
 ----
 $ oc -n openshift-monitoring edit configmap cluster-monitoring-config
 ----
 
-.. Specify the `nodeSelector` constraint for the component under `data/config.yaml`:
+.. Specify the node labels for the `nodeSelector` constraint for the component under `data/config.yaml`:
 +
 [source,yaml]
 ----
@@ -40,77 +52,32 @@ metadata:
   namespace: openshift-monitoring
 data:
   config.yaml: |
-    <component>:
+    <component>: <1>
       nodeSelector:
-        <node_key>: <node_value>
-        <node_key>: <node_value>
+        <node-label-1> <2>
+        <node-label-2> <3>
         <...>
 ----
-+
-Substitute `<component>` accordingly and substitute `<node_key>: <node_value>` with the map of key-value pairs that specifies a group of destination nodes. Often, only a single key-value pair is used.
-+
-The component can only run on nodes that have each of the specified key-value pairs as labels. The nodes can have additional labels as well.
-+
-[IMPORTANT]
-====
-Many of the monitoring components are deployed by using multiple pods across different nodes in the cluster to maintain high availability. When moving monitoring components to labeled nodes, ensure that enough matching nodes are available to maintain resilience for the component. If only one label is specified, ensure that enough nodes contain that label to distribute all of the pods for the component across separate nodes. Alternatively, you can specify multiple labels each relating to individual nodes.
-====
+<1> Substitute `<component>` with the appropriate monitoring stack component name.
+<2> Substitute `<node-label-1>` with the label you added to the node.
+<3> Optional: Specify additional labels.
+If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.
 +
 [NOTE]
 ====
-If monitoring components remain in a `Pending` state after configuring the `nodeSelector` constraint, check the pod logs for errors relating to taints and tolerations.
+If monitoring components remain in a `Pending` state after configuring the `nodeSelector` constraint, check the pod events for errors relating to taints and tolerations.
 ====
-+
-For example, to move monitoring components for core {product-title} projects to specific nodes that are labeled `nodename: controlplane1`, `nodename: worker1`, `nodename: worker2`, and `nodename: worker2`, use:
-+
-[source,yaml,subs=quotes]
-----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: cluster-monitoring-config
-  namespace: openshift-monitoring
-data:
-  config.yaml: |
-    prometheusOperator:
-      nodeSelector:
-        nodename: controlplane1
-    prometheusK8s:
-      nodeSelector:
-        nodename: worker1
-        nodename: worker2
-    alertmanagerMain:
-      nodeSelector:
-        nodename: worker1
-        nodename: worker2
-    kubeStateMetrics:
-      nodeSelector:
-        nodename: worker1
-    telemeterClient:
-      nodeSelector:
-        nodename: worker1
-    k8sPrometheusAdapter:
-      nodeSelector:
-        nodename: worker1
-        nodename: worker2
-    openshiftStateMetrics:
-      nodeSelector:
-        nodename: worker1
-    thanosQuerier:
-      nodeSelector:
-        nodename: worker1
-        nodename: worker2
-----
 
 ** *To move a component that monitors user-defined projects*:
+
 .. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project:
 +
 [source,terminal]
 ----
 $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
 ----
 
-.. Specify the `nodeSelector` constraint for the component under `data/config.yaml`:
+.. Specify the node labels for the `nodeSelector` constraint for the component under `data/config.yaml`:
 +
 [source,yaml]
 ----
@@ -121,52 +88,24 @@ metadata:
   namespace: openshift-user-workload-monitoring
 data:
   config.yaml: |
-    <component>:
+    <component>: <1>
       nodeSelector:
-        <node_key>: <node_value>
-        <node_key>: <node_value>
+        <node-label-1> <2>
+        <node-label-2> <3>
         <...>
 ----
-+
-Substitute `<component>` accordingly and substitute `<node_key>: <node_value>` with the map of key-value pairs that specifies the destination nodes. Often, only a single key-value pair is used.
-+
-The component can only run on nodes that have each of the specified key-value pairs as labels. The nodes can have additional labels as well.
-+
-[IMPORTANT]
-====
-Many of the monitoring components are deployed by using multiple pods across different nodes in the cluster to maintain high availability. When moving monitoring components to labeled nodes, ensure that enough matching nodes are available to maintain resilience for the component. If only one label is specified, ensure that enough nodes contain that label to distribute all of the pods for the component across separate nodes. Alternatively, you can specify multiple labels each relating to individual nodes.
-====
+<1> Substitute `<component>` with the appropriate monitoring stack component name.
+<2> Substitute `<node-label-1>` with the label you added to the node.
+<3> Optional: Specify additional labels.
+If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.
 +
 [NOTE]
 ====
-If monitoring components remain in a `Pending` state after configuring the `nodeSelector` constraint, check the pod logs for errors relating to taints and tolerations.
+If monitoring components remain in a `Pending` state after configuring the `nodeSelector` constraint, check the pod events for errors relating to taints and tolerations.
 ====
-+
-For example, to move monitoring components for user-defined projects to specific worker nodes labeled `nodename: worker1`, `nodename: worker2`, and `nodename: worker2`, use:
-+
-[source,yaml,subs=quotes]
-----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: user-workload-monitoring-config
-  namespace: openshift-user-workload-monitoring
-data:
-  config.yaml: |
-    prometheusOperator:
-      nodeSelector:
-        nodename: worker1
-    prometheus:
-      nodeSelector:
-        nodename: worker1
-        nodename: worker2
-    thanosRuler:
-      nodeSelector:
-        nodename: worker1
-        nodename: worker2
-----
 
-. Save the file to apply the changes. The components affected by the new configuration are moved to the new nodes automatically.
+. Save the file to apply the changes.
+The components specified in the new configuration are moved to the new nodes automatically.
 +
 [NOTE]
 ====
@@ -175,5 +114,6 @@ Configurations applied to the `user-workload-monitoring-config` `ConfigMap` obje
 +
 [WARNING]
 ====
-When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.
+When you save changes to a monitoring config map, the pods and other resources in the project might be redeployed.
+The running monitoring processes in that project might also restart.
 ====

modules/monitoring-using-node-selectors-to-move-monitoring-components.adoc

@@ -0,0 +1,28 @@
+// Module included in the following assemblies:
+//
+// * monitoring/configuring-the-monitoring-stack.adoc
+
+:_content-type: CONCEPT
+[id="using-node-selectors-to-move-monitoring-components_{context}"]
+= Using node selectors to move monitoring components
+
+By using the `nodeSelector` constraint with labeled nodes, you can move any of the monitoring stack components to specific nodes.
+By doing so, you can control the placement and distribution of the monitoring components across a cluster.
+
+By controlling placement and distribution of monitoring components, you can optimize system resource use, improve performance, and segregate workloads based on specific requirements or policies.
+
+[id="how-node-selectors-work-with-other-constraints_{context}"]
+== How node selectors work with other constraints
+
+
+If you move monitoring components by using node selector constraints, be aware that other constraints to control pod scheduling might exist for a cluster:
+
+* Topology spread constraints might be in place to control pod placement.
+* Hard anti-affinity rules are in place for Prometheus, Thanos Querier, Alertmanager, and other monitoring components to ensure that multiple pods for these components are always spread across different nodes and are therefore always highly available.
+
+When scheduling pods onto nodes, the pod scheduler tries to satisfy all existing constraints when determining pod placement.
+That is, all constraints compound when the pod scheduler determines which pods will be placed on which nodes.
+
+Therefore, if you configure a node selector constraint but existing constraints cannot all be satisfied, the pod scheduler cannot match all constraints and will not schedule a pod for placement onto a node.
+
+To maintain resilience and high availability for monitoring components, ensure that enough nodes are available and match all constraints when you configure a node selector constraint to move a component.

monitoring/configuring-the-monitoring-stack.adoc

@@ -46,16 +46,25 @@ include::modules/monitoring-configuring-the-monitoring-stack.adoc[leveloffset=+1
 include::modules/monitoring-configurable-monitoring-components.adoc[leveloffset=+1]
 
 // Moving monitoring components to different nodes
-include::modules/monitoring-moving-monitoring-components-to-different-nodes.adoc[leveloffset=+1]
+include::modules/monitoring-using-node-selectors-to-move-monitoring-components.adoc[leveloffset=+1]
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../nodes/nodes/nodes-nodes-working.adoc#nodes-nodes-working-updating_nodes-nodes-working[Understanding how to update labels on nodes]
+* xref:../nodes/scheduling/nodes-scheduler-node-selectors.adoc#nodes-scheduler-node-selectors[Placing pods on specific nodes using node selectors]
+* xref:../nodes/scheduling/nodes-scheduler-pod-affinity.adoc[Placing pods relative to other pods using affinity and anti-affinity rules]
+* xref:../nodes/scheduling/nodes-scheduler-pod-topology-spread-constraints.adoc[Controlling pod placement by using pod topology spread constraints]
+* xref:../monitoring/configuring-the-monitoring-stack.adoc#configuring_pod_topology_spread_constraintsfor_monitoring_configuring-the-monitoring-stack[Configuring pod topology spread constraints for monitoring]
+* link:https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector[Kubernetes documentation about node selectors]
+
+include::modules/monitoring-moving-monitoring-components-to-different-nodes.adoc[leveloffset=+2]
+
 
 [role="_additional-resources"]
 .Additional resources
 
 * See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps
 * xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]
-* xref:../nodes/nodes/nodes-nodes-working.adoc#nodes-nodes-working-updating_nodes-nodes-working[Understanding how to update labels on nodes]
-* xref:../nodes/scheduling/nodes-scheduler-node-selectors.adoc#nodes-scheduler-node-selectors[Placing pods on specific nodes using node selectors]
-* See the link:https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector[Kubernetes documentation] for details on the `nodeSelector` constraint
 
 // Assigning tolerations to monitoring components
 include::modules/monitoring-assigning-tolerations-to-monitoring-components.adoc[leveloffset=+1]
@@ -124,7 +133,7 @@ include::modules/monitoring-choosing-a-metrics-collection-profile.adoc[leveloffs
 .Additional resources
 
 * See xref:../monitoring/managing-metrics.adoc#viewing-a-list-of-available-metrics_managing-metrics[Viewing a list of available metrics] for steps to view a list of metrics being collected for a cluster.
-* See xref:../nodes/clusters/nodes-cluster-enabling-features.adoc[Enabling features using feature gates] for steps to enable Technology Preview features. 
+* See xref:../nodes/clusters/nodes-cluster-enabling-features.adoc[Enabling features using feature gates] for steps to enable Technology Preview features.
 
 // Managing scrape sample limits for user-defined projects
 include::modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc[leveloffset=+1]