Merge pull request #51466 from mburke5678/update-infra-nodes

mburke5678 · web-flow · commit d15f23b1bdca · 2022-11-09T19:51:26.000-05:00
Convert from KCS (L): Infrastructure Nodes in OpenShift 4
diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml
@@ -2071,6 +2071,8 @@ Topics:
 #    File: nodes-nodes-problem-detector
   - Name: Machine Config Daemon metrics
     File: nodes-nodes-machine-config-daemon-metrics
+  - Name: Creating infrastructure nodes
+    File: nodes-nodes-creating-infrastructure-nodes
 - Name: Working with containers
   Dir: containers
   Topics:
diff --git a/machine_management/creating-infrastructure-machinesets.adoc b/machine_management/creating-infrastructure-machinesets.adoc
@@ -11,10 +11,14 @@ include::modules/machine-user-provisioned-limitations.adoc[leveloffset=+1]
 
 You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.
 
+In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and {SMProductName} deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. This configuration requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.
+
 include::modules/infrastructure-components.adoc[leveloffset=+1]
 
 For information about infrastructure nodes and which components can run on infrastructure nodes, see the "Red Hat OpenShift control plane and infrastructure nodes" section in the link:https://www.redhat.com/en/resources/openshift-subscription-sizing-guide[OpenShift sizing and subscription guide for enterprise Kubernetes] document.
 
+To create an infrastructure node, you can xref:../machine_management/creating-infrastructure-machinesets.adoc#machineset-creating_creating-infrastructure-machinesets[use a machine set], xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-an-infra-node_creating-infrastructure-machinesets[label the node], or xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infra-machines_creating-infrastructure-machinesets[use a machine config pool].    
+
 [id="creating-infrastructure-machinesets-production"]
 == Creating infrastructure machine sets for production environments
 
@@ -100,7 +104,26 @@ include::modules/binding-infra-node-workloads-using-taints-tolerations.adoc[leve
 [id="moving-resources-to-infrastructure-machinesets"]
 == Moving resources to infrastructure machine sets
 
-Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created.
+Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created by adding the infrastructure node selector, as shown: 
+
+[source,yaml]
+----
+spec:
+  nodePlacement: <1>
+    nodeSelector:
+      matchLabels:
+        node-role.kubernetes.io/infra: ""
+    tolerations:
+    - effect: NoSchedule
+      key: node-role.kubernetes.io/infra
+      value: reserved
+    - effect: NoExecute
+      key: node-role.kubernetes.io/infra
+      value: reserved
+----
+<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node.  If you added a taint to the infrasructure node, also add a matching toleration.
+
+Applying a specific node selector to all infrastructure components causes {product-title} to xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets[schedule those workloads on nodes with that label].
 
 include::modules/infrastructure-moving-router.adoc[leveloffset=+2]
 
diff --git a/modules/infrastructure-components.adoc b/modules/infrastructure-components.adoc
@@ -2,6 +2,7 @@
 //
 // * machine_management/creating-infrastructure-machinesets.adoc
 // * post_installation_configuration/cluster-tasks.adoc
+// * nodes-nodes-creating-infrastructure-nodes.adoc
 
 [id="infrastructure-components_{context}"]
 = {product-title} infrastructure components
diff --git a/modules/infrastructure-moving-logging.adoc b/modules/infrastructure-moving-logging.adoc
@@ -42,6 +42,13 @@ spec:
       nodeCount: 3
       nodeSelector: <1>
         node-role.kubernetes.io/infra: ''
+      tolerations:
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/infra
+        value: reserved
+      - effect: NoExecute
+        key: node-role.kubernetes.io/infra
+        value: reserved
       redundancyPolicy: SingleRedundancy
       resources:
         limits:
@@ -57,6 +64,13 @@ spec:
     kibana:
       nodeSelector: <1>
         node-role.kubernetes.io/infra: ''
+      tolerations:
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/infra
+        value: reserved
+      - effect: NoExecute
+        key: node-role.kubernetes.io/infra
+        value: reserved
       proxy:
         resources: null
       replicas: 1
@@ -65,7 +79,7 @@ spec:
 
 ...
 ----
-<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node.
+<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node.  If you added a taint to the infrasructure node, also add a matching toleration.
 
 .Verification
 
diff --git a/modules/infrastructure-moving-monitoring.adoc b/modules/infrastructure-moving-monitoring.adoc
@@ -7,12 +7,16 @@
 = Moving the monitoring solution
 
 The monitoring stack includes multiple components, including Prometheus, Thanos Querier, and Alertmanager.
-The Cluster Monitoring Operator manages this stack.
-To redeploy the monitoring stack to infrastructure nodes, you can create and apply a custom config map.
+The Cluster Monitoring Operator manages this stack. To redeploy the monitoring stack to infrastructure nodes, you can create and apply a custom config map.
  
 .Procedure
 
-. Save the following `ConfigMap` definition as the `cluster-monitoring-configmap.yaml` file:
+. Edit the `cluster-monitoring-config` config map and change the `nodeSelector` to use the `infra` label:
++
+[source,terminal]
+----
+$ oc edit configmap cluster-monitoring-config -n openshift-monitoring
+----
 +
 [source,yaml]
 ----
@@ -24,39 +28,87 @@ metadata:
 data:
   config.yaml: |+
     alertmanagerMain:
-      nodeSelector:
+      nodeSelector: <1>
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     prometheusK8s:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     prometheusOperator:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     k8sPrometheusAdapter:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     kubeStateMetrics:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     telemeterClient:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     openshiftStateMetrics:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
     thanosQuerier:
       nodeSelector:
         node-role.kubernetes.io/infra: ""
+      tolerations:
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoSchedule
+      - key: node-role.kubernetes.io/infra
+        value: reserved
+        effect: NoExecute
 ----
-+
-Running this config map forces the components of the monitoring stack to redeploy to infrastructure nodes.
-
-. Apply the new config map:
-+
-[source,terminal]
-----
-$ oc create -f cluster-monitoring-configmap.yaml
-----
+<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node.  If you added a taint to the infrasructure node, also add a matching toleration.
 
 . Watch the monitoring pods move to the new machines:
 +
diff --git a/modules/infrastructure-moving-registry.adoc b/modules/infrastructure-moving-registry.adoc
@@ -58,8 +58,6 @@ status:
 ----
 $ oc edit configs.imageregistry.operator.openshift.io/cluster
 ----
-
-. Modify the `spec` section of the object to resemble the following YAML:
 +
 [source,yaml]
 ----
@@ -74,9 +72,17 @@ spec:
         weight: 100
   logLevel: Normal
   managementState: Managed
-  nodeSelector:
+  nodeSelector: <1>
     node-role.kubernetes.io/infra: ""
+  tolerations:
+  - effect: NoSchedule
+    key: node-role.kubernetes.io/infra
+    value: reserved
+  - effect: NoExecute
+    key: node-role.kubernetes.io/infra
+    value: reserved
 ----
+<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node.  If you added a taint to the infrasructure node, also add a matching toleration.
 
 . Verify the registry pod has been moved to the infrastructure node.
 +
diff --git a/modules/infrastructure-moving-router.adoc b/modules/infrastructure-moving-router.adoc
@@ -57,16 +57,22 @@ status:
 $ oc edit ingresscontroller default -n openshift-ingress-operator
 ----
 +
-Add the `nodeSelector` stanza that references the `infra` label to the `spec` section, as shown:
-+
 [source,yaml]
 ----
   spec:
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/infra: ""
-----
+    tolerations:
+    - effect: NoSchedule
+      key: node-role.kubernetes.io/infra
+      value: reserved
+    - effect: NoExecute
+      key: node-role.kubernetes.io/infra
+      value: reserved
+----
+<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node.  If you added a taint to the infrasructure node, also add a matching toleration.
 
 . Confirm that the router pod is running on the `infra` node.
 .. View the list of router pods and note the node name of the running pod:
diff --git a/modules/machine-user-provisioned-limitations.adoc b/modules/machine-user-provisioned-limitations.adoc
@@ -11,6 +11,7 @@
 // * machine_management/deploying-machine-health-checks.adoc
 // * machine_management/manually-scaling-machinesets.adoc
 // * post_installation_configuration/node-tasks.adoc
+// * nodes-nodes-creating-infrastructure-nodes.adoc
 
 [IMPORTANT]
 ====
diff --git a/nodes/nodes/nodes-nodes-creating-infrastructure-nodes.adoc b/nodes/nodes/nodes-nodes-creating-infrastructure-nodes.adoc
@@ -0,0 +1,28 @@
+:_content-type: ASSEMBLY
+[id="nodes-nodes-creating-infrastructure-nodes"]
+= Creating infrastructure nodes
+include::_attributes/common-attributes.adoc[]
+:context: creating-infrastructure-nodes
+
+toc::[]
+
+include::modules/machine-user-provisioned-limitations.adoc[leveloffset=+1]
+
+
+You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.
+
+In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and {SMProductName} deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. This configuration requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.
+
+include::modules/infrastructure-components.adoc[leveloffset=+1]
+
+For information about infrastructure nodes and which components can run on infrastructure nodes, see the "Red Hat OpenShift control plane and infrastructure nodes" section in the link:https://www.redhat.com/en/resources/openshift-subscription-sizing-guide[OpenShift sizing and subscription guide for enterprise Kubernetes] document.
+
+To create an infrastructure node, you can xref:../../machine_management/creating-infrastructure-machinesets.adoc#machineset-creating_creating-infrastructure-machinesets[use a machine set], xref:../../nodes/nodes/nodes-nodes-creating-infrastructure-nodes.adoc#creating-an-infra-node_creating-infrastructure-nodes[label the node], or xref:../../machine_management/creating-infrastructure-machinesets.adoc#creating-infra-machines_creating-infrastructure-machinesets[use a machine config pool].  
+
+include::modules/creating-an-infra-node.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets[Moving resources to infrastructure machine sets]
+
diff --git a/post_installation_configuration/cluster-tasks.adoc b/post_installation_configuration/cluster-tasks.adoc
@@ -521,7 +521,11 @@ In a production deployment, it is recommended that you deploy at least three com
 
 For information on infrastructure nodes and which components can run on infrastructure nodes, see xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infrastructure-machinesets[Creating infrastructure machine sets].
 
-For sample machine sets that you can use with these procedures, see xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infrastructure-machinesets-clouds[Creating infrastructure machine sets for different clouds].
+To create an infrastructure node, you can xref:../post_installation_configuration/cluster-tasks.adoc#machineset-creating_post-install-cluster-tasks[use a machine set], post_installation_configuration/cluster-tasks.adoc#creating-an-infra-node_post-install-cluster-tasks[assign a label to the nodes], or xref:../post_installation_configuration/cluster-tasks.adoc#creating-infra-machines_post-install-cluster-tasks[use a machine config pool].  
+
+For sample machine sets that you can use with these procedures, see xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infrastructure-machinesets-clouds[Creating machine sets for different clouds].
+
+Applying a specific node selector to all infrastructure components causes {product-title} to xref:../post_installation_configuration/cluster-tasks.adoc#moving-resources-to-infrastructure-machinesets[schedule those workloads on nodes with that label].
 
 include::modules/machineset-creating.adoc[leveloffset=+2]
 
@@ -642,4 +646,3 @@ include::modules/installation-images-samples-disconnected-mirroring-assist.adoc[
 include::modules/installation-restricted-network-samples.adoc[leveloffset=+2]
 
 include::modules/installation-preparing-restricted-cluster-to-gather-support-data.adoc[leveloffset=+2]
-
