Merge pull request #29273 from mburke5678/BZ-1825417

The containerruntimecontroller doesn't roll back to CR-1 if we delete CR-2

Author: mburke5678

modules/create-a-containerruntimeconfig-crd.adoc

@@ -5,72 +5,152 @@
 [id="create-a-containerruntimeconfig_{context}"]
 = Creating a `ContainerRuntimeConfig` CR to edit CRI-O parameters
 
-The `ContainerRuntimeConfig` custom resource definition (CRD) provides a structured way of changing settings associated with the {product-title} CRI-O runtime. Using a `ContainerRuntimeConfig` custom resource (CR), you select the configuration values you want and the MCO handles rebuilding the `crio.conf` and `storage.conf` configuration files.
+You can change some of the settings associated with the {product-title} CRI-O runtime for the nodes associated with a specific machine config pool (MCP). Using a `ContainerRuntimeConfig` custom resource (CR), you set the configuration values and add a label to match the MCP. The MCO then rebuilds the `crio.conf` and `storage.conf` configuration files on the associated nodes with the updated values.
 
-Parameters you can set in a `ContainerRuntimeConfig` CR include:
+[NOTE]
+====
+To revert the changes implemented by using a `ContainerRuntimeConfig` CR, you must delete the CR. Removing the label from the machine config pool does not revert the changes.
+==== 
 
-* **PIDs limit**: Sets the maximum number of processes allowed in a container. By default, the limit is set to 1024 (`pids_limit = 1024`).
-* **Log level**: Sets the level of verbosity for log messages. The default is `info` (`log_level = info`). Other options include `fatal`, `panic`, `error`, `warn`, `debug`, and `trace`.
-* **Overlay size**: Sets the maxim size of a container image. The default is 10 GB.
-* **Maximum log size**: Sets the maximum size allowed for the container log file. The default maximum log size is unlimited (`log_size_max = -1`). If it is set to a positive number, it must be at least 8192 to not be smaller than  the `conmon` read buffer. Conmon is a program that
+You can modify the following settings by using a `ContainerRuntimeConfig` CR:
+
+* **PIDs limit**: The `pidsLimit` parameter sets the CRI-O `pids_limit` parameter, which is maximum number of processes allowed in a container. The default is 1024 (`pids_limit = 1024`).
+* **Log level**: The `logLevel` parameter sets the CRI-O `log_level` parameter, which is the level of verbosity for log messages. The default is `info` (`log_level = info`). Other options include `fatal`, `panic`, `error`, `warn`, `debug`, and `trace`.
+* **Overlay size**: The `overlaySize` parameter sets the CRI-O Overlay storage driver `size` parameter, which is the maximum size of a container image. The default is 10 GB (size = "10G").
+* **Maximum log size**: The `logSizeMax` parameter sets the CRI-O `log_size_max` parameter, which is the maximum size allowed for the container log file. The default is unlimited (`log_size_max = -1`). If set to a positive number, it must be at least 8192 to not be smaller than  the ConMon read buffer. ConMon is a program that
 monitors communications between a container manager (such as Podman or CRI-O) and the OCI runtime (such as runc or crun) for a single container.
 
-The following procedure describes how to change CRI-O settings using the `ContainerRuntimeConfig` CR.
+You can create multiple `ContainerRuntimeConfig` CRs, as needed, with a limit of ten. For the first `ContainerRuntimeConfig` CR, the MCO creates a machine config appended with `containerruntime`. With each subsequent CR, the controller creates a new `containerruntime` machine config with a numeric suffix. For example, if you have a `containerruntime` machine config with a `-2` suffix, the next `containerruntime` machine config is appended with `-3`.
+
+You can also edit an existing `ContainerRuntimeConfig` CRs to edit existing settings or add new settings instead of creating a new CR. It is recommended to create a new CR to modify a different machine config pool, rather than add additional labels to an existing CR.
+
+If you want to delete the machine configs, you should delete them in reverse order to avoid exceeding the limit. For example, you should delete the `containerruntime-3` machine config before deleting the `containerruntime-2` machine config. 
+
+[NOTE]
+====
+If you have a machine config with a `containerruntime-9` suffix, the next machine config is appended with `containerruntime-10` and will fail as it exceeds the limit, even if there are fewer than 10 `containerruntime` machine configs.
+====
+
+.Example showing multiple `ContainerRuntimeConfig` CRs
+[source,terminal]
+----
+$ oc get ctrcfg
+----
+
+.Example output
+[source, terminal]
+----
+NAME         AGE
+ctr-pid      24m
+ctr-overlay  15m
+ctr-level    5m45s
+----
+
+.Example showing multiple `containerruntime` machine configs
+[source,terminal]
+----
+$ oc get mc | grep container
+----
+
+.Example output
+[source, terminal]
+----
+...
+01-master-container-runtime                        b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             57m
+...
+01-worker-container-runtime                        b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             57m
+...
+99-worker-generated-containerruntime               b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             26m
+99-worker-generated-containerruntime-1             b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             17m
+99-worker-generated-containerruntime-2             b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             7m26s
+...
+----
+
+The following example raises the `pids_limit` to 2048, sets the `log_level` to `debug`, sets the overlay size to 8 GB, and sets the `log_size_max` to unlimited:
 
+.Example `ContainerRuntimeConfig` CR
+[source,yaml]
+----
+apiVersion: machineconfiguration.openshift.io/v1
+kind: ContainerRuntimeConfig
+metadata:
+ name: overlay-size
+spec:
+ machineConfigPoolSelector:
+   matchLabels:
+     pools.operator.machineconfiguration.openshift.io/worker: '' <1>
+ containerRuntimeConfig:
+   pidsLimit: 2048 <2>
+   logLevel: debug <3>
+   overlaySize: 8G <4>
+   logSizeMax: "-1" <5>
+----
+<1> Specifies the machine config pool label.
+<2> Optional: Specifies the level of verbosity for log messages.
+<3> Optional: Specifies the maximum size allowed for the container log file. If set to a positive number, it must be at least `8192`.
+<4> Optional: Specifies the maximum size of a container image. 
+<5> Optional: Specifies the maximum number of processes allowed in a container.
+	
 .Procedure
 
-. To raise the `pidsLimit` to 2048, set the `logLevel` to `debug`, and set the `overlaySize` to 8 GB, create a CR file (for example, `overlay-size.yaml`) that contains that setting:
+To change CRI-O settings using the `ContainerRuntimeConfig` CR:
+
+. Create a YAML file for the `ContainerRuntimeConfig` CR:
 +
 [source,yaml]
 ----
-$ cat << EOF > /tmp/overlay-size.yaml
 apiVersion: machineconfiguration.openshift.io/v1
 kind: ContainerRuntimeConfig
 metadata:
  name: overlay-size
 spec:
  machineConfigPoolSelector:
    matchLabels:
-     custom-crio: overlay-size
- containerRuntimeConfig:
+     pools.operator.machineconfiguration.openshift.io/worker: '' <1>
+ containerRuntimeConfig: <2>
    pidsLimit: 2048
    logLevel: debug
    overlaySize: 8G
-EOF
+   logSizeMax: "-1"
 ----
+<1> Specify a label for the machine config pool that you want you want to modify.
+<2> Set the parameters as needed.  
 
-. To apply the `ContainerRuntimeConfig` object settings, run:
+. Create the `ContainerRuntimeConfig` CR:
 +
 [source,terminal]
 ----
-$ oc create -f /tmp/overlay-size
+$ oc create -f <file_name>.yaml
 ----
 
-. To verify that the settings wer applied, run:
+. Verify that the CR is created:
 +
 [source,terminal]
 ----
 $ oc get ContainerRuntimeConfig
+----
++
+.Example output
+[source,terminal]
+----
 NAME           AGE
 overlay-size   3m19s
-
 ----
 
-. To edit a pool of machines, such as `worker`, run the following command to open a machine config pool:
+. Check that a new `containerruntime` machine config is created:
 +
 [source,terminal]
 ----
-$ oc edit machineconfigpool worker
+$ oc get machineconfigs | grep containerrun
 ----
-
-. Check that a new `containerruntime` object has appeared under the `machineconfigs`:
 +
+.Example output
 [source,terminal]
 ----
-$ oc get machineconfigs | grep containerrun
 99-worker-generated-containerruntime   2c9371fbb673b97a6fe8b1c52691999ed3a1bfc2  3.1.0  31s
 ----
-. Monitor the machine config pool as the changes are rolled into the machines until all are shown as ready:
+
+. Monitor the machine config pool until all are shown as ready:
 +
 [source,terminal]
 ----
@@ -85,26 +165,41 @@ NAME    CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMAC
 worker  rendered-worker-169  False    True      False     3             1                  1                    0                     9h
 ----
 
-. Open an `oc debug` session to a worker node and run `chroot /host`.
+. Verify that the settings were applied in CRI-O:
+
+.. Open an `oc debug` session to a node in the machine config pool and run `chroot /host`.
++
+[source, terminal]
+----
+$ oc debug node/<node_name>
+----
++
+[source, terminal]
+----
+sh-4.4# chroot /host
+----
 
-. Verify the changes by running:
+.. Verify the changes in the `crio.conf` file:
 +
 [source,terminal]
 ----
-$ crio config | egrep 'log_level|pids_limit'
+sh-4.4# crio config | egrep 'log_level|pids_limit|log_size_max'
 ----
 +
 .Example output
 +
 [source,terminal]
 ----
 pids_limit = 2048
+log_size_max = -1
 log_level = "debug"
 ----
+
+.. Verify the changes in the `storage.conf`file:
 +
 [source,terminal]
 ----
-$ head -n 7 /etc/containers/storage.conf
+sh-4.4# head -n 7 /etc/containers/storage.conf
 ----
 +
 .Example output