Merge pull request #68228 from abrennan89/OBSDOCS-479-2

OBSDOCS-479: Add flow control / rate limit docs

Author: abrennan89

_topic_maps/_topic_map.yml

@@ -2608,6 +2608,11 @@ Topics:
     File: custom-logging-alerts
   - Name: Troubleshooting logging alerts
     File: troubleshooting-logging-alerts
+- Name: Performance and reliability tuning
+  Dir: performance_reliability
+  Topics:
+  - Name: Flow control mechanisms
+    File: logging-flow-control-mechanisms
 - Name: Troubleshooting Logging
   Dir: troubleshooting
   Distros: openshift-enterprise,openshift-origin

_topic_maps/_topic_map_osd.yml

@@ -1096,6 +1096,11 @@ Topics:
     File: custom-logging-alerts
   - Name: Troubleshooting logging alerts
     File: troubleshooting-logging-alerts
+- Name: Performance and reliability tuning
+  Dir: performance_reliability
+  Topics:
+  - Name: Flow control mechanisms
+    File: logging-flow-control-mechanisms
 - Name: Troubleshooting Logging
   Dir: troubleshooting
   Topics:

_topic_maps/_topic_map_rosa.yml

@@ -1305,6 +1305,11 @@ Topics:
     File: custom-logging-alerts
   - Name: Troubleshooting logging alerts
     File: troubleshooting-logging-alerts
+- Name: Performance and reliability tuning
+  Dir: performance_reliability
+  Topics:
+  - Name: Flow control mechanisms
+    File: logging-flow-control-mechanisms
 - Name: Troubleshooting Logging
   Dir: troubleshooting
   Topics:

logging/performance_reliability/_attributes

@@ -0,0 +1 @@
+../../_attributes/

logging/performance_reliability/images

@@ -0,0 +1 @@
+../../images/

logging/performance_reliability/logging-flow-control-mechanisms.adoc

@@ -0,0 +1,44 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+include::_attributes/attributes-openshift-dedicated.adoc[]
+[id="logging-flow-control-mechanisms"]
+= Flow control mechanisms
+:context: logging-flow-control-mechanisms
+
+toc::[]
+
+If logs are produced faster than they can be collected, it can be difficult to predict or control the volume of logs being sent to an output.
+Not being able to predict or control the volume of logs being sent to an output can result in logs being lost. If there is a system outage and log buffers are accumulated without user control, this can also cause long recovery times and high latency when the connection is restored.
+
+As an administrator, you can limit logging rates by configuring flow control mechanisms for your {logging}.
+
+[id="logging-configuring-flow-control-benefits"]
+== Benefits of flow control mechanisms
+
+* The cost and volume of logging can be predicted more accurately in advance.
+* Noisy containers cannot produce unbounded log traffic that drowns out other containers.
+* Ignoring low-value logs reduces the load on the logging infrastructure.
+* High-value logs can be preferred over low-value logs by assigning higher rate limits.
+
+[id="logging-configuring-flow-control-about-rate-limits"]
+== Configuring rate limits
+
+Rate limits are configured per collector, which means that the maximum rate of log collection is the number of collector instances multiplied by the rate limit.
+
+Because logs are collected from each node's file system, a collector is deployed on each cluster node. For example, in a 3-node cluster, with a maximum rate limit of 10 records per second per collector, the maximum rate of log collection is 30 records per second.
+
+Because the exact byte size of a record as written to an output can vary due to transformations, different encodings, or other factors, rate limits are set in number of records instead of bytes.
+
+You can configure rate limits in the `ClusterLogForwarder` custom resource (CR) in two ways:
+
+Output rate limit:: Limit the rate of outbound logs to selected outputs, for example, to match the network or storage capacity of an output. The output rate limit controls the aggregated per-output rate.
+
+Input rate limit:: Limit the per-container rate of log collection for selected containers.
+
+include::modules/logging-set-output-rate-limit.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* xref:../../logging/log_collection_forwarding/logging-output-types.adoc#logging-output-types[Log output types]
+
+include::modules/logging-set-input-rate-limit.adoc[leveloffset=+1]

logging/performance_reliability/modules

@@ -0,0 +1 @@
+../../modules/

logging/performance_reliability/snippets

@@ -0,0 +1 @@
+../../snippets/

modules/logging-create-clf.adoc

@@ -19,7 +19,7 @@ You need administrator permissions for the namespace where you create the `Clust
 .ClusterLogForwarder resource example
 [source,yaml]
 ----
-apiVersion: "logging.openshift.io/v1"
+apiVersion: logging.openshift.io/v1
 kind: ClusterLogForwarder
 metadata:
   name: <log_forwarder_name> <1>

modules/logging-set-input-rate-limit.adoc

@@ -0,0 +1,76 @@
+// Module included in the following assemblies:
+//
+// * logging/performance_reliability/logging-flow-control-mechanisms.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="logging-set-input-rate-limit_{context}"]
+= Configuring log forwarder input rate limits
+
+You can limit the rate of incoming logs that are collected by configuring the `ClusterLogForwarder` custom resource (CR). You can set input limits on a per-container or per-namespace basis.
+
+.Prerequisites
+
+* You have installed the {clo}.
+* You have administrator permissions.
+
+.Procedure
+
+. Add a `maxRecordsPerSecond` limit value to the `ClusterLogForwarder` CR for a specified input.
++
+The following examples show how to configure input rate limits for different scenarios:
++
+.Example `ClusterLogForwarder` CR that sets a per-container limit for containers with certain labels
+[source,yaml]
+----
+apiVersion: logging.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+# ...
+spec:
+# ...
+  inputs:
+    - name: <input_name> <1>
+      application:
+        selector:
+          matchLabels: { example: label } <2>
+      limitPerContainer:
+          maxRecordsPerSecond: 0 <3>
+# ...
+----
+<1> The input name.
+<2> A list of labels. If these labels match labels that are applied to a pod, the per-container limit specified in the `maxRecordsPerSecond` field is applied to those containers.
+<3> Configures the rate limit. Setting the `maxRecordsPerSecond` field to `0` means that no logs are collected for the container. Setting the `maxRecordsPerSecond` field to some other value means that a maximum of that number of records per second are collected for the container.
++
+.Example `ClusterLogForwarder` CR that sets a per-container limit for containers in selected namespaces
+[source,yaml]
+----
+apiVersion: logging.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+# ...
+spec:
+# ...
+  inputs:
+    - name: <input_name> <1>
+      application:
+        namespaces: [ example-ns-1, example-ns-2 ] <2>
+      limitPerContainer:
+        maxRecordsPerSecond: 10 <3>
+    - name: <input_name>
+      application:
+        namespaces: [ test ]
+      limitPerContainer:
+          maxRecordsPerSecond: 1000
+# ...
+----
+<1> The input name.
+<2> A list of namespaces. The per-container limit specified in the `maxRecordsPerSecond` field is applied to all containers in the namespaces listed.
+<3> Configures the rate limit. Setting the `maxRecordsPerSecond` field to `10` means that a maximum of 10 records per second are collected for each container in the namespaces listed.
+
+. Apply the `ClusterLogForwarder` CR:
++
+.Example command
+[source,terminal]
+----
+$ oc apply -f <filename>.yaml
+----