Merge pull request #49065 from bburt-rh/RHDEVDOCS-4036-CMO-config-map-option-for-body-size-limit-b

RHDEVDoCS-4036 - CMO config map option for body size limit for metrics scraping

Author: bburt-rh

modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc

@@ -0,0 +1,61 @@
+// Module included in the following assemblies:
+//
+// * monitoring/configuring-the-monitoring-stack.adoc
+
+:_content-type: PROCEDURE
+[id="setting-the-body-size-limit-for-metrics-scraping_{context}"]
+= Setting the body size limit for metrics scraping
+
+By default, no limit exists for the uncompressed body size for data returned from scraped metrics targets.
+You can set a body size limit to help avoid situations in which Prometheus consumes excessive amounts of memory when scraped targets return a response that contains a large amount of data.
+In addition, by setting a body size limit, you can reduce the impact that a malicious target might have on Prometheus and on the cluster as a whole.
+
+After you set a value for `enforcedBodySizeLimit`, the alert `PrometheusScrapeBodySizeLimitHit` fires when at least one Prometheus scrape target replies with a response body larger than the configured value.
+
+[NOTE]
+====
+If metrics data scraped from a target has an uncompressed body size exceeding the configured size limit, the scrape fails.
+Prometheus then considers this target to be down and sets its `up` metric value to `0`, which can trigger the `TargetDown` alert.
+====
+
+.Prerequisites
+
+* You have access to the cluster as a user with the `cluster-admin` role.
+* You have installed the OpenShift CLI (`oc`).
+
+.Procedure
+
+. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` namespace:
++
+[source,terminal]
+----
+$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
+----
+
+. Add a value for `enforcedBodySizeLimit` to `data/config.yaml/prometheusK8s` to limit the body size that can be accepted per target scrape:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-monitoring-config
+  namespace: openshift-monitoring
+data:
+  config.yaml: |-
+    prometheusK8s:
+      enforcedBodySizeLimit: 40MB <1>
+----
+<1> Specify the maximum body size for scraped metrics targets.
+This `enforcedBodySizeLimit` example limits the uncompressed size per target scrape to 40 megabytes.
+Valid numeric values use the Prometheus data size format: B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes).
+The default value is `0`, which specifies no limit.
+You can also set the value to `automatic` to calculate the limit automatically based on cluster capacity.
+
+. Save the file to apply the changes automatically.
++
+[WARNING]
+====
+When you save changes to a `cluster-monitoring-config` config map, the pods and other resources in the `openshift-monitoring` project might be redeployed. 
+The running monitoring processes in that project might also restart.
+====

monitoring/configuring-the-monitoring-stack.adoc

@@ -68,6 +68,14 @@ include::modules/monitoring-assigning-tolerations-to-monitoring-components.adoc[
 * See the xref:../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations[{product-title} documentation] on taints and tolerations
 * See the link:https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/[Kubernetes documentation] on taints and tolerations
 
+// Setting the body size limit for metrics scraping
+include::modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+
+* link:https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config[Prometheus scrape configuration documentation]
+
 // Configuring persistent storage
 include::modules/monitoring-configuring-persistent-storage.adoc[leveloffset=+1]
 include::modules/monitoring-configuring-a-local-persistent-volume-claim.adoc[leveloffset=+2]
@@ -168,4 +176,4 @@ include::modules/monitoring-disabling-the-local-alertmanager.adoc[leveloffset=+1
 == Next steps
 
 * xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]
-* Learn about xref:../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[remote health reporting] and, if necessary, opt out of it.
+* Learn about xref:../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[remote health reporting] and, if necessary, opt out of it.