Skip to content

Commit fb964b8

Browse files
bburt-rhbburt-rh
authored
Merge pull request #46476 from bburt-rh/RHDEVDOCS-3924-option-to-add-cluster-id-to-off-cluster-integrations
RHDEVDOCS-3924 - document option to add cluster id to off-cluster int…
2 parents 63ae59e + 2cf7383 commit fb964b8

File tree

3 files changed

+124
-1
lines changed

3 files changed

+124
-1
lines changed

‎modules/monitoring-adding-cluster-id-labels-to-metrics.adoc

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/configuring-the-monitoring-stack.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="adding-cluster-id-labels-to-metrics_{context}"]
7+
= Adding cluster ID labels to metrics
8+
9+
If you manage multiple {product-title} clusters and use the remote write feature to send metrics data from these clusters to an external storage location, you can add cluster ID labels to identify the metrics data coming from different clusters.
10+
You can then query these labels to identify the source cluster for a metric and distinguish that data from similar metrics data sent by other clusters.
11+
12+
This way, if you manage many clusters for multiple customers and send metrics data to a single centralized storage system, you can use cluster ID labels to query metrics for a particular cluster or customer.
13+
14+
Creating and using cluster ID labels involves three general steps:
15+
16+
* Configuring the write relabel settings for remote write storage.
17+
18+
* Adding cluster ID labels to the metrics.
19+
20+
* Querying these labels to identify the source cluster or customer for a metric.

‎modules/monitoring-creating-cluster-id-labels-for-metrics.adoc

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/configuring-the-monitoring-stack.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="creating-cluster-id-labels-for-metrics_{context}"]
7+
= Creating cluster ID labels for metrics
8+
9+
You can create cluster ID labels for metrics for default platform monitoring and for user workload monitoring.
10+
11+
For default platform monitoring, you add cluster ID labels for metrics in the `write_relabel` settings for remote write storage in the `cluster-monitoring-config` config map in the `openshift-monitoring` namespace.
12+
13+
For user workload monitoring, you edit the settings in the `user-workload-monitoring-config` config map in the `openshift-user-workload-monitoring` namespace.
14+
15+
.Prerequsites
16+
17+
* You have installed the OpenShift CLI (`oc`).
18+
* You have configured remote write storage.
19+
* *If you are configuring default platform monitoring components:*
20+
** You have access to the cluster as a user with the `cluster-admin` role.
21+
** You have created the `cluster-monitoring-config` `ConfigMap` object.
22+
* *If you are configuring components that monitor user-defined projects:*
23+
** You have access to the cluster as a user with the `cluster-admin` role or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
24+
** You have created the `user-workload-monitoring-config` `ConfigMap` object.
25+
26+
.Procedure
27+
28+
. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
29+
+
30+
[source,terminal]
31+
----
32+
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
33+
----
34+
+
35+
[NOTE]
36+
====
37+
If you configure cluster ID labels for metrics for the Prometheus instance that monitors user-defined projects, edit the `user-workload-monitoring-config` config map in the `openshift-user-workload-monitoring` namespace.
38+
Note that the Prometheus component is called `prometheus` in this config map and not `prometheusK8s`, which is the name used in the `cluster-monitoring-config` config map.
39+
====
40+
41+
. In the `writeRelabelConfigs:` section under `data/config.yaml/prometheusK8s/remoteWrite`, add cluster ID relabel configuration values:
42+
+
43+
[source,yaml]
44+
----
45+
apiVersion: v1
46+
kind: ConfigMap
47+
metadata:
48+
name: cluster-monitoring-config
49+
namespace: openshift-monitoring
50+
data:
51+
config.yaml: |
52+
prometheusK8s:
53+
remoteWrite:
54+
- url: "https://remote-write-endpoint.example.com"
55+
<endpoint_authentication_credentials>
56+
writeRelabelConfigs: <1>
57+
- <relabel_config> <2>
58+
----
59+
<1> Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.
60+
<2> Substitute the label configuration for the metrics sent to the remote write endpoint.
61+
+
62+
The following sample shows how to forward a metric with the cluster ID label `cluster_id` in default platform monitoring:
63+
+
64+
[source,yaml]
65+
----
66+
apiVersion: v1
67+
kind: ConfigMap
68+
metadata:
69+
name: cluster-monitoring-config
70+
namespace: openshift-monitoring
71+
data:
72+
config.yaml: |
73+
prometheusK8s:
74+
remoteWrite:
75+
- url: "https://remote-write-endpoint.example.com"
76+
writeRelabelConfigs:
77+
- sourceLabels:
78+
- __tmp_openshift_cluster_id__ <1>
79+
targetLabel: cluster_id <2>
80+
action: replace <3>
81+
----
82+
<1> The system initially applies a temporary cluster ID source label named `+++__tmp_openshift_cluster_id__+++`. This temporary label gets replaced by the cluster ID label name that you specify.
83+
<2> Specify the name of the cluster ID label for metrics sent to remote write storage.
84+
If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label.
85+
For the label name, do not use `+++__tmp_openshift_cluster_id__+++`. The final relabeling step removes labels that use this name.
86+
<3> The `replace` write relabel action replaces the temporary label with the target label for outgoing metrics.
87+
This action is the default and is applied if no action is specified.
88+
89+
. Save the file to apply the changes to the `ConfigMap` object.
90+
The pods affected by the updated configuration automatically restart.
91+
+
92+
[WARNING]
93+
====
94+
Saving changes to a monitoring `ConfigMap` object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.
95+
====

‎monitoring/configuring-the-monitoring-stack.adoc

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,15 @@ include::modules/monitoring-supported-remote-write-authentication-settings.adoc[
9797
* See xref:../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets-about_nodes-pods-secrets[Understanding secrets] for steps to create and configure `Secret` objects in {product-title}.
9898
* See the xref:../rest_api/monitoring_apis/prometheus-monitoring-coreos-com-v1.adoc#spec-remotewrite-2[Prometheus REST API reference for remote write] for information about additional optional fields.
9999

100+
// Configuring labels for outgoing metrics
101+
include::modules/monitoring-adding-cluster-id-labels-to-metrics.adoc[leveloffset=+1]
102+
include::modules/monitoring-creating-cluster-id-labels-for-metrics.adoc[leveloffset=+2]
103+
104+
[role="_additional-resources"]
105+
.Additional resources
106+
107+
* For details about write relabel configuration, see xref:../monitoring/configuring-the-monitoring-stack.adoc#configuring_remote_write_storage_configuring-the-monitoring-stack[Configuring remote write storage].
108+
100109
// Managing scrape sample limits for user-defined projects
101110
include::modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc[leveloffset=+1]
102111
include::modules/monitoring-setting-a-scrape-sample-limit-for-user-defined-projects.adoc[leveloffset=+2]
@@ -129,7 +138,6 @@ include::modules/monitoring-setting-query-log-file-for-prometheus.adoc[leveloffs
129138

130139
[role="_additional-resources"]
131140
.Additional resources
132-
133141
* See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps
134142
* See xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] for steps to enable user-defined monitoring.
135143

0 commit comments

Comments
 (0)