Skip to content

Commit 3924245

Browse files
abrennan89abrennan89
authored
Merge pull request #36729 from bburt-rh/MON-1799-document-configuring-external-alertmanagers
MON-1799 - Add content for configuring external Alertmanagers
2 parents e98ebcc + 1a67483 commit 3924245

File tree

2 files changed

+166
-0
lines changed

2 files changed

+166
-0
lines changed

‎modules/monitoring-configuring-external-alertmanagers.adoc

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/configuring-the-monitoring-stack.adoc
4+
5+
[id="monitoring-configuring-external-alertmanagers_{context}"]
6+
= Configuring external alertmanager instances
7+
8+
The {product-title} monitoring stack includes a local Alertmanager instance that routes alerts from Prometheus.
9+
You can add external Alertmanager instances by configuring the `cluster-monitoring-config` config map in either the `openshift-monitoring` project or the `user-workload-monitoring-config` project.
10+
11+
If you add the same external Alertmanager configuration for multiple clusters and disable the local instance for each cluster, you can then manage alert routing for multiple clusters by using a single external Alertmanager instance.
12+
13+
.Prerequisites
14+
15+
* You have installed the OpenShift CLI (`oc`).
16+
* *If you are configuring core {product-title} monitoring components in the `openshift-monitoring` project*:
17+
** You have access to the cluster as a user with the `cluster-admin` role.
18+
** You have created the `cluster-monitoring-config` config map.
19+
* *If you are configuring components that monitor user-defined projects*:
20+
** You have access to the cluster as a user with the `cluster-admin` role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
21+
** You have created the `user-workload-monitoring-config` config map.
22+
23+
.Procedure
24+
25+
. Edit the `ConfigMap` object.
26+
** *To configure additional Alertmanagers for routing alerts from core {product-title} projects*:
27+
.. Edit the `cluster-monitoring-config` config map in the `openshift-monitoring` project:
28+
+
29+
[source,terminal]
30+
----
31+
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
32+
----
33+
34+
.. Add an `additionalAlertmanagerConfigs:` section under `data/config.yaml/prometheusK8s`.
35+
36+
.. Add the configuration details for additional Alertmanagers in this section:
37+
+
38+
[source,yaml]
39+
----
40+
apiVersion: v1
41+
kind: ConfigMap
42+
metadata:
43+
name: cluster-monitoring-config
44+
namespace: openshift-monitoring
45+
data:
46+
config.yaml: |
47+
prometheusK8s:
48+
additionalAlertmanagerConfigs:
49+
- <alertmanager_specification>
50+
----
51+
+
52+
For `<alertmanager_specification>`, substitute authentication and other configuration details for additional Alertmanager instances.
53+
Currently supported authentication methods are bearer token (`bearerToken`) and client TLS (`tlsConfig`).
54+
The following sample config map configures an additional Alertmanager using a bearer token with client TLS authentication:
55+
+
56+
[source,yaml]
57+
----
58+
apiVersion: v1
59+
kind: ConfigMap
60+
metadata:
61+
name: cluster-monitoring-config
62+
namespace: openshift-monitoring
63+
data:
64+
config.yaml: |
65+
prometheusK8s:
66+
additionalAlertmanagerConfigs:
67+
- scheme: https
68+
pathPrefix: /
69+
timeout: "30s"
70+
apiVersion: v1
71+
bearerToken:
72+
name: alertmanager-bearer-token
73+
key: token
74+
tlsConfig:
75+
key:
76+
name: alertmanager-tls
77+
key: tls.key
78+
cert:
79+
name: alertmanager-tls
80+
key: tls.crt
81+
ca:
82+
name: alertmanager-tls
83+
key: tls.ca
84+
staticConfigs:
85+
- external-alertmanager1-remote.com
86+
- external-alertmanager1-remote2.com
87+
----
88+
89+
** *To configure additional Alertmanager instances for routing alerts from user-defined projects*:
90+
91+
.. Edit the `user-workload-monitoring-config` config map in the `openshift-user-workload-monitoring` project:
92+
+
93+
[source,terminal]
94+
----
95+
$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
96+
----
97+
98+
.. Add a `<component>/additionalAlertmanagerConfigs:` section under `data/config.yaml/`.
99+
100+
.. Add the configuration details for additional Alertmanagers in this section:
101+
+
102+
[source,yaml]
103+
----
104+
apiVersion: v1
105+
kind: ConfigMap
106+
metadata:
107+
name: user-workload-monitoring-config
108+
namespace: openshift-user-workload-monitoring
109+
data:
110+
config.yaml: |
111+
<component>:
112+
additionalAlertmanagerConfigs:
113+
- <alertmanager_specification>
114+
----
115+
+
116+
For `<component>`, substitute one of two supported external Alertmanager components: `prometheus` or `thanosRuler`.
117+
+
118+
For `<alertmanager_specification>`, substitute authentication and other configuration details for additional Alertmanager instances.
119+
Currently supported authentication methods are bearer token (`bearerToken`) and client TLS (`tlsConfig`).
120+
The following sample config map configures an additional Alertmanager using Thanos Ruler with a bearer token and client TLS authentication:
121+
+
122+
[source,yaml]
123+
----
124+
apiVersion: v1
125+
kind: ConfigMap
126+
metadata:
127+
name: user-workload-monitoring-config
128+
namespace: openshift-user-workload-monitoring
129+
data:
130+
config.yaml: |
131+
thanosRuler:
132+
additionalAlertmanagerConfigs:
133+
- scheme: https
134+
pathPrefix: /
135+
timeout: "30s"
136+
apiVersion: v1
137+
bearerToken:
138+
name: alertmanager-bearer-token
139+
key: token
140+
tlsConfig:
141+
key:
142+
name: alertmanager-tls
143+
key: tls.key
144+
cert:
145+
name: alertmanager-tls
146+
key: tls.crt
147+
ca:
148+
name: alertmanager-tls
149+
key: tls.ca
150+
staticConfigs:
151+
- external-alertmanager1-remote.com
152+
- external-alertmanager1-remote2.com
153+
----
154+
+
155+
[NOTE]
156+
====
157+
Configurations applied to the `user-workload-monitoring-config` `ConfigMap` object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.
158+
====
159+
160+
. Save the file to apply the changes to the `ConfigMap` object.
161+
The new component placement configuration is applied automatically.
162+
163+

‎monitoring/configuring-the-monitoring-stack.adoc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,9 @@ include::modules/monitoring-creating-scrape-sample-alerts.adoc[leveloffset=+2]
125125
* xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]
126126
* See xref:../monitoring/troubleshooting-monitoring-issues.html#determining-why-prometheus-is-consuming-disk-space_troubleshooting-monitoring-issues[Determining why Prometheus is consuming a lot of disk space] for steps to query which metrics have the highest number of scrape samples
127127

128+
//Configuring external alertmanagers
129+
include::modules/monitoring-configuring-external-alertmanagers.adoc[leveloffset=1]
130+
128131
//Attaching additional labels to your time series and alerts
129132
include::modules/monitoring-attaching-additional-labels-to-your-time-series-and-alerts.adoc[leveloffset=+1]
130133

0 commit comments

Comments
 (0)