Skip to content

Commit ef5cf8f

Browse files
abrennan89abrennan89
authored
Merge pull request #46357 from bburt-rh/RHDEVDOCS-3918-alertmanager-config-for-user-defined-monitoring
RHDEVDOCS-3918 - Document Alertmanager for user workload monitoring
2 parents c21e5b5 + bab16c9 commit ef5cf8f

14 files changed

+195
-124
lines changed

‎modules/monitoring-applying-a-custom-configuration-to-alertmanager-for-user-defined-alert-routing.adoc

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/managing-alerts.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="applying-a-custom-configuration-to-alertmanager-for-user-defined-alert-routing_{context}"]
7+
= Applying a custom configuration to Alertmanager for user-defined alert routing
8+
9+
If you have enabled a separate instance of Alertmanager dedicated to user-defined alert routing, you can overwrite the configuration for this instance of Alertmanager by editing the `alertmanager-user-workload` secret in the `openshift-user-workload-monitoring` namespace.
10+
11+
.Prerequisites
12+
13+
* You have access to the cluster as a user with the `cluster-admin` role.
14+
15+
.Procedure
16+
17+
. Print the currently active Alertmanager configuration into the file `alertmanager.yaml`:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml
22+
----
23+
+
24+
. Edit the configuration in `alertmanager.yaml`:
25+
+
26+
[source,yaml]
27+
----
28+
route:
29+
receiver: Default
30+
group_by:
31+
- name: Default
32+
routes:
33+
- matchers:
34+
- "service = prometheus-example-monitor" <1>
35+
receiver: <receiver> <2>
36+
receivers:
37+
- name: Default
38+
- name: <receiver>
39+
# <receiver_configuration>
40+
----
41+
<1> Specifies which alerts match the route. This example shows all alerts that have the `service="prometheus-example-monitor"` label.
42+
<2> Specifies the receiver to use for the alerts group.
43+
+
44+
. Apply the new configuration in the file:
45+
+
46+
[source,terminal]
47+
----
48+
$ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml | oc -n openshift-user-workload-monitoring replace secret --filename=-
49+
----

‎modules/monitoring-applying-custom-alertmanager-configuration.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
[id="applying-custom-alertmanager-configuration_{context}"]
77
= Applying a custom Alertmanager configuration
88

9-
You can overwrite the default Alertmanager configuration by editing the `alertmanager-main` secret inside the `openshift-monitoring` project.
9+
You can overwrite the default Alertmanager configuration by editing the `alertmanager-main` secret in the `openshift-monitoring` namespace for the platform instance of Alertmanager.
1010

1111
.Prerequisites
1212

‎modules/monitoring-components-for-monitoring-user-defined-projects.adoc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@
2323
|Thanos Ruler
2424
|The Thanos Ruler is a rule evaluation engine for Prometheus that is deployed as a separate process. In {product-title} {product-version}, Thanos Ruler provides rule and alerting evaluation for the monitoring of user-defined projects.
2525

26+
|Alertmanager
27+
|The Alertmanager service handles alerts received from Prometheus and Thanos Ruler. Alertmanager is also responsible for sending user-defined alerts to external notification systems. Deploying this service is optional.
28+
2629
|===
2730

2831
[NOTE]

‎modules/monitoring-configurable-monitoring-components.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ This table shows the monitoring components you can configure and the keys used t
1313
|Component |cluster-monitoring-config config map key |user-workload-monitoring-config config map key
1414
|Prometheus Operator |`prometheusOperator` |`prometheusOperator`
1515
|Prometheus |`prometheusK8s` |`prometheus`
16-
|Alertmanager |`alertmanagerMain` |
16+
|Alertmanager |`alertmanagerMain` | `alertmanager`
1717
|kube-state-metrics |`kubeStateMetrics` |
1818
|openshift-state-metrics |`openshiftStateMetrics` |
1919
|Telemeter Client |`telemeterClient` |

‎modules/monitoring-creating-alert-routing-for-user-defined-projects.adoc

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,6 @@
55
:_content-type: PROCEDURE
66
[id="creating-alert-routing-for-user-defined-projects_{context}"]
77
= Creating alert routing for user-defined projects
8-
9-
[IMPORTANT]
10-
====
11-
[subs="attributes+"]
12-
Alert routing for user-defined projects is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
13-
14-
For more information about the support scope of Red Hat Technology Preview features, see link:https://access.redhat.com/support/offerings/techpreview[https://access.redhat.com/support/offerings/techpreview].
15-
====
168

179
[role="_abstract"]
1810
If you are a non-administrator user who has been given the `alert-routing-edit` role, you can create or edit alert routing for user-defined projects.
@@ -32,7 +24,7 @@ If you are a non-administrator user who has been given the `alert-routing-edit`
3224
+
3325
[source,yaml]
3426
----
35-
apiVersion: monitoring.coreos.com/v1alpha1
27+
apiVersion: monitoring.coreos.com/v1beta1
3628
kind: AlertmanagerConfig
3729
metadata:
3830
name: example-routing

‎modules/monitoring-disabling-alert-routing-for-user-defined-projects.adoc

Lines changed: 0 additions & 50 deletions
This file was deleted.

‎modules/monitoring-enabling-a-separate-alertmanager-instance-for-user-defined-alert-routing.adoc

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/enabling-alert-routing-for-user-defined-projects.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="enabling-a-separate-alertmanager-instance-for-user-defined-alert-routing_{context}"]
7+
= Enabling a separate Alertmanager instance for user-defined alert routing
8+
9+
In some clusters, you might want to deploy a dedicated Alertmanager instance for user-defined projects, which can help reduce the load on the default platform Alertmanager instance and can better separate user-defined alerts from default platform alerts.
10+
In these cases, you can optionally enable a separate instance of Alertmanager to send alerts only for user-defined projects.
11+
12+
.Prerequisites
13+
14+
* You have access to the cluster as a user with the `cluster-admin` role.
15+
* You have enabled monitoring for user-defined projects in the `cluster-monitoring-config` config map for the `openshift-monitoring` namespace.
16+
* You have installed the OpenShift CLI (`oc`).
17+
18+
.Procedure
19+
20+
. Edit the `user-workload-monitoring-config` `ConfigMap` object:
21+
+
22+
[source,terminal]
23+
----
24+
$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
25+
----
26+
+
27+
. Add `enabled: true` and `enableAlertmanagerConfig: true` in the `alertmanager` section under `data/config.yaml`:
28+
+
29+
[source,yaml]
30+
----
31+
apiVersion: v1
32+
kind: ConfigMap
33+
metadata:
34+
name: user-workload-monitoring-config
35+
namespace: openshift-user-workload-monitoring
36+
data:
37+
config.yaml: |
38+
alertmanager:
39+
enabled: true <1>
40+
enableAlertmanagerConfig: true <2>
41+
----
42+
<1> Set the `enabled` value to `true` to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to `false` or omit the key entirely to disable the Alertmanager for user-defined projects.
43+
If you set this value to `false` or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.
44+
<2> Set the `enableAlertmanagerConfig` value to `true` to enable users to define their own alert routing configurations with `AlertmanagerConfig` objects.
45+
+
46+
. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.
47+
48+
.Verification
49+
50+
* Verify that the `user-workload` Alertmanager instance has started:
51+
+
52+
[source,terminal]
53+
----
54+
# oc -n openshift-user-workload-monitoring get alertmanager
55+
----
56+
+
57+
.Example output
58+
+
59+
[source,terminal]
60+
----
61+
NAME VERSION REPLICAS AGE
62+
user-workload 0.24.0 2 100s
63+
----

‎modules/monitoring-enabling-alert-routing-for-user-defined-projects.adoc

Lines changed: 0 additions & 44 deletions
This file was deleted.

‎modules/monitoring-enabling-the-platform-alertmanager-instance-for-user-defined-alert-routing.adoc

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/enabling-alert-routing-for-user-defined-projects.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="enabling-the-platform-alertmanager-instance-for-user-defined-alert-routing_{context}"]
7+
= Enabling the platform Alertmanager instance for user-defined alert routing
8+
9+
You can allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.
10+
11+
.Prerequisites
12+
13+
* You have access to the cluster as a user with the `cluster-admin` role.
14+
* You have installed the OpenShift CLI (`oc`).
15+
16+
.Procedure
17+
18+
. Edit the `cluster-monitoring-config` `ConfigMap` object:
19+
+
20+
[source,terminal]
21+
----
22+
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
23+
----
24+
+
25+
. Add `enableUserAlertmanagerConfig: true` in the `alertmanagerMain` section under `data/config.yaml`:
26+
+
27+
[source,yaml]
28+
----
29+
apiVersion: v1
30+
kind: ConfigMap
31+
metadata:
32+
name: cluster-monitoring-config
33+
namespace: openshift-monitoring
34+
data:
35+
config.yaml: |
36+
alertmanagerMain:
37+
enableUserAlertmanagerConfig: true <1>
38+
----
39+
<1> Set the `enableUserAlertmanagerConfig` value to `true` to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.
40+
+
41+
. Save the file to apply the changes.

‎modules/monitoring-support-considerations.adoc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,16 @@ The following modifications are explicitly not supported:
1313
+
1414
[NOTE]
1515
====
16-
The Alertmanager configuration is deployed as a secret resource in the `openshift-monitoring` project. To configure additional routes for Alertmanager, you need to decode, modify, and then encode that secret. This procedure is a supported exception to the preceding statement.
16+
The Alertmanager configuration is deployed as a secret resource in the `openshift-monitoring` namespace.
17+
If you have enabled a separate Alertmanager instance for user-defined alert routing, an Alertmanager configuration is also deployed as a secret resource in the `openshift-user-workload-monitoring` namespace.
18+
To configure additional routes for any instance of Alertmanager, you need to decode, modify, and then encode that secret.
19+
This procedure is a supported exception to the preceding statement.
1720
====
1821
+
1922
* *Modifying resources of the stack.* The {product-title} monitoring stack ensures its resources are always in the state it expects them to be. If they are modified, the stack will reset them.
2023
* *Deploying user-defined workloads to `openshift-&#42;`, and `kube-&#42;` projects.* These projects are reserved for Red Hat provided components and they should not be used for user-defined workloads.
2124
* *Installing custom Prometheus instances on {product-title}.*
2225
* *Enabling symptom based monitoring by using the `Probe` custom resource definition (CRD) in Prometheus Operator.*
23-
* *Modifying Alertmanager configurations by using the `AlertmanagerConfig` CRD in Prometheus Operator.*
2426
2527
[NOTE]
2628
====

0 commit comments

Comments
 (0)