Skip to content

Commit 8829bf6

Browse files
AkashSirimannaAkash Sirimannanschad
authored
Add guide for configuration am and ruler with mounted config maps (#313)
* Add guide for configuration am and ruler with mounted config maps Signed-off-by: Akash Sirimanna <[email protected]> * improve doc Signed-off-by: nschad <[email protected]> Co-authored-by: Akash Sirimanna <[email protected]> Co-authored-by: nschad <[email protected]>
1 parent 423b1c8 commit 8829bf6

File tree

3 files changed

+186
-1
lines changed

3 files changed

+186
-1
lines changed

‎CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
* config.querier.query_store_after: 12h -> 0s (default)
99
* [ENHANCEMENT] Fix the indentation of memcached guide #309
1010
* [ENHANCEMENT] Added api endpoints for Grafana 8 unified alerting #291
11+
* [ENHANCEMENT] Add guide on how to configure alertmanager/ruler with configmap #313
1112

1213
## 1.2.0 / 2021-12-29
1314

‎docs/assets/scripts/copyCode.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ const template = `
66

77

88
document.addEventListener("DOMContentLoaded", function () {
9-
const codeBlocks = document.querySelectorAll('.highlighter-rouge');
9+
const codeBlocks = document.querySelectorAll('div.highlighter-rouge');
1010
codeBlocks.forEach((codeBlock, index) => {
1111
const code = codeBlocks[index].innerText;
1212
const wrapper = document.createElement("div");

‎docs/guides/configure_ruler_and_alertmanager_with_configmap.markdown

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
---
2+
layout: page
3+
title: Configure Ruler and Alertmanager with Configmap
4+
parent: Guides
5+
has_children: false
6+
has_toc: false
7+
---
8+
# Configure Ruler and Alertmanager with Configmap
9+
{: .no_toc }
10+
11+
## Table of contents
12+
{: .no_toc .text-delta }
13+
14+
1. TOC
15+
{:toc}
16+
17+
---
18+
# Preface
19+
20+
Cortex's Ruler and Alertmanager can be setup to use ConfigMaps to provide themselves with a configuration that can be dynamically updated.
21+
We simply put ConfigMaps into the same namespace where cortex is running, and they are automatically detected and mounted as files to the Ruler and/or AlertManager containers.
22+
The containers are also constantly polling for changes to the ConfigMaps. If any changes are found, the configuration file on the container will be updated.
23+
24+
It should be noted that this guide is assuming that multi-tenancy is enabled :
25+
```yaml
26+
auth_enabled: true
27+
```
28+
# Setup
29+
30+
In order to get this working, we must do a couple things.
31+
32+
## Create Ruler ConfigMap
33+
34+
To provide the Ruler with the *rules* to alert on, we must create a ConfigMap with the desired ruleset. How you maintain and deploy the ConfigMap is completely in your control, the only requirement is that the ConfigMap be on the same namespace as the Ruler.
35+
36+
**IMPORTANT** things to note:
37+
38+
- This ruleset is ONLY for tenantfoo (We will explicitly map it to tenantfoo later)
39+
- The section under `tenantfoo.yaml` is validated and interpreted exactly as a [Prometheus rules configuration](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) would be. How you configure this is completely up to you.
40+
41+
```yaml
42+
kind: ConfigMap
43+
metadata:
44+
name: tenantfoo-ruler-config
45+
namespace: cortex
46+
apiVersion: v1
47+
data:
48+
tenantfoo.yaml: |-
49+
groups:
50+
- name: <group name>
51+
rules:
52+
- alert: <alert name>
53+
expr: <promql expression>
54+
for: <duration>
55+
labels:
56+
severity: <severity>
57+
annotations:
58+
description: <description>
59+
<remainder of alerts...>
60+
```
61+
62+
## Create Alertmanager ConfigMap
63+
64+
To provide the Alertmanager with the information needed to *route* notifications to stakeholders, we must create a ConfigMap with the desired routing rules. How you maintain and deploy the ConfigMap is completely in your control, the only requirement is that the ConfigMap be on the same namespace as the Alertmanager.
65+
66+
**IMPORTANT** things to note:
67+
68+
- These routing rules are ONLY for tenantfoo. The Alertmanager takes the key under `data` (excluding `.yaml`) and uses it as the tenant name for that specific config.
69+
- The section under `tenantfoo.yaml` is validated and interpreted exactly as a [Prometheus alerting rules configuration](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) would be. How you configure this is completely up to you.
70+
71+
```yaml
72+
kind: ConfigMap
73+
metadata:
74+
name: tenantfoo-alertmanager-config
75+
namespace: cortex
76+
apiVersion: v1
77+
data:
78+
tenantfoo.yaml: |
79+
global:
80+
resolve_timeout: <duration>
81+
http_config: {}
82+
smtp_from: ExamplePerson <[email protected]>
83+
smtp_hello: example.com
84+
smtp_smarthost: example-smtp.example.com
85+
route:
86+
receiver: team-X-mails
87+
group_by: ['alertname']
88+
group_wait: <duration>
89+
group_interval: <duration>
90+
repeat_interval: <duration>
91+
receivers:
92+
- name: 'team-X-mails'
93+
email_configs:
94+
- to: '[email protected]'
95+
```
96+
97+
## Helm Chart Configuration
98+
99+
Now that the ConfigMaps are created, we can tinker the helm chart to achieve our goal.
100+
101+
### Configuration for Ruler and Alertmanager Config
102+
Add this to the `values.yaml` under the `config` section as shown.
103+
104+
```yaml
105+
config:
106+
ruler:
107+
enable_api: true
108+
# already set-up temporary emptyDir volume. Cortex will parse rules from /data/rules
109+
# and copy them here for prometheus rule evaluation
110+
rule_path: /rules
111+
112+
ruler_storage:
113+
backend: "local"
114+
local:
115+
# where your mounted configmap data will be persistently stored
116+
directory: /data/rules
117+
118+
alertmanager:
119+
enable_api: true
120+
# temporary volume if not using stateful alertmanagers
121+
data_dir: /data
122+
123+
alertmanager_storage:
124+
backend: "local"
125+
local:
126+
# temporary volume if not using stateful alertmanagers
127+
path: /data
128+
```
129+
130+
### Configuration for Ruler and Alertmanager
131+
Add this to the `values.yaml`.
132+
Note that when we mount `tenantfoo-ruler-config`, we mount it under a folder which should be named after the tenant it corresponds to. In this situation, since we want `tenantfoo-ruler-config` to be the alerting rules for `tenantfoo`, we mount it under `/data/rules/tenantfoo`.
133+
Also note that for `tenantfoo-alertmanager-config`, we similarily mount it under a folder specific to that tenant. Although it will not use the folder name as the tenant name (it instead uses the key name under `data` in the ConfigMap), it simplifies things if you do so.
134+
135+
```yaml
136+
ruler:
137+
extraVolumes:
138+
- configMap:
139+
defaultMode: 420
140+
name: tenantfoo-ruler-config
141+
name: tenantfoo-ruler-config
142+
extraVolumeMounts:
143+
- name: tenantfoo-ruler-config
144+
mountPath: /data/rules/tenantfoo
145+
alertmanager:
146+
extraVolumes:
147+
- configMap:
148+
defaultMode: 420
149+
name: tenantfoo-alertmanager-config
150+
name: tenantfoo-alertmanager-config
151+
extraVolumeMounts:
152+
- name: tenantfoo-alertmanager-config
153+
mountPath: /data/tenantfoo
154+
```
155+
156+
# Result
157+
158+
You can now install the helm chart.
159+
160+
You may use the section below to verify the configuration went according to plan.
161+
162+
### Verify Alertmanager
163+
164+
1. Port-forward the alertmanager pod
165+
2. Run `curl -X GET -H "X-Scope-OrgID: <name of tenant>" localhost:8080/api/v1/alerts`
166+
3. Verify the correct configuration shows up
167+
168+
### Verify Ruler
169+
170+
1. Port-forward the ruler pod
171+
2. Run `curl -X GET -H "X-Scope-OrgID: <name of tenant>" localhost:8080/api/v1/rules`
172+
3. Verify the correct configuration shows up
173+
174+
# Moving forward
175+
176+
You can also follow this guide if you don't have multi-tenancy enabled. However, in this case, you will not get alerting/routing rules specific to each tenant.
177+
178+
Defining fine-grained alerting/routing rules per tenant is when you really get to eat the fruits of your labour when deploying multi-tenant alertmanager/ruler. For example, lets say you currently only have routing rules for tenantfoo. If you wanted to add specific routing rules for another tenant, tenantB, you will do the following:
179+
180+
1. Create another set of ConfigMaps for the new tenants, being sure to adhere to the naming policies stated in this guide
181+
182+
2. Add entries under `extraVolumes` and `extraVolumeMounts` for the Ruler and Alertmanager. All while being sure to adhere to the naming policies stated in this guide
183+
184+
3. [Verify the new tenant is configured](#result)

0 commit comments

Comments
 (0)