You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cortex Alertmanager notification setup follow mostly the syntax of Prometheus Alertmanager since it is based on the same codebase. The following is a description on how to load the configuration setup so that Alertmanager can use for notification when an alert event happened.
10
+
Cortex Alertmanager notification setup follows mostly the syntax of Prometheus Alertmanager since it is based on the same codebase. The following is a description on how to load the configuration setup so that Alertmanager can use it for notification when an alert event happens.
11
11
12
12
### Configuring the Cortex Alertmanager storage backend
13
13
14
-
With the introduction of Cortex 1.8 the storage backend config option shifted to the new pattern [#3888](https://github.com/cortexproject/cortex/pull/3888). You can find the new configuration [here](../configuration/config-file-reference.md#alertmanager_storage_config)
14
+
With the introduction of Cortex 1.8, the storage backend config option shifted to the new pattern [#3888](https://github.com/cortexproject/cortex/pull/3888). You can find the new configuration [here](../configuration/config-file-reference.md#alertmanager_storage_config)
15
15
16
16
Note that when using `-alertmanager.sharding-enabled=true`, the following storage backends are not supported: `local`, `configdb`.
17
17
18
-
When using the new configuration pattern it is important that any of the old configuration pattern flags are unset (`-alertmanager.storage`), as well as `-<prefix>.configs.url`. This is because the old pattern still takes precedence over the new one. The old configuration pattern (`-alertmanager.storage`) is marked as deprecated and will be removed by Cortex version 1.11. However this change doesn't apply to `-alertmanager.storage.path` and `-alertmanager.storage.retention`.
18
+
When using the new configuration pattern, it is important that any of the old configuration pattern flags are unset (`-alertmanager.storage`), as well as `-<prefix>.configs.url`. This is because the old pattern still takes precedence over the new one. The old configuration pattern (`-alertmanager.storage`) is marked as deprecated and will be removed by Cortex version 1.11. However, this change doesn't apply to `-alertmanager.storage.path` and `-alertmanager.storage.retention`.
19
19
20
20
### Cortex Alertmanager configuration
21
21
22
22
Cortex Alertmanager can be uploaded via Cortex [Set Alertmanager configuration API](../api/_index.md#set-alertmanager-configuration) or using [Cortex Tools](https://github.com/cortexproject/cortex-tools).
23
23
24
-
Follow the instruction at the `cortextool` link above to download or update to the latest version of the tool.
24
+
Follow the instructions at the `cortextool` link above to download or update to the latest version of the tool.
25
25
26
26
To obtain the full help of how to use `cortextool` for all commands and flags, use
27
27
`cortextool --help-long`.
28
28
29
29
The following example shows the steps to upload the configuration to Cortex `Alertmanager` using `cortextool`.
30
30
31
-
#### 1. Create the Alertmanager configuration `yml` file.
31
+
#### 1. Create the Alertmanager configuration YAML file.
32
32
33
-
The following is `amconfig.yml`, an example of a configuration for Cortex `Alertmanager` to send notification via email:
33
+
The following is `amconfig.yml`, an example of a configuration for Cortex `Alertmanager` to send notifications via email:
34
34
35
35
```
36
36
global:
@@ -50,7 +50,7 @@ receivers:
50
50
- to: 'someone@localhost'
51
51
```
52
52
53
-
[Example on how to setup Slack](https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/#:~:text=To%20set%20up%20alerting%20in,to%20receive%20notifications%20from%20Alertmanager.) to support receiving Alertmanager notification.
53
+
[Example on how to set up Slack](https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/#:~:text=To%20set%20up%20alerting%20in,to%20receive%20notifications%20from%20Alertmanager.) to support receiving Alertmanager notifications.
4. How compressible the time-series data are. If a metric stays at
30
+
the same value constantly, then Cortex can compress it very well, so
31
+
12 hours of data sampled every 15 seconds would be around 2KB. On
32
+
the other hand, if the value jumps around a lot, it might take 10KB.
33
+
There are not currently any tools available to analyse this.
34
+
5. How long you want to retain data for, e.g. 1 month or 2 years.
35
35
36
36
Other parameters which can become important if you have particularly
37
37
high values:
38
38
39
-
6. Number of different series under one metric name.
40
-
7. Number of labels per series.
41
-
8. Rate and complexity of queries.
39
+
6. Number of different series under one metric name.
40
+
7. Number of labels per series.
41
+
8. Rate and complexity of queries.
42
42
43
43
Now, some rules of thumb:
44
44
45
-
1. Each million series in an ingester takes 15GB of RAM. Total number
46
-
of series in ingesters is number of active series times the
47
-
replication factor. This is with the default of 12-hour chunks - RAM
48
-
required will reduce if you set `-ingester.max-chunk-age` lower
49
-
(trading off more back-end database IO).
50
-
There are some additional considerations for planning for ingester memory usage.
51
-
1. Memory increases during writeahead log (WAL) replay, [See Prometheus issue #6934](https://github.com/prometheus/prometheus/issues/6934#issuecomment-726039115). If you do not have enough memory for WAL replay, the ingester will not be able to restart successfully without intervention.
52
-
2. Memory temporarily increases during resharding since timeseries are temporarily on both the new and old ingesters. This means you should scale up the number of ingesters before memory utilization is too high, otherwise you will not have the headroom to account for the temporary increase.
53
-
2. Each million series (including churn) consumes 15GB of chunk
54
-
storage and 4GB of index, per day (so multiply by the retention
55
-
period).
56
-
3. The distributors CPU utilization depends on the specific Cortex cluster
57
-
setup, while they don't need much RAM. Typically, distributors are capable
58
-
to process between 20,000 and 100,000 samples/sec with 1 CPU core. It's also
59
-
highly recommended to configure Prometheus `max_samples_per_send` to 1,000
60
-
samples, in order to reduce the distributors CPU utilization given the same
61
-
total samples/sec throughput.
45
+
1. Each million series in an ingester takes 15GB of RAM. The total number
46
+
of series in ingesters is the number of active series times the
47
+
replication factor. This is with the default of 12-hour chunks - RAM
48
+
required will reduce if you set `-ingester.max-chunk-age` lower
49
+
(trading off more back-end database I/O).
50
+
There are some additional considerations for planning for ingester memory usage.
51
+
1. Memory increases during write-ahead log (WAL) replay, [See Prometheus issue #6934](https://github.com/prometheus/prometheus/issues/6934#issuecomment-726039115). If you do not have enough memory for WAL replay, the ingester will not be able to restart successfully without intervention.
52
+
2. Memory temporarily increases during resharding since timeseries are temporarily on both the new and old ingesters. This means you should scale up the number of ingesters before memory utilization is too high, otherwise you will not have the headroom to account for the temporary increase.
53
+
2. Each million series (including churn) consumes 15GB of chunk
54
+
storage and 4GB of index, per day (so multiply by the retention
55
+
period).
56
+
3. The distributors’ CPU utilization depends on the specific Cortex cluster
57
+
setup, while they don't need much RAM. Typically, distributors are capable
58
+
of processing between 20,000 and 100,000 samples/sec with 1 CPU core. It's also
59
+
highly recommended to configure Prometheus `max_samples_per_send` to 1,000
60
+
samples, in order to reduce the distributors’ CPU utilization given the same
61
+
total samples/sec throughput.
62
62
63
63
If you turn on compression between distributors and ingesters (for
64
-
example to save on inter-zone bandwidth charges at AWS/GCP) they will use
65
-
significantly more CPU (approx 100% more for distributor and 50% more
64
+
example, to save on inter-zone bandwidth charges at AWS/GCP), they will use
65
+
significantly more CPU (approx. 100% more for distributor and 50% more
Copy file name to clipboardExpand all lines: docs/guides/encryption-at-rest.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,8 +48,8 @@ The alertmanager S3 server-side encryption can be configured similarly to the bl
48
48
49
49
### Per-tenant config overrides
50
50
51
-
The S3 client used by the blocks storage, ruler and alertmanager supports S3 SSE config overrides on a per-tenant basis, using the [runtime configuration file](../configuration/arguments.md#runtime-configuration-file).
52
-
The following settings can ben overridden for each tenant:
51
+
The S3 client used by the blocks storage, ruler, and alertmanager supports S3 SSE config overrides on a per-tenant basis, using the [runtime configuration file](../configuration/arguments.md#runtime-configuration-file).
52
+
The following settings can be overridden for each tenant:
53
53
54
54
- **`s3_sse_type`**<br />
55
55
S3 server-side encryption type. It must be set to enable the SSE config override for a given tenant.
@@ -60,4 +60,5 @@ The following settings can ben overridden for each tenant:
60
60
61
61
## Other storages
62
62
63
-
Other storage backends may support encryption at rest configuring it directly at the storage level.
63
+
Other storage backends may support encryption at rest, configuring it directly at the storage level.
Copy file name to clipboardExpand all lines: docs/guides/encryption-at-rest.template
+4-3Lines changed: 4 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -30,8 +30,8 @@ The alertmanager S3 server-side encryption can be configured similarly to the bl
30
30
31
31
### Per-tenant config overrides
32
32
33
-
The S3 client used by the blocks storage, ruler and alertmanager supports S3 SSE config overrides on a per-tenant basis, using the [runtime configuration file](../configuration/arguments.md#runtime-configuration-file).
34
-
The following settings can ben overridden for each tenant:
33
+
The S3 client used by the blocks storage, ruler, and alertmanager supports S3 SSE config overrides on a per-tenant basis, using the [runtime configuration file](../configuration/arguments.md#runtime-configuration-file).
34
+
The following settings can be overridden for each tenant:
35
35
36
36
- **`s3_sse_type`**<br />
37
37
S3 server-side encryption type. It must be set to enable the SSE config override for a given tenant.
@@ -42,4 +42,5 @@ The following settings can ben overridden for each tenant:
42
42
43
43
## Other storages
44
44
45
-
Other storage backends may support encryption at rest configuring it directly at the storage level.
45
+
Other storage backends may support encryption at rest, configuring it directly at the storage level.
Copy file name to clipboardExpand all lines: docs/guides/glossary.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ A single chunk contains timestamp-value pairs for several series.
21
21
22
22
Churn is the frequency at which series become idle.
23
23
24
-
A series become idle once it's not exported anymore by the monitored targets. Typically, series become idle when the monitored target itself disappear (eg. the process or node gets terminated).
24
+
A series becomes idle once it's not exported anymore by the monitored targets. Typically, series become idle when the monitored target itself disappears (eg. the process or node gets terminated).
25
25
26
26
### Flushing
27
27
@@ -35,7 +35,7 @@ For more information, please refer to the guide "[Config for sending HA Pairs da
35
35
36
36
### Hash ring
37
37
38
-
The hash ring is a distributed data structure used by Cortex for sharding, replication and service discovery. The hash ring data structure gets shared across Cortex replicas via gossip or a key-value store.
38
+
The hash ring is a distributed data structure used by Cortex for sharding, replication, and service discovery. The hash ring data structure gets shared across Cortex replicas via gossip or a key-value store.
39
39
40
40
For more information, please refer to the [Architecture](../architecture.md#the-hash-ring) documentation.
41
41
@@ -94,6 +94,6 @@ _See [Tenant](#tenant)._
94
94
95
95
### WAL
96
96
97
-
The Write-Ahead Log (WAL) is an appendonly log stored on disk used by ingesters to recover their in-memory state after the process gets restarted, either after a clear shutdown or an abruptly termination.
97
+
The Write-Ahead Log (WAL) is an append-only log stored on disk used by ingesters to recover their in-memory state after the process gets restarted, either after a clear shutdown or an abrupt termination.
98
98
99
99
For more information, please refer to [Ingesters with WAL](../blocks-storage/_index.md#the-write-path).
0 commit comments