Skip to content

Commit 6448f4a

Browse files
author
Tyler Reid
committed
Update runbook to reflect it's a cli flag
1 parent 9c4da69 commit 6448f4a

File tree

2 files changed

+8
-15
lines changed

2 files changed

+8
-15
lines changed

cortex-mixin/alerts/alerts.libsonnet

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -356,7 +356,7 @@
356356
name: 'cortex_distributor_inflight_push_request_alert',
357357
rules: [
358358
{
359-
alert: 'CortexDistributorReachingInflightPushRequestLimits',
359+
alert: 'CortexDistributorReachingInflightPushRequestLimit',
360360
expr: |||
361361
(
362362
(cortex_distributor_inflight_push_requests / ignoring(limit) cortex_distributor_instance_limits{limit="max_inflight_push_requests"})

cortex-mixin/docs/playbooks.md

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -108,29 +108,22 @@ How to **fix**:
108108
1. Ensure shuffle-sharding is enabled in the Cortex cluster
109109
1. Assuming shuffle-sharding is enabled, scaling up ingesters will lower the number of tenants per ingester. However, the effect of this change will be visible only after `-blocks-storage.tsdb.close-idle-tsdb-timeout` period so you may have to temporarily increase the limit
110110
111-
### CortexDistributorReachingInflightPushRequestLimits
111+
### CortexDistributorReachingInflightPushRequestLimit
112112
113113
This alert fires when the `cortex_distributor_inflight_push_requests` per distributor instance limit is enabled and the actual number of inflight push requests is approaching the set limit. Once the limit is reached, push requests to the distributor will fail (5xx) for new requests, while existing inflight push requests will continue to succeed.
114114
115115
In case of **emergency**:
116-
- If the actual number of inflight push requests is very close to or already at the set limit, then you can increase the limit via runtime config to gain some time
117-
- Increasing the limit will increase the distributor' memory utilization. Please monitor the distributors' memory utilization via the `Cortex / Writes Resources` dashboard
116+
- If the actual number of inflight push requests is very close to or already at the set limit, then you can increase the limit via CLI flag or config to gain some time
117+
- Increasing the limit will increase the the number of inflight push requests which will increase distributors' memory utilization. Please monitor the distributors' memory utilization via the `Cortex / Writes Resources` dashboard
118118
119119
How the limit is **configured**:
120-
- The limit can be configured either on CLI (`-distributor.instance-limits.max-inflight-push-requests`) or in the runtime config:
120+
- The limit can be configured either by the CLI flag (`-distributor.instance-limits.max-inflight-push-requests`) or in the config:
121121
```
122-
distributor_instance_limits:
123-
max_inflight_push_requests: <int>
124-
```
125-
- The mixin configures the limit in the runtime config and can be fine-tuned via:
126-
```
127-
_config+:: {
128-
distributor_instance_limits+:: {
122+
distributor:
123+
instance_limits:
129124
max_inflight_push_requests: <int>
130-
}
131-
}
132125
```
133-
- When configured in the runtime config, changes are applied live without requiring an distributor restart
126+
- When configured the via the CLI flag or in the config these changes are applied with a distributor restart.
134127
- The configured limit can be queried via `cortex_distributor_instance_limits{limit="max_inflight_push_requests"})`
135128
136129
How to **fix**:

0 commit comments

Comments
 (0)