Skip to content

Commit 9c4da69

Browse files
author
Tyler Reid
committed
Add playbook for inflight push limit
1 parent 0ed69e8 commit 9c4da69

File tree

1 file changed

+29
-1
lines changed

1 file changed

+29
-1
lines changed

cortex-mixin/docs/playbooks.md

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,35 @@ How to **fix**:
109109
1. Assuming shuffle-sharding is enabled, scaling up ingesters will lower the number of tenants per ingester. However, the effect of this change will be visible only after `-blocks-storage.tsdb.close-idle-tsdb-timeout` period so you may have to temporarily increase the limit
110110
111111
### CortexDistributorReachingInflightPushRequestLimits
112-
_TODO: this playbook has not been written yet._
112+
113+
This alert fires when the `cortex_distributor_inflight_push_requests` per distributor instance limit is enabled and the actual number of inflight push requests is approaching the set limit. Once the limit is reached, push requests to the distributor will fail (5xx) for new requests, while existing inflight push requests will continue to succeed.
114+
115+
In case of **emergency**:
116+
- If the actual number of inflight push requests is very close to or already at the set limit, then you can increase the limit via runtime config to gain some time
117+
- Increasing the limit will increase the distributor' memory utilization. Please monitor the distributors' memory utilization via the `Cortex / Writes Resources` dashboard
118+
119+
How the limit is **configured**:
120+
- The limit can be configured either on CLI (`-distributor.instance-limits.max-inflight-push-requests`) or in the runtime config:
121+
```
122+
distributor_instance_limits:
123+
max_inflight_push_requests: <int>
124+
```
125+
- The mixin configures the limit in the runtime config and can be fine-tuned via:
126+
```
127+
_config+:: {
128+
distributor_instance_limits+:: {
129+
max_inflight_push_requests: <int>
130+
}
131+
}
132+
```
133+
- When configured in the runtime config, changes are applied live without requiring an distributor restart
134+
- The configured limit can be queried via `cortex_distributor_instance_limits{limit="max_inflight_push_requests"})`
135+
136+
How to **fix**:
137+
1. **Temporarily increase the limit**<br />
138+
If the actual number of inflight push requests is very close to or already hit the limit.
139+
2. **Scale up distributors**<br />
140+
Scaling up distributors will lower the number of inflight push requests per distributor.
113141
114142
### CortexRequestLatency
115143

0 commit comments

Comments
 (0)