remove group for single alert, update runbook, add changelog entry

Author: Tyler Reid

CHANGELOG.md

@@ -67,6 +67,7 @@
 * [ENHANCEMENT] Add recording rules to improve responsiveness of Alertmanager dashboard. #387
 * [ENHANCEMENT] Add `CortexRolloutStuck` alert. #405
 * [ENHANCEMENT] Added `CortexKVStoreFailure` alert. #406
+* [ENHANCEMENT] Added `CortexDistributorReachingInflightPushRequestLimit` alert. #408
 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
 * [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
 * [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335

cortex-mixin/alerts/alerts.libsonnet

@@ -350,11 +350,6 @@
             |||,
           },
         },
-      ],
-    },
-    {
-      name: 'cortex_distributor_inflight_push_request_alert',
-      rules: [
         {
           alert: 'CortexDistributorReachingInflightPushRequestLimit',
           expr: |||

cortex-mixin/docs/playbooks.md

@@ -114,7 +114,7 @@ This alert fires when the `cortex_distributor_inflight_push_requests` per distri
 
 In case of **emergency**:
 - If the actual number of inflight push requests is very close to or already at the set limit, then you can increase the limit via CLI flag or config to gain some time
-- Increasing the limit will increase the the number of inflight push requests which will increase distributors' memory utilization. Please monitor the distributors' memory utilization via the `Cortex / Writes Resources` dashboard
+- Increasing the limit will increase the number of inflight push requests which will increase distributors' memory utilization. Please monitor the distributors' memory utilization via the `Cortex / Writes Resources` dashboard
 
 How the limit is **configured**:
 - The limit can be configured either by the CLI flag (`-distributor.instance-limits.max-inflight-push-requests`) or in the config:
@@ -123,7 +123,7 @@ How the limit is **configured**:
     instance_limits:
       max_inflight_push_requests: <int>
   ```
-- When configured the via the CLI flag or in the config these changes are applied with a distributor restart.
+- These changes are applied with a distributor restart.
 - The configured limit can be queried via `cortex_distributor_instance_limits{limit="max_inflight_push_requests"})`
 
 How to **fix**: