Added CortexFailingToTalkToConsul alert

Signed-off-by: Marco Pracucci <[email protected]>

Author: pracucci

CHANGELOG.md

@@ -66,6 +66,7 @@
 * [ENHANCEMENT] Allow to customize PromQL engine settings via `queryEngineConfig`. #399
 * [ENHANCEMENT] Add recording rules to improve responsiveness of Alertmanager dashboard. #387
 * [ENHANCEMENT] Add `CortexRolloutStuck` alert. #405
+* [ENHANCEMENT] Added `CortexFailingToTalkToConsul` alert. #406
 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
 * [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
 * [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335

cortex-mixin/alerts/alerts.libsonnet

@@ -715,5 +715,31 @@
         },
       ],
     },
+    {
+      name: 'cortex-consul-alerts',
+      rules: [
+        {
+          alert: 'CortexFailingToTalkToConsul',
+          expr: |||
+            (
+              sum by(%s, pod, status_code, kv_name) (rate(cortex_consul_request_duration_seconds_count{status_code!~"2.+"}[1m]))
+              /
+              sum by(%s, pod, status_code, kv_name) (rate(cortex_consul_request_duration_seconds_count[1m]))
+            )
+            # We want to get alerted only in case there's a constant failure.
+            == 1
+          ||| % [$._config.alert_aggregation_labels, $._config.alert_aggregation_labels],
+          'for': '5m',
+          labels: {
+            severity: 'warning',
+          },
+          annotations: {
+            message: |||
+              Cortex {{ $labels.pod }} in  %(alert_aggregation_variables)s is failing to talk to Consul store ${{ labels.kv_name }}.
+            ||| % $._config,
+          },
+        },
+      ],
+    },
   ],
 }

cortex-mixin/docs/playbooks.md

@@ -734,6 +734,18 @@ How to **investigate**:
 - Ensure there's no pod `NotReady` (the number of ready containers should match the total number of containers, eg. `1/1` or `2/2`)
 - Run `kubectl -n <namespace> describe statefulset <name>` or `kubectl -n <namespace> describe deployment <name>` and look at "Pod Status" and "Events" to get more information
 
+### CortexFailingToTalkToConsul
+
+This alert fires if a Cortex instance is failing to run any operation on Consul.
+
+How it **works**:
+- Consul is typically used to store the hash ring state.
+- If an instance is failing to talk to Consul, either the instance can't update the heartbeat in the ring or is failing to receive ring updates.
+
+How to **investigate**:
+- Ensure Consul is up and running.
+- Investigate the logs of the affected instance to find the specific error occurring when talking to Consul.
+
 ## Cortex routes by path
 
 **Write path**: