Merge pull request #206 from grafana/compactor-alert

Added alert for failed compator run.

Author: pracucci

CHANGELOG.md

@@ -11,6 +11,7 @@
 * [ENHANCEMENT] Add the Ruler to the read resources dashboard #205
 * [ENHANCEMENT] Read dashboards now use `cortex_querier_request_duration_seconds` metrics to allow for accurate dashboards when deploying Cortex as a single-binary. #199
 * [ENHANCEMENT] Improved Ruler dashboard. Includes information about notifications, reads/writes, and per user per rule group evaluation. #197, #205
+* [ENHANCEMENT] Add new `CortexCompactorRunFailed` alert when compactor run fails. #206
 * [ENHANCEMENT] Add `flusher-job-blocks.libsonnet` with support for flushing blocks disks. #187
 * [ENHANCEMENT] Add more alerts on failure conditions for ingesters when running the blocks storage. #208
 * [FEATURE] Latency recording rules for the metric`cortex_querier_request_duration_seconds` are now part of a `cortex_querier_api` rule group. #199

cortex-mixin/alerts/compactor.libsonnet

@@ -63,6 +63,21 @@
             message: 'Cortex Compactor {{ $labels.namespace }}/{{ $labels.instance }} has not uploaded any block in the last 24 hours.',
           },
         },
+        {
+          // Alert if compactor fails.
+          alert: 'CortexCompactorRunFailed',
+          expr: |||
+            increase(cortex_compactor_runs_failed_total[2h]) > 1
+          |||,
+          labels: {
+            severity: 'warning',
+          },
+          annotations: {
+            message: |||
+              {{ $labels.job }}/{{ $labels.instance }} failed to run compaction.
+            |||,
+          },
+        },
       ],
     },
   ],