Redfish exporter: Decrease sensitivity of alert (#1358)

The redfish exporter is prone to failed scrapes. Lets wait for mulitple
failed scrapes before triggering an alert. This should still catch the
case where it is completely dead, but reduce the false positives from
failed scrapes.

Author: jovial

etc/kayobe/kolla/config/prometheus/prometheus.rules

@@ -7,14 +7,23 @@ groups:
   rules:
 
   - alert: PrometheusTargetMissing
-    expr: up == 0
+    expr: up{job!="redfish-exporter-seed"} == 0
     for: 5m
     labels:
       severity: critical
     annotations:
       summary: "Prometheus target missing (instance {{ $labels.instance }})"
       description: "A Prometheus target has disappeared. An exporter might have crashed."
 
+  - alert: PrometheusTargetMissing
+    expr: up{job="redfish-exporter-seed"} == 0
+    for: 15m
+    labels:
+      severity: critical
+    annotations:
+      summary: "Prometheus target missing (instance {{ $labels.instance }})"
+      description: "A Prometheus target has disappeared. An exporter might have crashed."
+
   - alert: PrometheusAllTargetsMissing
     expr: count by (job) (up) == 0
     for: 1m

releasenotes/notes/reduces-sensitivity-of-redfish-target-alerts-a3d77a3f0c3dac8a.yaml

@@ -0,0 +1,6 @@
+---
+fixes:
+  - |
+    Changes the duration for which redfish exporter must continually fail
+    scrapes before triggering an alert to 15 minutes.  This should hopefully
+    reduce some alert spam.