Add inline scanning alerts for sysdig admission controller

Signed-off-by: daviddetorres <[email protected]>

Author: daviddetorres

resources/sysdig-admission-controller/ALERTS.md

@@ -6,4 +6,16 @@ The Admission Controller is not receiving Kubernetes Audit events.
 Kubernetes Audit events is being throttled.
 
 ## [Sysdig Admission Controller] Scanning Events Throttling
-Scanning events is being throttled.
+Scanning events is being throttled.
+
+## [Sysdig Admission Controller] Inline Scanning Throttling
+Scanning events is being throttled.
+
+## [Sysdig Admission Controller] High Error Rate In Scan Status From Backend
+High Error Rate In Scan Status From Backend.
+
+## [Sysdig Admission Controller] High Error Rate In Scan Report From Backend
+High Error Rate In Scan Report From Backend.
+
+## [Sysdig Admission Controller] High Error Rate In Image Scan
+High Error Rate In Image Scan.

resources/sysdig-admission-controller/alerts.yaml

@@ -26,7 +26,7 @@ configurations:
           ) > 32
         for: 15m
         labels:
-          severity: Medium
+          severity: medium
         annotations:
           summary: Kubernetes Audit events is being throttled.
       - alert: "[Sysdig Admission Controller] Scanning Events Throttling"
@@ -36,7 +36,50 @@ configurations:
           ) > 32
         for: 15m
         labels:
-          severity: Medium
+          severity: medium
         annotations:
           summary: Scanning events is being throttled.
-
+      - alert: "[Sysdig Admission Controller] Inline Scanning Throttling"
+        expr: |
+          sum(queue_length) > 0
+        for: 15m
+        labels:
+          severity: medium
+        annotations:
+          summary: The inline scanning queue is not empty for a long time.
+      - alert: "[Sysdig Admission Controller] High Error Rate In Scan Status From Backend"
+        expr: |
+          sum
+            ( rate(scanner_scan_status_error_from_backend_count[5m]) 
+            / (rate(scanner_scan_status_retrieved_from_backend_count[5m]) 
+              + rate(scanner_scan_status_error_from_backend_count[5m]))
+          ) > 0.15
+        for: 15m
+        labels:
+          severity: high
+        annotations:
+          summary: High Error Rate In Scan Status From Backend.
+      - alert: "[Sysdig Admission Controller] High Error Rate In Scan Report From Backend"
+        expr: |
+          sum
+            ( rate(scanner_scan_report_error_from_backend_count[5m]) 
+            / (rate(scanner_scan_report_retrieved_from_backend_count[5m]) 
+              + rate(scanner_scan_report_error_from_backend_count[5m]))
+          ) > 0.15
+        for: 15m
+        labels:
+          severity: high
+        annotations:
+          summary: High Error Rate In Scan Report From Backend.
+      - alert: "[Sysdig Admission Controller] High Error Rate In Image Scan"
+        expr: |
+          sum
+            ( rate(scanner_scan_errors[5m]) 
+            / (rate(scanner_scan_success[5m]) 
+              + rate(scanner_scan_errors[5m]))
+          ) > 0.15
+        for: 15m
+        labels:
+          severity: high
+        annotations:
+          summary: High Error Rate In Image Scan.