Alert NoOutputBytesProcessed should combine outputs (fluent#640)

We were seeing issues where we got alerted due to some outputs not sending data for more than 15 minutes.
For example, this is expected from journald logs on worker nodes in a cluster, thus this adjustment to the alert.

We now aggregate output per pod and alert if no bytes are sent for 15 minutes.

Signed-off-by: Victor Sollerhed <[email protected]>
Signed-off-by: Tom <[email protected]>

Author: MPV

monitoring/alerts.yaml

@@ -2,11 +2,10 @@ groups:
   - name: fluent-bit
     rules:
       - alert: NoOutputBytesProcessed
-        expr: rate(fluentbit_output_proc_bytes_total[5m]) == 0
+        expr: sum(rate(fluentbit_output_proc_bytes_total[5m])) by (pod) == 0
         annotations:
           message: |
-            Fluent Bit instance {{ $labels.instance }}'s output plugin {{ $labels.name }} has not processed any
-            bytes for at least 15 minutes.
+            Fluent Bit pod {{ $labels.pod }} has not processed any output bytes for at least 15 min.
           summary: No Output Bytes Processed
         for: 15m
         labels: