Improve dropped packages PrometheusRules #2877
Description
Description
We should look over improving our current NetworkPolicy PrometheusRules, make them less sensitive for for when they fire alerts that would ping ops people, while still keeping information about any NetworkPolicies that would cause any dropped package.
See the discussion here for more context: #2837 (comment)
As suggested in the linked PR discussion, one approach could be to make the current Frequent alerts (this would be for both Calico and Cilium) which trigger for any dropped packages (due to the threshold being > 0) be changed to be a separate alert with severity set to info as to not ping on call engineers, and change the Frequent rules to be set to a more reasonable threshold.
The severity for the Scarce rules could also be bumped with this change to warning instead.
Additional context
No response
Definition of done
- The DroppedPackages prometheus rules for both Calico and Cilium have been updated to be more clear and improve both ops and dev work