-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Description
Issue:
When we configure sc-alert as a sidecar, it takes a long time (about 5min ?) to populate our ~900 alert rules.
(also, if 1 alert rule has an issue, like having a too long uid, the sidecar will stop there and not process anymore alert)
Workaround:
I figured that I could setup sc-alert as Init, where it load up all the alerts in one go, which works great !
New issue:
But I have no lost the side-car utility that watches for changes and push them to grafana.
Fix ideas:
1) Run both sidecar and init:
=> Do you think we could change the line:
{{- if and .Values.sidecar.alerts.enabled (not .Values.sidecar.alerts.initAlerts) }}
from https://github.com/grafana/helm-charts/blob/main/charts/grafana/templates/_pod.tpl#L321C1-L321C85
to something else that would allow both init and sidecar would be enabled ?
Maybe to this:
New value:
sidecar.alerts.initAndSidecar: false # default to false to avoid changing current behavior
New line:
{{- if and .Values.sidecar.alerts.enabled (or Values.sidecar.alerts.initAndSidecar (not .Values.sidecar.alerts.initAlerts)) }}
2) Improve the sidecar:
is there a way to make the sidecar load all alert rules in one go at its start ?