-
Notifications
You must be signed in to change notification settings - Fork 2k
Description
Bug Description
When creating multi-threshold (multi-level) alerts via the SigNoz UI, notification channels configured at condition.thresholds.spec[].channels are not triggered. The alert fires but no notifications are sent (email, webhook, etc. fail silently).
This happens because preferredChannels at the rule level (rule.data.preferredChannels) is not being set when channels are configured per-threshold in the UI, but GetMatchers() in pkg/alertmanager/alertmanagerstore/sqlalertmanagerstore/config.go reads from preferredChannels to determine which receivers to route alerts to.
Root Cause
The issue is in pkg/types/ruletypes/api_params.go, specifically in the processRuleDefaults() function (lines 187-258), which processes rules when they are unmarshalled from JSON.
For basic threshold alerts, the code correctly populates preferredChannels from the top-level rule (line 241):
Channels: r.PreferredChannels,
However, for multi-threshold alerts where channels are configured per-threshold level via the UI (stored at condition.thresholds.spec[].channels), the processRuleDefaults() function only handles the DefaultSchemaVersion case and does not sync the threshold-level channels back to the top-level preferredChannels field. This causes GetMatchers() to return an empty channel list, and the alertmanager dispatcher has no receivers to route to.
Expected Behavior
When channels are configured for thresholds in the multi-threshold alert UI, the rule should send notifications to those configured channels.
Actual Behavior
Alerts fire (state transitions to Firing), but no notifications are sent because GetMatchers() returns an empty channel list when preferredChannels is not set at the rule level.
Affected Code
pkg/types/ruletypes/api_params.go-processRuleDefaults()functionpkg/alertmanager/alertmanagerstore/sqlalertmanagerstore/config.go-GetMatchers()function (line 196)
Reproduction Steps
- Create a multi-threshold alert rule in the UI with multiple thresholds (e.g., critical, warning)
- Configure notification channels for each threshold level
- Trigger the alert
- Observe that the alert fires but no notifications are sent
Workaround
Manually set preferredChannels via the API:
# GET the rule, add preferredChannels, PUT it back
curl -X GET .../api/v1/rules/{id} | jq '.data.preferredChannels = ["email", "webhook-xyz"]' | curl -X PUT .../api/v1/rules/{id} -d @-
Then restart the SigNoz instance to trigger a config sync.