Skip to content

Multi-threshold alerts with UI-provided channels fail to send notificationsΒ #10591

@mango766

Description

@mango766

Bug Description

When creating multi-threshold (multi-level) alerts via the SigNoz UI, notification channels configured at condition.thresholds.spec[].channels are not triggered. The alert fires but no notifications are sent (email, webhook, etc. fail silently).

This happens because preferredChannels at the rule level (rule.data.preferredChannels) is not being set when channels are configured per-threshold in the UI, but GetMatchers() in pkg/alertmanager/alertmanagerstore/sqlalertmanagerstore/config.go reads from preferredChannels to determine which receivers to route alerts to.

Root Cause

The issue is in pkg/types/ruletypes/api_params.go, specifically in the processRuleDefaults() function (lines 187-258), which processes rules when they are unmarshalled from JSON.

For basic threshold alerts, the code correctly populates preferredChannels from the top-level rule (line 241):

Channels: r.PreferredChannels,

However, for multi-threshold alerts where channels are configured per-threshold level via the UI (stored at condition.thresholds.spec[].channels), the processRuleDefaults() function only handles the DefaultSchemaVersion case and does not sync the threshold-level channels back to the top-level preferredChannels field. This causes GetMatchers() to return an empty channel list, and the alertmanager dispatcher has no receivers to route to.

Expected Behavior

When channels are configured for thresholds in the multi-threshold alert UI, the rule should send notifications to those configured channels.

Actual Behavior

Alerts fire (state transitions to Firing), but no notifications are sent because GetMatchers() returns an empty channel list when preferredChannels is not set at the rule level.

Affected Code

  • pkg/types/ruletypes/api_params.go - processRuleDefaults() function
  • pkg/alertmanager/alertmanagerstore/sqlalertmanagerstore/config.go - GetMatchers() function (line 196)

Reproduction Steps

  1. Create a multi-threshold alert rule in the UI with multiple thresholds (e.g., critical, warning)
  2. Configure notification channels for each threshold level
  3. Trigger the alert
  4. Observe that the alert fires but no notifications are sent

Workaround

Manually set preferredChannels via the API:

# GET the rule, add preferredChannels, PUT it back
curl -X GET .../api/v1/rules/{id} | jq '.data.preferredChannels = ["email", "webhook-xyz"]' | curl -X PUT .../api/v1/rules/{id} -d @-

Then restart the SigNoz instance to trigger a config sync.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions