Enhanced Alert Configuration with Multiple, Escalating Alerts per Status

[dug42]

I'd like to propose an enhancement to the alert configuration system that would significantly improve our ability to manage service status changes and automate appropriate responses.

Current Limitation:
Currently, we are limited to a single alert configuration per status type (e.g., one "DOWN" alert, one DEGRADED alert). This can restrict our ability to implement nuanced and escalating responses to service degradation or outages.

Proposed Feature: Multiple, Configurable Alerts per Status
My suggestion is to allow users to configure zero to N (0-N) distinct alerts for a given service status, such as DOWN or DEGRADED. Each of these individual alerts would possess its own independent configuration settings, including:

• Type: The status type it monitors (e.g., DOWN, DEGRADED).
• Failure Threshold: The number of consecutive failures (or duration) required to trigger this specific alert.
• Success Threshold: The number of consecutive successes required to resolve this specific alert.
• Create Incident: A flag to determine if this alert should automatically create a formal incident within the system.
• Message: A custom message for this specific alert.
• Triggers: A list of actions to execute when this alert is triggered (e.g., send email, fire a webhook, send Slack message).

Benefits of this Approach:
This flexible configuration would enable us to implement sophisticated, escalating alert patterns tailored to the severity and duration of an issue. This offers several key advantages:

1. Nuanced Response: Differentiate between a brief, transient blip and a sustained outage, triggering different actions for each.
2. Prevent Alert Fatigue: Avoid immediate, high-priority alerts for minor, self-resolving issues.
3. Automated Escalation: Ensure the right teams are notified and the appropriate response actions are initiated as an issue persists or worsens.
4. Tailored Notifications: Send targeted notifications to different stakeholders (e.g., internal teams, on-call engineers, management) based on the severity.

Illustrative Example: Escalating "DOWN" Alerts

Consider the following scenario for a service experiencing DOWN status, demonstrating how multiple alerts could work in an escalating fashion:

• First Alert - Initial Blip Notification

  • Type: DOWN
  • Failure Threshold: 2 (e.g., 2 consecutive checks fail)
  • Success Threshold: 2
  • Create Incident: YES (for immediate logging and initial investigation)
  • Triggers: None (or perhaps a very low-priority internal log entry)
  • Purpose: This alert would catch the very first signs of an issue, creating an initial record for rapid internal awareness.

• Second Alert - Sustained Issue Notification

  • Type: DOWN
  • Failure Threshold: 5 (e.g., 5 consecutive checks fail, indicating a more persistent problem)
  • Success Threshold: 2
  • Create Incident: NO (assuming first Alert already created one, or this is just for broader team awareness)
  • Triggers: email (to a broader team distribution list for awareness, but not yet a high-priority PagerDuty alert)
  • Purpose: If the DOWN status persists beyond a minor blip, this alert notifies a wider group that the problem is not self-correcting.

• Third Alert - Critical Outage Escalation

  • Type: DOWN
  • Failure Threshold: 10 (e.g., 10 consecutive checks fail, indicating a significant outage)
  • Success Threshold: 2
  • Create Incident: NO (as the primary incident should already be active)
  • Triggers: webhook (to an incident management system like PagerDuty or Opsgenie, triggering on-call rotations and critical response procedures)
  • Purpose: For severe and prolonged outages, this alert ensures immediate, high-priority action is taken by the on-call team and triggers full incident response workflows.

This proposed functionality would empower users with much greater control and automation capabilities for their monitoring and incident response strategies.

[rajnandan1]

Thanks for the suggestion. I have implemented in v4