Add notifications to the high number of messages alert

[jordanconway]

No description provided.

[github-actions]

OpenTofu plan for prod

Plan: 1 to add, 1 to change, 0 to destroy.

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
!~  update in-place

OpenTofu will perform the following actions:

  # datadog_monitor.ci_retry_deadletter will be updated in-place
!~  resource "datadog_monitor" "ci_retry_deadletter" {
        id                   = "9264784"
!~      message              = <<-EOT
            An unusually high number of messages is in the `ghci-lf-queued-builds-dead-letter` queue. This may indicate a potential outage with the project's autoscaler.
            Verify that system is able to scale up EC2 instances by checking logs.
            
            @webhook-lf-incident-io
          + @slack-PyTorch-pytorch-infra-alerts
          + @slack-Linux_Foundation-pytorch-alerts
        EOT
        name                 = "ALI AutoScaler Dead Letter Queue High Number Of Messages"
        tags                 = []
#        (16 unchanged attributes hidden)

#        (1 unchanged block hidden)
    }

  # datadog_webhook.lf-incident-io will be created
+   resource "datadog_webhook" "lf-incident-io" {
+       custom_headers = (sensitive value)
+       encode_as      = "json"
+       id             = (known after apply)
+       name           = "lf-incident-io"
+       payload        = jsonencode(
            {
+               alert_transition  = "$ALERT_TRANSITION"
+               deduplication_key = "****************************"
+               description       = "$EVENT_MSG"
+               metadata          = {
+                   alert_metric   = "$ALERT_METRIC"
+                   alert_priority = "$ALERT_PRIORITY"
+                   alert_query    = "$ALERT_QUERY"
+                   alert_scope    = "$ALERT_SCOPE"
+                   alert_status   = "$ALERT_STATUS"
+                   alert_title    = "$ALERT_TITLE"
+                   alert_type     = "$ALERT_TYPE"
+                   alert_url      = "$LINK"
+                   date           = "$DATE"
+                   event_type     = "$EVENT_TYPE"
+                   hostname       = "$HOSTNAME"
+                   id             = "$ID"
+                   last_updated   = "$LAST_UPDATED"
+                   logs_sample    = "$LOGS_SAMPLE"
+                   org            = {
+                       id   = "$ORG_ID"
+                       name = "$ORG_NAME"
                    }
+                   snapshot_url   = "$SNAPSHOT"
+                   tags           = "$TAGS"
                }
+               source_url        = "$LINK"
+               title             = "$EVENT_TITLE"
            }
        )
+       url            = "https://api.incident.io/v2/alert_events/datadog/01JKTRSFTE6H2SR4AFM4VGWZFN"
    }

Plan: 1 to add, 1 to change, 0 to destroy.

❌ Error applying plan in Tofu Apply #10

[jordanconway]

made some manual change to the incident.io webhook which may explain the lack of idempotency

[github-actions]

OpenTofu plan for prod

Plan: 1 to add, 0 to change, 0 to destroy.

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create

OpenTofu will perform the following actions:

  # datadog_webhook.lf-incident-io will be created
+   resource "datadog_webhook" "lf-incident-io" {
+       custom_headers = (sensitive value)
+       encode_as      = "json"
+       id             = (known after apply)
+       name           = "lf-incident-io"
+       payload        = jsonencode(
            {
+               alert_transition  = "$ALERT_TRANSITION"
+               deduplication_key = "****************************"
+               description       = "$EVENT_MSG"
+               metadata          = {
+                   alert_metric   = "$ALERT_METRIC"
+                   alert_priority = "$ALERT_PRIORITY"
+                   alert_query    = "$ALERT_QUERY"
+                   alert_scope    = "$ALERT_SCOPE"
+                   alert_status   = "$ALERT_STATUS"
+                   alert_title    = "$ALERT_TITLE"
+                   alert_type     = "$ALERT_TYPE"
+                   alert_url      = "$LINK"
+                   date           = "$DATE"
+                   event_type     = "$EVENT_TYPE"
+                   hostname       = "$HOSTNAME"
+                   id             = "$ID"
+                   last_updated   = "$LAST_UPDATED"
+                   logs_sample    = "$LOGS_SAMPLE"
+                   org            = {
+                       id   = "$ORG_ID"
+                       name = "$ORG_NAME"
                    }
+                   snapshot_url   = "$SNAPSHOT"
+                   tags           = "$TAGS"
                }
+               source_url        = "$LINK"
+               title             = "$EVENT_TITLE"
            }
        )
+       url            = "https://api.incident.io/v2/alert_events/datadog/01JKTRSFTE6H2SR4AFM4VGWZFN"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

✅ Plan applied in Tofu Apply #10