Add a check for long runner queues

[zxiiro]

Uses the PyTorch HUD API to check for long runner queues (>1h) and report to Slack if any runner queues have jobs waiting unusually long for a runner.

[github-actions]

OpenTofu plan for prod

Plan: 1 to add, 0 to change, 0 to destroy.

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create

OpenTofu will perform the following actions:

  # datadog_synthetics_test.pytorch-gha-runners-queue-check will be created
+   resource "datadog_synthetics_test" "pytorch-gha-runners-queue-check" {
+       id         = (known after apply)
+       locations  = [
+           "aws:us-west-2",
        ]
+       message    = <<-EOT
            Detected GitHub Runner Queue has jobs waiting unusually long for runners.
            
              Check https://hud.pytorch.org/metrics to determine which ones.
            
              @slack-pytorch-infra-alerts
        EOT
+       monitor_id = (known after apply)
+       name       = "GHA Runner Queue Check"
+       status     = "live"
+       tags       = [
+           "env:project",
+           "project:pytorch",
+           "service:gha-runners",
        ]
+       type       = "api"

+       assertion {
+           operator = "is"
+           target   = "200"
+           type     = "statusCode"
        }
+       assertion {
+           operator = "validatesJSONPath"
+           type     = "body"

+           targetjsonpath {
+               elementsoperator = "everyElementMatches"
+               jsonpath         = "$[?(@.avg_queue_s > 3600)].avg_queue_s"
+               operator         = "is"
            }
        }

+       options_list {
+           http_version        = "any"
+           min_location_failed = 1
+           tick_every          = 900
        }

+       request_definition {
+           method = "GET"
+           url    = "https://hud.pytorch.org/api/clickhouse/queued_jobs_by_label?parameters=%7B%7D"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

✅ Plan applied in Tofu Apply #17

[Copilot]

Pull Request Overview

This PR adds monitoring for GitHub Actions runner queue health by creating a Datadog synthetic test that checks for jobs waiting longer than 1 hour for available runners.

• Introduces a new synthetic API test to monitor GitHub runner queue times
• Configures the test to query PyTorch HUD API and alert via Slack when queues exceed 1 hour
• Sets up automated monitoring with 15-minute intervals to catch runner capacity issues