Skip to content

Conversation

zxiiro
Copy link
Contributor

@zxiiro zxiiro commented Jul 30, 2025

This adds a monitor to alert when unusually high GitHub API usage has been detected.

This adds a monitor to alert when unusually high GitHub API usage
has been detected.

Signed-off-by: Thanh Ha <[email protected]>
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a Datadog monitor to detect and alert on unusually high GitHub API usage. The monitor is designed to catch potential issues in the ALI system that might cause excessive API calls.

  • Adds a new Datadog event-based monitor for GitHub API usage
  • Configures alerting thresholds and notification messages
  • Provides actionable guidance for investigating high API usage incidents

Copy link

github-actions bot commented Jul 30, 2025

OpenTofu plan for prod

Plan: 1 to add, 0 to change, 0 to destroy.
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create

OpenTofu will perform the following actions:

  # datadog_monitor.GitHub_API_usage_unusually_high will be created
+   resource "datadog_monitor" "GitHub_API_usage_unusually_high" {
+       evaluation_delay    = (known after apply)
+       id                  = (known after apply)
+       include_tags        = false
+       message             = <<-EOT
            # GitHub API usage is unusually high
            
            We've detected that the GitHub API rate limit usage is higher than usual. This could be an indication of a problem in the ALI system causing higher than expected API calls.
            
            ## Action
            
            Review the rate limit metrics as well as API call count from the ALI for each API call to see if anything unusual is occurring.
        EOT
+       name                = "GitHub API usage unusually high"
+       new_host_delay      = 300
+       notify_no_data      = false
+       query               = "events(\"source:amazon_sns @title:\\\"GitHub API usage unusually high\\\"\").rollup(\"count\").last(\"5m\") > 0"
+       require_full_window = false
+       tags                = (known after apply)
+       type                = "event-v2 alert"

+       monitor_thresholds {
+           critical = "0"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

✅ Plan applied in Tofu Apply #13

Co-authored-by: Copilot <[email protected]>
@zxiiro zxiiro merged commit 81d43ed into main Jul 30, 2025
2 checks passed
@zxiiro zxiiro deleted the zxiiro/github-ratelimit branch July 30, 2025 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants