sentry-scanner.yml

name: Sentry Proactive Scanner

on:
  schedule:
    - cron: '0 * * * *'
  workflow_dispatch:

permissions:
  contents: read
  issues: write
  id-token: write

jobs:
  scan:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          sparse-checkout: |
            CLAUDE.md
            run/
            .github/

      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

      - name: Add known hosts
        run: |
          mkdir -p ~/.ssh
          ssh-keyscan -H 157.90.154.200 >> ~/.ssh/known_hosts

      - name: Claude Code - Scan
        uses: anthropics/claude-code-action@v1
        with:
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
          claude_args: "--model claude-sonnet-4-20250514 --max-turns 60 --allowedTools Bash,Read,Glob,Grep"
          show_full_output: true
          prompt: |
            You are a proactive Sentry scanner for the Ethernal project. Your job is to scan ALL unresolved Sentry issues (errors AND performance) and decide which ones need GitHub issues created.

            Sentry base URL: https://sentry.io
            Org: sentry
            Backend project: ethernal-backend (id=2)
            Frontend project: ethernal-frontend (id=3)

            **IMPORTANT API NOTES for this Sentry instance (v26.2.1)**:
            - Only `statsPeriod` values of `24h` and `14d` work. Do NOT use `1h`.
            - Always save curl output to a temp file first, then process with jq. Do NOT pipe curl directly to jq — it fails intermittently.
            - Pattern: `curl -s -o /tmp/resp.json "URL" -H "Authorization: Bearer $SENTRY_API_TOKEN" && jq '...' /tmp/resp.json`
            - **CRITICAL — `count` is LIFETIME, not 24h.** The `count` field on each issue in the list response is the lifetime `times_seen` total, NOT the 24h rolling volume. Do NOT use it for thresholds or report it as "Events (24h)" — that caused a false-incident storm (issues #1218–#1232 were all closed as self-resolved because the scanner reported 6,000+ "24h events" when actual 24h volume was 2–16). The TRUE 24h volume is `([.stats."24h"[]?[1]] | add)` — sum of the hourly buckets in the `stats."24h"` array. Always compute and use this value.

            ## Step 0: Pre-flight health & auth probe (MANDATORY)

            Before doing anything else, distinguish three top-level states:
            - **Service down** (Sentry API itself returns 5xx)
            - **Auth broken** (Sentry API up, but token is rejected with 401/403)
            - **Healthy** (proceed to Step 1)

            Past incidents created **separate issues per scanner run for the same outage** (#1301/#1302/#1303 + #1305 were all the same Sentry outage, and #1304 was the actual token-regen item that got buried). This step deduplicates against open infra issues and **must run before any Sentry query**.

            ```bash
            BASE_UNAUTH=$(curl -s -o /tmp/preflight_base.json -w "%{http_code}" "https://sentry.io/api/0/" --max-time 15 || echo "000")
            AUTH_PROBE=$(curl -s -o /tmp/preflight_auth.json -w "%{http_code}" "https://sentry.io/api/0/projects/" -H "Authorization: Bearer $SENTRY_API_TOKEN" --max-time 15 || echo "000")
            echo "preflight base=$BASE_UNAUTH auth=$AUTH_PROBE"
            ```

            Decision matrix:

            | base | auth | Verdict | Action |
            |------|------|---------|--------|
            | 2xx  | 2xx  | Healthy | Continue to Step 1 |
            | 2xx  | 401/403 | Token bad | Run "AUTH_BROKEN handler" below, then exit 0 |
            | 5xx / 000 | any | Service down | Run "SERVICE_DOWN handler" below, then exit 0 |
            | other | other | Unknown | Run "SERVICE_DOWN handler" with verdict="unknown", exit 0 |

            ### SERVICE_DOWN handler

            Dedup against an open service-down issue from the last 24h, then either comment on it or create one. **Never create a second open service-down issue.**

            1. Search for an existing open issue:
               ```bash
               gh issue list --state open --label sentry --search 'in:title "Sentry monitoring service down"' \
                 --json number,createdAt \
                 --jq '[.[] | select((now - (.createdAt | fromdateiso8601)) < 86400)] | .[0].number'
               ```
            2. If a number comes back: comment on it with `gh issue comment <num>` noting the current timestamp, the HTTP codes from the preflight, and "outage still in progress". Then `exit 0`.
            3. If no existing issue: `gh issue create` with:
               - title: `Sentry monitoring service down - API returning <BASE_UNAUTH> errors`
               - labels: `sentry`, `needs-human`, `infra-alert`
               - body: the timestamp, both HTTP codes, the first 500 chars of `/tmp/preflight_base.json` if it has content, and "Scanner skipped this cycle — will retry on next hourly run."
               Then `exit 0`.

            ### AUTH_BROKEN handler

            Dedup against an open auth-issue from the last 24h. Title pattern: `Sentry API scanner failing - authentication`.

            1. Search for an existing open issue:
               ```bash
               gh issue list --state open --label sentry --search 'in:title "Sentry API scanner failing"' \
                 --json number,createdAt \
                 --jq '[.[] | select((now - (.createdAt | fromdateiso8601)) < 86400)] | .[0].number'
               ```
            2. If a number comes back: comment on it noting the current timestamp and that the authenticated API still returns HTTP `<AUTH_PROBE>`. Then `exit 0`.
            3. If no existing issue: `gh issue create` with:
               - title: `Sentry API scanner failing - authentication/server errors`
               - labels: `sentry`, `needs-human`
               - body: timestamps, both HTTP codes, and these manual action items:
                 1. Log into https://sentry.io → Settings → Account → API → Auth Tokens
                 2. Regenerate the token with scopes: `org:read`, `project:read`, `event:read`, `event:admin`
                 3. Update the `SENTRY_API_TOKEN` GitHub Actions secret
                 4. Re-run this workflow manually
               Then `exit 0`.

            Only proceed past Step 0 if the preflight verdict is "Healthy".

            ## Step 1: Query Sentry for all unresolved issues

            Run ALL 6 queries in a single bash command to save turns. Save each to a temp file:

            ```bash
            AUTH="Authorization: Bearer $SENTRY_API_TOKEN"
            BASE="https://sentry.io/api/0/projects/antoine-0l"

            curl -s -o /tmp/be_err.json "$BASE/ethernal-backend/issues/?query=is:unresolved+issue.category:error&statsPeriod=24h&limit=100" -H "$AUTH"
            curl -s -o /tmp/be_perf.json "$BASE/ethernal-backend/issues/?query=is:unresolved+issue.category:performance&statsPeriod=24h&limit=100" -H "$AUTH"
            curl -s -o /tmp/be_reg.json "$BASE/ethernal-backend/issues/?query=is:regressed&statsPeriod=24h&limit=50" -H "$AUTH"
            curl -s -o /tmp/fe_err.json "$BASE/ethernal-frontend/issues/?query=is:unresolved+issue.category:error&statsPeriod=24h&limit=100" -H "$AUTH"
            curl -s -o /tmp/fe_perf.json "$BASE/ethernal-frontend/issues/?query=is:unresolved+issue.category:performance&statsPeriod=24h&limit=100" -H "$AUTH"
            curl -s -o /tmp/fe_reg.json "$BASE/ethernal-frontend/issues/?query=is:regressed&statsPeriod=24h&limit=50" -H "$AUTH"

            # ALWAYS project events_24h (rolling) — never use raw `count` (lifetime) for decisions or reports.
            PROJ='[.[] | {id, title, lifetime_count: (.count | tonumber), events_24h: ([.stats."24h"[]?[1]] | add // 0), lastSeen, shortId, isRegression: (.isRegression // false)}]'
            echo "=== Backend Errors ===" && jq "$PROJ" /tmp/be_err.json
            echo "=== Backend Performance ===" && jq "$PROJ" /tmp/be_perf.json
            echo "=== Backend Regressed ===" && jq "$PROJ" /tmp/be_reg.json
            echo "=== Frontend Errors ===" && jq "$PROJ" /tmp/fe_err.json
            echo "=== Frontend Performance ===" && jq "$PROJ" /tmp/fe_perf.json
            echo "=== Frontend Regressed ===" && jq "$PROJ" /tmp/fe_reg.json
            ```

            From this point on, **always use `events_24h` (computed above) for thresholds and reporting**. The `lifetime_count` is informational only — useful as context ("issue has fired 6,953 times total but only 2 in last 24h") but never a decision input.

            Deduplicate by `(project, id)` — regressed issues may appear in both the error/performance and regressed queries.

            ## Step 2: Filter already-tracked issues

            In the SAME bash call, also fetch existing GitHub sentry issues:
            ```bash
            gh issue list --label sentry --state all --limit 200 --json number,title,body -q '.[].body' > /tmp/gh_issues.txt
            ```
            For each Sentry issue, check: `grep -c "issues/SENTRY_ID/" /tmp/gh_issues.txt`
            Skip any issue that already has a GitHub issue (open or closed).

            ## Step 3: Correlate related issues into incidents

            Before evaluating issues individually, **group them by error class**. Issues that share the same exception type (e.g., `SequelizeConnectionAcquireTimeoutError`, `SequelizeDatabaseError: query_wait_timeout`) AND have `lastSeen` within 30 minutes of each other are symptoms of a single incident, not independent bugs.

            For each group of 3+ correlated issues:
            - Create ONE umbrella GitHub issue (not one per symptom)
            - Title: `Sentry (incident): [shared error class] across [N] endpoints`
            - Body: list all affected Sentry issues, their stack traces, and event counts
            - Add labels: `sentry`, `incident`, `needs-human`
            - Do NOT create individual issues for the symptoms
            - This signals to the auto-fix workflow that it should NOT attempt individual fixes

            For groups of 1-2 issues, proceed to individual evaluation in Step 4.

            ## Step 4: Evaluate each new issue

            For each new issue that passes the filter AND was not grouped into an incident above, fetch event context. Use temp files:
            ```bash
            curl -s -o /tmp/event.json "https://sentry.io/api/0/issues/{id}/events/latest/" -H "Authorization: Bearer $SENTRY_API_TOKEN"
            jq '{message: .message, tags: [.tags[]? | select(.key == "transaction" or .key == "url") | {key, value}], exception: .entries[0]?.data.values[0]?.stacktrace.frames[-3:]?}' /tmp/event.json
            ```

            Then categorize into ONE of:

            ### AUTO-SKIP + RESOLVE in Sentry
            - **Any issue with `events_24h == 0`** (regardless of lifetime count) — it has self-resolved, mark resolved in Sentry
            - Connection/transient errors (SequelizeConnectionError, ECONNRESET, "Connection terminated unexpectedly") — UNLESS `events_24h >= 30`
            - Rate limiting errors
            - Expected validation errors (user input)
            - Third-party service errors we can't control
            - Low-impact edge cases (deprecated browser, obscure user input)
            - Performance issues on endpoints that no longer exist
            - Performance issues with `events_24h == 0` that are stale

            To resolve: `curl -s -X PUT "https://sentry.io/api/0/issues/{id}/" -H "Authorization: Bearer $SENTRY_API_TOKEN" -H "Content-Type: application/json" -d '{"status": "resolved"}'`

            ### CREATE GITHUB ISSUE (prioritized) — all event counts below are `events_24h`, NOT lifetime
            **Priority 1 — Regressions** (always create, regardless of event count):
            - Any issue where `isRegression: true` AND `events_24h >= 1` — a previous fix didn't hold AND it's still firing
            - Regressions with `events_24h == 0` go to AUTO-SKIP+RESOLVE — the regression flag is stale
            - Title prefix: "Sentry (regression):" for errors, "Perf (regression):" for performance

            **Priority 2 — High-impact errors**:
            - Null/type errors, unhandled promise rejections with `events_24h >= 2`
            - Systematic issues with `events_24h >= 5`

            **Priority 3 — High-impact performance** (user-facing hot paths only):
            - N+1 queries with `events_24h >= 50` AND on a user-facing endpoint or blockSync hot path
            - Slow DB queries with `events_24h >= 50` AND p95 > 2s

            **Priority 4 — Medium performance**:
            - N+1 queries with `events_24h` 20–49 on user-facing endpoints
            - Background job performance issues with `events_24h >= 100` (higher bar since they don't affect UX)

            ### SKIP (leave unresolved in Sentry)
            - Non-regressed issues with `events_24h < 2` (might be one-off)
            - Performance issues on admin/debug endpoints (e.g., /bull/*)
            - Performance issues on background jobs with `events_24h < 100`
            - Performance issues where the slow span is < 2s on a background job
            - Issues where the slow span is an external API call we can't control
            - Issues that need more data to evaluate
            - Performance issues where multiple related Sentry issues point to the same code path — group them mentally and only create ONE issue for the root cause, not one per symptom

            ### AUTO-SKIP — BullMQ Redis ops misclassified as N+1
            Sentry's N+1 detector occasionally flags BullMQ's normal Lua-script Redis operations as N+1 query patterns. These are NOT real N+1 bugs and should be auto-resolved.

            **Detection — auto-resolve in Sentry without creating a GH issue if ALL of:**
            1. Issue type is `N+1 Query` (or `issue.category:performance` with N+1 fingerprint)
            2. The repeated span is a Redis op — `db.redis`, `cache.get`, or the span description contains `evalsha` / `evalSha`
            3. The span description references a BullMQ key — matches `bull:`, `bullmq:`, or one of the BullMQ Lua script SHAs (`840cf612b9e4155aeb79853f3502814792769274` and similar 40-char hex SHAs paired with `bull:` keys)

            Resolve these in Sentry with a short comment ("BullMQ Redis ops are not N+1 — scanner filter") and skip creating a GitHub issue. Issue #1221 was a recent example.

            ## Step 5: Create GitHub issues

            For **error** issues:
            ```bash
            gh issue create \
              --title "Sentry: [error title]" \
              --label "sentry" \
              --label "[backend|frontend]" \
              --body "## Sentry Error

            **Project:** [ethernal-backend|ethernal-frontend]
            **Level:** [error|warning]
            **Events (24h):** [events_24h]   ← rolling, used for thresholds
            **Lifetime events:** [lifetime_count]
            **Regression:** [Yes/No]
            **Link:** https://sentry.io/organizations/antoine-0l/issues/[ID]/

            ### Error
            \`\`\`
            [error message and key stack trace frames]
            \`\`\`

            ### Context
            [any relevant tags, transaction name, or URL]

            ---
            *Created by Sentry Scanner*"
            ```

            For **performance** issues:
            ```bash
            gh issue create \
              --title "Perf: [concise description]" \
              --label "sentry" \
              --label "performance" \
              --label "[backend|frontend]" \
              --body "## Performance Issue

            **Project:** [ethernal-backend|ethernal-frontend]
            **Type:** [N+1 Query | Slow DB | Slow Transaction | Regression | ...]
            **Impact (24h):** [events_24h] events   ← rolling, used for thresholds
            **Lifetime events:** [lifetime_count]
            **Regression:** [Yes/No]
            **Transaction:** \`[transaction name]\`
            **Link:** https://sentry.io/organizations/antoine-0l/issues/[ID]/

            ### Problem
            [Clear description of the bottleneck — what's slow and why]

            ### Suggested Fix
            [Concrete suggestion: add eager loading, use Promise.all, add index, batch queries, etc.]

            ### Evidence
            \`\`\`
            [Key spans, query patterns, or timing data]
            \`\`\`

            ---
            *Created by Sentry Scanner*"
            ```

            For regressions, prefix the title with "(regression)" e.g. `Sentry (regression): [title]` or `Perf (regression): [title]`.

            **IMPORTANT: Stagger issue creation.** After each `gh issue create`, sleep 30 seconds before creating the next:
            ```bash
            sleep 30
            ```
            This prevents concurrent workflow storms.

            **Limit to 3 issues per scan.** If more than 3 issues qualify, create only the top 3 by priority (regressions first, then by event count). The rest will be picked up in the next hourly scan.

            ## Step 6: Notify dashboard

            For each issue created, notify the dashboard webhook:
            ```bash
            curl -s -X POST "$APP_URL/webhooks/github-actions" \
              -H "Authorization: Bearer $ETHERNAL_WEBHOOK_SECRET" \
              -H "Content-Type: application/json" \
              -d "{
                \"githubIssueNumber\": ISSUE_NUMBER,
                \"sentryIssueId\": \"SENTRY_ID\",
                \"sentryProject\": \"PROJECT\",
                \"sentryTitle\": \"TITLE\",
                \"sentryLevel\": \"LEVEL\",
                \"sentryEventCount\": EVENTS_24H,
                \"sentryLink\": \"https://sentry.io/organizations/antoine-0l/issues/SENTRY_ID/\",
                \"status\": \"discovered\",
                \"currentStep\": \"Discovered by scanner\"
              }"
            ```

            ## Step 7: Print summary

            At the end, print a summary:
            ```
            === Sentry Scanner Summary ===
            Errors scanned: X
            Performance issues scanned: Y
            Regressions found: Z
            GitHub issues created: A (B errors, C performance)
            Auto-resolved: D
            Skipped (already tracked): E
            Skipped (not actionable): F
            ```

            ## Rules
            - NEVER create duplicate GitHub issues — always check first
            - **NEVER create your own "Sentry service down" or "Sentry API failing" issue from Steps 1-7.** Those are infra issues, not Sentry-issue findings, and they're owned by Step 0 (preflight). If you find yourself wanting to file one mid-scan, abort and re-run Step 0 instead. Past scanner runs created #1301/#1302/#1303 + #1305 for the same outage because this rule wasn't enforced.
            - Be conservative: only create issues for clear code bugs or significant performance problems
            - When resolving in Sentry, always include a reason
            - Keep issue descriptions concise but include enough for the auto-fix agent
            - For N+1 queries, always identify the model/relation involved
            - For slow queries, suggest specific indexes or query optimizations
            - Regressions are ALWAYS high priority — create issues for them even with low event counts
        env:
          SENTRY_API_TOKEN: ${{ secrets.SENTRY_API_TOKEN }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          APP_URL: ${{ secrets.APP_URL }}
          ETHERNAL_WEBHOOK_SECRET: ${{ secrets.ETHERNAL_WEBHOOK_SECRET }}