Sentry Proactive Scanner #3316
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Sentry Proactive Scanner | |
| on: | |
| schedule: | |
| - cron: '0 * * * *' | |
| workflow_dispatch: | |
| permissions: | |
| contents: read | |
| issues: write | |
| id-token: write | |
| jobs: | |
| scan: | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 25 | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v4 | |
| with: | |
| sparse-checkout: | | |
| CLAUDE.md | |
| run/ | |
| .github/ | |
| - name: Setup SSH | |
| uses: webfactory/ssh-agent@v0.9.0 | |
| with: | |
| ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} | |
| - name: Add known hosts | |
| run: | | |
| mkdir -p ~/.ssh | |
| ssh-keyscan -H 157.90.154.200 >> ~/.ssh/known_hosts | |
| - name: Claude Code - Scan | |
| uses: anthropics/claude-code-action@v1 | |
| with: | |
| claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} | |
| claude_args: "--model claude-sonnet-4-20250514 --max-turns 60 --allowedTools Bash,Read,Glob,Grep" | |
| show_full_output: true | |
| prompt: | | |
| You are a proactive Sentry scanner for the Ethernal project. Your job is to scan ALL unresolved Sentry issues (errors AND performance) and decide which ones need GitHub issues created. | |
| Sentry base URL: https://sentry.io | |
| Org: sentry | |
| Backend project: ethernal-backend (id=2) | |
| Frontend project: ethernal-frontend (id=3) | |
| **IMPORTANT API NOTES for this Sentry instance (v26.2.1)**: | |
| - Only `statsPeriod` values of `24h` and `14d` work. Do NOT use `1h`. | |
| - Always save curl output to a temp file first, then process with jq. Do NOT pipe curl directly to jq — it fails intermittently. | |
| - Pattern: `curl -s -o /tmp/resp.json "URL" -H "Authorization: Bearer $SENTRY_API_TOKEN" && jq '...' /tmp/resp.json` | |
| - **CRITICAL — `count` is LIFETIME, not 24h.** The `count` field on each issue in the list response is the lifetime `times_seen` total, NOT the 24h rolling volume. Do NOT use it for thresholds or report it as "Events (24h)" — that caused a false-incident storm (issues #1218–#1232 were all closed as self-resolved because the scanner reported 6,000+ "24h events" when actual 24h volume was 2–16). The TRUE 24h volume is `([.stats."24h"[]?[1]] | add)` — sum of the hourly buckets in the `stats."24h"` array. Always compute and use this value. | |
| ## Step 0: Pre-flight health & auth probe (MANDATORY) | |
| Before doing anything else, distinguish three top-level states: | |
| - **Service down** (Sentry API itself returns 5xx) | |
| - **Auth broken** (Sentry API up, but token is rejected with 401/403) | |
| - **Healthy** (proceed to Step 1) | |
| Past incidents created **separate issues per scanner run for the same outage** (#1301/#1302/#1303 + #1305 were all the same Sentry outage, and #1304 was the actual token-regen item that got buried). This step deduplicates against open infra issues and **must run before any Sentry query**. | |
| ```bash | |
| BASE_UNAUTH=$(curl -s -o /tmp/preflight_base.json -w "%{http_code}" "https://sentry.io/api/0/" --max-time 15 || echo "000") | |
| AUTH_PROBE=$(curl -s -o /tmp/preflight_auth.json -w "%{http_code}" "https://sentry.io/api/0/projects/" -H "Authorization: Bearer $SENTRY_API_TOKEN" --max-time 15 || echo "000") | |
| echo "preflight base=$BASE_UNAUTH auth=$AUTH_PROBE" | |
| ``` | |
| Decision matrix: | |
| | base | auth | Verdict | Action | | |
| |------|------|---------|--------| | |
| | 2xx | 2xx | Healthy | Continue to Step 1 | | |
| | 2xx | 401/403 | Token bad | Run "AUTH_BROKEN handler" below, then exit 0 | | |
| | 5xx / 000 | any | Service down | Run "SERVICE_DOWN handler" below, then exit 0 | | |
| | other | other | Unknown | Run "SERVICE_DOWN handler" with verdict="unknown", exit 0 | | |
| ### SERVICE_DOWN handler | |
| Dedup against an open service-down issue from the last 24h, then either comment on it or create one. **Never create a second open service-down issue.** | |
| 1. Search for an existing open issue: | |
| ```bash | |
| gh issue list --state open --label sentry --search 'in:title "Sentry monitoring service down"' \ | |
| --json number,createdAt \ | |
| --jq '[.[] | select((now - (.createdAt | fromdateiso8601)) < 86400)] | .[0].number' | |
| ``` | |
| 2. If a number comes back: comment on it with `gh issue comment <num>` noting the current timestamp, the HTTP codes from the preflight, and "outage still in progress". Then `exit 0`. | |
| 3. If no existing issue: `gh issue create` with: | |
| - title: `Sentry monitoring service down - API returning <BASE_UNAUTH> errors` | |
| - labels: `sentry`, `needs-human`, `infra-alert` | |
| - body: the timestamp, both HTTP codes, the first 500 chars of `/tmp/preflight_base.json` if it has content, and "Scanner skipped this cycle — will retry on next hourly run." | |
| Then `exit 0`. | |
| ### AUTH_BROKEN handler | |
| Dedup against an open auth-issue from the last 24h. Title pattern: `Sentry API scanner failing - authentication`. | |
| 1. Search for an existing open issue: | |
| ```bash | |
| gh issue list --state open --label sentry --search 'in:title "Sentry API scanner failing"' \ | |
| --json number,createdAt \ | |
| --jq '[.[] | select((now - (.createdAt | fromdateiso8601)) < 86400)] | .[0].number' | |
| ``` | |
| 2. If a number comes back: comment on it noting the current timestamp and that the authenticated API still returns HTTP `<AUTH_PROBE>`. Then `exit 0`. | |
| 3. If no existing issue: `gh issue create` with: | |
| - title: `Sentry API scanner failing - authentication/server errors` | |
| - labels: `sentry`, `needs-human` | |
| - body: timestamps, both HTTP codes, and these manual action items: | |
| 1. Log into https://sentry.io → Settings → Account → API → Auth Tokens | |
| 2. Regenerate the token with scopes: `org:read`, `project:read`, `event:read`, `event:admin` | |
| 3. Update the `SENTRY_API_TOKEN` GitHub Actions secret | |
| 4. Re-run this workflow manually | |
| Then `exit 0`. | |
| Only proceed past Step 0 if the preflight verdict is "Healthy". | |
| ## Step 1: Query Sentry for all unresolved issues | |
| Run ALL 6 queries in a single bash command to save turns. Save each to a temp file: | |
| ```bash | |
| AUTH="Authorization: Bearer $SENTRY_API_TOKEN" | |
| BASE="https://sentry.io/api/0/projects/antoine-0l" | |
| curl -s -o /tmp/be_err.json "$BASE/ethernal-backend/issues/?query=is:unresolved+issue.category:error&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/be_perf.json "$BASE/ethernal-backend/issues/?query=is:unresolved+issue.category:performance&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/be_reg.json "$BASE/ethernal-backend/issues/?query=is:regressed&statsPeriod=24h&limit=50" -H "$AUTH" | |
| curl -s -o /tmp/fe_err.json "$BASE/ethernal-frontend/issues/?query=is:unresolved+issue.category:error&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/fe_perf.json "$BASE/ethernal-frontend/issues/?query=is:unresolved+issue.category:performance&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/fe_reg.json "$BASE/ethernal-frontend/issues/?query=is:regressed&statsPeriod=24h&limit=50" -H "$AUTH" | |
| # ALWAYS project events_24h (rolling) — never use raw `count` (lifetime) for decisions or reports. | |
| PROJ='[.[] | {id, title, lifetime_count: (.count | tonumber), events_24h: ([.stats."24h"[]?[1]] | add // 0), lastSeen, shortId, isRegression: (.isRegression // false)}]' | |
| echo "=== Backend Errors ===" && jq "$PROJ" /tmp/be_err.json | |
| echo "=== Backend Performance ===" && jq "$PROJ" /tmp/be_perf.json | |
| echo "=== Backend Regressed ===" && jq "$PROJ" /tmp/be_reg.json | |
| echo "=== Frontend Errors ===" && jq "$PROJ" /tmp/fe_err.json | |
| echo "=== Frontend Performance ===" && jq "$PROJ" /tmp/fe_perf.json | |
| echo "=== Frontend Regressed ===" && jq "$PROJ" /tmp/fe_reg.json | |
| ``` | |
| From this point on, **always use `events_24h` (computed above) for thresholds and reporting**. The `lifetime_count` is informational only — useful as context ("issue has fired 6,953 times total but only 2 in last 24h") but never a decision input. | |
| Deduplicate by `(project, id)` — regressed issues may appear in both the error/performance and regressed queries. | |
| ## Step 2: Filter already-tracked issues | |
| In the SAME bash call, also fetch existing GitHub sentry issues: | |
| ```bash | |
| gh issue list --label sentry --state all --limit 200 --json number,title,body -q '.[].body' > /tmp/gh_issues.txt | |
| ``` | |
| For each Sentry issue, check: `grep -c "issues/SENTRY_ID/" /tmp/gh_issues.txt` | |
| Skip any issue that already has a GitHub issue (open or closed). | |
| ## Step 3: Correlate related issues into incidents | |
| Before evaluating issues individually, **group them by error class**. Issues that share the same exception type (e.g., `SequelizeConnectionAcquireTimeoutError`, `SequelizeDatabaseError: query_wait_timeout`) AND have `lastSeen` within 30 minutes of each other are symptoms of a single incident, not independent bugs. | |
| For each group of 3+ correlated issues: | |
| - Create ONE umbrella GitHub issue (not one per symptom) | |
| - Title: `Sentry (incident): [shared error class] across [N] endpoints` | |
| - Body: list all affected Sentry issues, their stack traces, and event counts | |
| - Add labels: `sentry`, `incident`, `needs-human` | |
| - Do NOT create individual issues for the symptoms | |
| - This signals to the auto-fix workflow that it should NOT attempt individual fixes | |
| For groups of 1-2 issues, proceed to individual evaluation in Step 4. | |
| ## Step 4: Evaluate each new issue | |
| For each new issue that passes the filter AND was not grouped into an incident above, fetch event context. Use temp files: | |
| ```bash | |
| curl -s -o /tmp/event.json "https://sentry.io/api/0/issues/{id}/events/latest/" -H "Authorization: Bearer $SENTRY_API_TOKEN" | |
| jq '{message: .message, tags: [.tags[]? | select(.key == "transaction" or .key == "url") | {key, value}], exception: .entries[0]?.data.values[0]?.stacktrace.frames[-3:]?}' /tmp/event.json | |
| ``` | |
| Then categorize into ONE of: | |
| ### AUTO-SKIP + RESOLVE in Sentry | |
| - **Any issue with `events_24h == 0`** (regardless of lifetime count) — it has self-resolved, mark resolved in Sentry | |
| - Connection/transient errors (SequelizeConnectionError, ECONNRESET, "Connection terminated unexpectedly") — UNLESS `events_24h >= 30` | |
| - Rate limiting errors | |
| - Expected validation errors (user input) | |
| - Third-party service errors we can't control | |
| - Low-impact edge cases (deprecated browser, obscure user input) | |
| - Performance issues on endpoints that no longer exist | |
| - Performance issues with `events_24h == 0` that are stale | |
| To resolve: `curl -s -X PUT "https://sentry.io/api/0/issues/{id}/" -H "Authorization: Bearer $SENTRY_API_TOKEN" -H "Content-Type: application/json" -d '{"status": "resolved"}'` | |
| ### CREATE GITHUB ISSUE (prioritized) — all event counts below are `events_24h`, NOT lifetime | |
| **Priority 1 — Regressions** (always create, regardless of event count): | |
| - Any issue where `isRegression: true` AND `events_24h >= 1` — a previous fix didn't hold AND it's still firing | |
| - Regressions with `events_24h == 0` go to AUTO-SKIP+RESOLVE — the regression flag is stale | |
| - Title prefix: "Sentry (regression):" for errors, "Perf (regression):" for performance | |
| **Priority 2 — High-impact errors**: | |
| - Null/type errors, unhandled promise rejections with `events_24h >= 2` | |
| - Systematic issues with `events_24h >= 5` | |
| **Priority 3 — High-impact performance** (user-facing hot paths only): | |
| - N+1 queries with `events_24h >= 50` AND on a user-facing endpoint or blockSync hot path | |
| - Slow DB queries with `events_24h >= 50` AND p95 > 2s | |
| **Priority 4 — Medium performance**: | |
| - N+1 queries with `events_24h` 20–49 on user-facing endpoints | |
| - Background job performance issues with `events_24h >= 100` (higher bar since they don't affect UX) | |
| ### SKIP (leave unresolved in Sentry) | |
| - Non-regressed issues with `events_24h < 2` (might be one-off) | |
| - Performance issues on admin/debug endpoints (e.g., /bull/*) | |
| - Performance issues on background jobs with `events_24h < 100` | |
| - Performance issues where the slow span is < 2s on a background job | |
| - Issues where the slow span is an external API call we can't control | |
| - Issues that need more data to evaluate | |
| - Performance issues where multiple related Sentry issues point to the same code path — group them mentally and only create ONE issue for the root cause, not one per symptom | |
| ### AUTO-SKIP — BullMQ Redis ops misclassified as N+1 | |
| Sentry's N+1 detector occasionally flags BullMQ's normal Lua-script Redis operations as N+1 query patterns. These are NOT real N+1 bugs and should be auto-resolved. | |
| **Detection — auto-resolve in Sentry without creating a GH issue if ALL of:** | |
| 1. Issue type is `N+1 Query` (or `issue.category:performance` with N+1 fingerprint) | |
| 2. The repeated span is a Redis op — `db.redis`, `cache.get`, or the span description contains `evalsha` / `evalSha` | |
| 3. The span description references a BullMQ key — matches `bull:`, `bullmq:`, or one of the BullMQ Lua script SHAs (`840cf612b9e4155aeb79853f3502814792769274` and similar 40-char hex SHAs paired with `bull:` keys) | |
| Resolve these in Sentry with a short comment ("BullMQ Redis ops are not N+1 — scanner filter") and skip creating a GitHub issue. Issue #1221 was a recent example. | |
| ## Step 5: Create GitHub issues | |
| For **error** issues: | |
| ```bash | |
| gh issue create \ | |
| --title "Sentry: [error title]" \ | |
| --label "sentry" \ | |
| --label "[backend|frontend]" \ | |
| --body "## Sentry Error | |
| **Project:** [ethernal-backend|ethernal-frontend] | |
| **Level:** [error|warning] | |
| **Events (24h):** [events_24h] ← rolling, used for thresholds | |
| **Lifetime events:** [lifetime_count] | |
| **Regression:** [Yes/No] | |
| **Link:** https://sentry.io/organizations/antoine-0l/issues/[ID]/ | |
| ### Error | |
| \`\`\` | |
| [error message and key stack trace frames] | |
| \`\`\` | |
| ### Context | |
| [any relevant tags, transaction name, or URL] | |
| --- | |
| *Created by Sentry Scanner*" | |
| ``` | |
| For **performance** issues: | |
| ```bash | |
| gh issue create \ | |
| --title "Perf: [concise description]" \ | |
| --label "sentry" \ | |
| --label "performance" \ | |
| --label "[backend|frontend]" \ | |
| --body "## Performance Issue | |
| **Project:** [ethernal-backend|ethernal-frontend] | |
| **Type:** [N+1 Query | Slow DB | Slow Transaction | Regression | ...] | |
| **Impact (24h):** [events_24h] events ← rolling, used for thresholds | |
| **Lifetime events:** [lifetime_count] | |
| **Regression:** [Yes/No] | |
| **Transaction:** \`[transaction name]\` | |
| **Link:** https://sentry.io/organizations/antoine-0l/issues/[ID]/ | |
| ### Problem | |
| [Clear description of the bottleneck — what's slow and why] | |
| ### Suggested Fix | |
| [Concrete suggestion: add eager loading, use Promise.all, add index, batch queries, etc.] | |
| ### Evidence | |
| \`\`\` | |
| [Key spans, query patterns, or timing data] | |
| \`\`\` | |
| --- | |
| *Created by Sentry Scanner*" | |
| ``` | |
| For regressions, prefix the title with "(regression)" e.g. `Sentry (regression): [title]` or `Perf (regression): [title]`. | |
| **IMPORTANT: Stagger issue creation.** After each `gh issue create`, sleep 30 seconds before creating the next: | |
| ```bash | |
| sleep 30 | |
| ``` | |
| This prevents concurrent workflow storms. | |
| **Limit to 3 issues per scan.** If more than 3 issues qualify, create only the top 3 by priority (regressions first, then by event count). The rest will be picked up in the next hourly scan. | |
| ## Step 6: Notify dashboard | |
| For each issue created, notify the dashboard webhook: | |
| ```bash | |
| curl -s -X POST "$APP_URL/webhooks/github-actions" \ | |
| -H "Authorization: Bearer $ETHERNAL_WEBHOOK_SECRET" \ | |
| -H "Content-Type: application/json" \ | |
| -d "{ | |
| \"githubIssueNumber\": ISSUE_NUMBER, | |
| \"sentryIssueId\": \"SENTRY_ID\", | |
| \"sentryProject\": \"PROJECT\", | |
| \"sentryTitle\": \"TITLE\", | |
| \"sentryLevel\": \"LEVEL\", | |
| \"sentryEventCount\": EVENTS_24H, | |
| \"sentryLink\": \"https://sentry.io/organizations/antoine-0l/issues/SENTRY_ID/\", | |
| \"status\": \"discovered\", | |
| \"currentStep\": \"Discovered by scanner\" | |
| }" | |
| ``` | |
| ## Step 7: Print summary | |
| At the end, print a summary: | |
| ``` | |
| === Sentry Scanner Summary === | |
| Errors scanned: X | |
| Performance issues scanned: Y | |
| Regressions found: Z | |
| GitHub issues created: A (B errors, C performance) | |
| Auto-resolved: D | |
| Skipped (already tracked): E | |
| Skipped (not actionable): F | |
| ``` | |
| ## Rules | |
| - NEVER create duplicate GitHub issues — always check first | |
| - **NEVER create your own "Sentry service down" or "Sentry API failing" issue from Steps 1-7.** Those are infra issues, not Sentry-issue findings, and they're owned by Step 0 (preflight). If you find yourself wanting to file one mid-scan, abort and re-run Step 0 instead. Past scanner runs created #1301/#1302/#1303 + #1305 for the same outage because this rule wasn't enforced. | |
| - Be conservative: only create issues for clear code bugs or significant performance problems | |
| - When resolving in Sentry, always include a reason | |
| - Keep issue descriptions concise but include enough for the auto-fix agent | |
| - For N+1 queries, always identify the model/relation involved | |
| - For slow queries, suggest specific indexes or query optimizations | |
| - Regressions are ALWAYS high priority — create issues for them even with low event counts | |
| env: | |
| SENTRY_API_TOKEN: ${{ secrets.SENTRY_API_TOKEN }} | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| APP_URL: ${{ secrets.APP_URL }} | |
| ETHERNAL_WEBHOOK_SECRET: ${{ secrets.ETHERNAL_WEBHOOK_SECRET }} |