Sentry Proactive Scanner #946
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Sentry Proactive Scanner | |
| on: | |
| schedule: | |
| - cron: '0 * * * *' | |
| workflow_dispatch: | |
| permissions: | |
| contents: read | |
| issues: write | |
| id-token: write | |
| jobs: | |
| scan: | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 25 | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v4 | |
| with: | |
| sparse-checkout: | | |
| CLAUDE.md | |
| run/ | |
| .github/ | |
| - name: Setup SSH | |
| uses: webfactory/ssh-agent@v0.9.0 | |
| with: | |
| ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} | |
| - name: Add known hosts | |
| run: | | |
| mkdir -p ~/.ssh | |
| ssh-keyscan -H 157.90.154.200 >> ~/.ssh/known_hosts | |
| - name: Claude Code - Scan | |
| uses: anthropics/claude-code-action@v1 | |
| with: | |
| claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} | |
| claude_args: "--model claude-sonnet-4-20250514 --max-turns 60 --allowedTools Bash,Read,Glob,Grep" | |
| show_full_output: true | |
| prompt: | | |
| You are a proactive Sentry scanner for the Ethernal project. Your job is to scan ALL unresolved Sentry issues (errors AND performance) and decide which ones need GitHub issues created. | |
| Sentry base URL: https://sentry.tryethernal.com | |
| Org: sentry | |
| Backend project: ethernal-backend (id=2) | |
| Frontend project: ethernal-frontend (id=3) | |
| **IMPORTANT API NOTES for this Sentry instance (v26.2.1)**: | |
| - Only `statsPeriod` values of `24h` and `14d` work. Do NOT use `1h`. | |
| - Always save curl output to a temp file first, then process with jq. Do NOT pipe curl directly to jq — it fails intermittently. | |
| - Pattern: `curl -s -o /tmp/resp.json "URL" -H "Authorization: Bearer $SENTRY_API_TOKEN" && jq '...' /tmp/resp.json` | |
| ## Step 1: Query Sentry for all unresolved issues | |
| Run ALL 6 queries in a single bash command to save turns. Save each to a temp file: | |
| ```bash | |
| AUTH="Authorization: Bearer $SENTRY_API_TOKEN" | |
| BASE="https://sentry.tryethernal.com/api/0/projects/sentry" | |
| curl -s -o /tmp/be_err.json "$BASE/ethernal-backend/issues/?query=is:unresolved+issue.category:error&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/be_perf.json "$BASE/ethernal-backend/issues/?query=is:unresolved+issue.category:performance&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/be_reg.json "$BASE/ethernal-backend/issues/?query=is:regressed&limit=50" -H "$AUTH" | |
| curl -s -o /tmp/fe_err.json "$BASE/ethernal-frontend/issues/?query=is:unresolved+issue.category:error&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/fe_perf.json "$BASE/ethernal-frontend/issues/?query=is:unresolved+issue.category:performance&statsPeriod=24h&limit=100" -H "$AUTH" | |
| curl -s -o /tmp/fe_reg.json "$BASE/ethernal-frontend/issues/?query=is:regressed&limit=50" -H "$AUTH" | |
| echo "=== Backend Errors ===" && jq '[.[] | {id, title, count, lastSeen, shortId}]' /tmp/be_err.json | |
| echo "=== Backend Performance ===" && jq '[.[] | {id, title, count, lastSeen, shortId}]' /tmp/be_perf.json | |
| echo "=== Backend Regressed ===" && jq '[.[] | {id, title, count, lastSeen, shortId}]' /tmp/be_reg.json | |
| echo "=== Frontend Errors ===" && jq '[.[] | {id, title, count, lastSeen, shortId}]' /tmp/fe_err.json | |
| echo "=== Frontend Performance ===" && jq '[.[] | {id, title, count, lastSeen, shortId}]' /tmp/fe_perf.json | |
| echo "=== Frontend Regressed ===" && jq '[.[] | {id, title, count, lastSeen, shortId}]' /tmp/fe_reg.json | |
| ``` | |
| Deduplicate by `(project, id)` — regressed issues may appear in both the error/performance and regressed queries. | |
| ## Step 2: Filter already-tracked issues | |
| In the SAME bash call, also fetch existing GitHub sentry issues: | |
| ```bash | |
| gh issue list --label sentry --state all --limit 200 --json number,title,body -q '.[].body' > /tmp/gh_issues.txt | |
| ``` | |
| For each Sentry issue, check: `grep -c "issues/SENTRY_ID/" /tmp/gh_issues.txt` | |
| Skip any issue that already has a GitHub issue (open or closed). | |
| ## Step 3: Correlate related issues into incidents | |
| Before evaluating issues individually, **group them by error class**. Issues that share the same exception type (e.g., `SequelizeConnectionAcquireTimeoutError`, `SequelizeDatabaseError: query_wait_timeout`) AND have `lastSeen` within 30 minutes of each other are symptoms of a single incident, not independent bugs. | |
| For each group of 3+ correlated issues: | |
| - Create ONE umbrella GitHub issue (not one per symptom) | |
| - Title: `Sentry (incident): [shared error class] across [N] endpoints` | |
| - Body: list all affected Sentry issues, their stack traces, and event counts | |
| - Add labels: `sentry`, `incident`, `needs-human` | |
| - Do NOT create individual issues for the symptoms | |
| - This signals to the auto-fix workflow that it should NOT attempt individual fixes | |
| For groups of 1-2 issues, proceed to individual evaluation in Step 4. | |
| ## Step 4: Evaluate each new issue | |
| For each new issue that passes the filter AND was not grouped into an incident above, fetch event context. Use temp files: | |
| ```bash | |
| curl -s -o /tmp/event.json "https://sentry.tryethernal.com/api/0/issues/{id}/events/latest/" -H "Authorization: Bearer $SENTRY_API_TOKEN" | |
| jq '{message: .message, tags: [.tags[]? | select(.key == "transaction" or .key == "url") | {key, value}], exception: .entries[0]?.data.values[0]?.stacktrace.frames[-3:]?}' /tmp/event.json | |
| ``` | |
| Then categorize into ONE of: | |
| ### AUTO-SKIP + RESOLVE in Sentry | |
| - Connection/transient errors (SequelizeConnectionError, ECONNRESET, "Connection terminated unexpectedly") — UNLESS 30+ events in 24h | |
| - Rate limiting errors | |
| - Expected validation errors (user input) | |
| - Third-party service errors we can't control | |
| - Low-impact edge cases (deprecated browser, obscure user input) | |
| - Performance issues on endpoints that no longer exist | |
| - Performance issues with 0 events in the last 24h that are stale | |
| To resolve: `curl -s -X PUT "https://sentry.tryethernal.com/api/0/issues/{id}/" -H "Authorization: Bearer $SENTRY_API_TOKEN" -H "Content-Type: application/json" -d '{"status": "resolved"}'` | |
| ### CREATE GITHUB ISSUE (prioritized) | |
| **Priority 1 — Regressions** (always create, regardless of event count): | |
| - Any issue where `isRegression: true` — a previous fix didn't hold | |
| - Title prefix: "Sentry (regression):" for errors, "Perf (regression):" for performance | |
| **Priority 2 — High-impact errors**: | |
| - Null/type errors, unhandled promise rejections with 2+ events | |
| - Systematic issues with 5+ events in 24h | |
| **Priority 3 — High-impact performance** (user-facing hot paths only): | |
| - N+1 queries with 50+ events AND on a user-facing endpoint or blockSync hot path | |
| - Slow DB queries with 50+ events AND p95 > 2s | |
| **Priority 4 — Medium performance**: | |
| - N+1 queries with 20-49 events on user-facing endpoints | |
| - Background job performance issues with 100+ events (higher bar since they don't affect UX) | |
| ### SKIP (leave unresolved in Sentry) | |
| - Non-regressed issues with fewer than 2 events (might be one-off) | |
| - Performance issues on admin/debug endpoints (e.g., /bull/*) | |
| - Performance issues on background jobs with < 100 events | |
| - Performance issues where the slow span is < 2s on a background job | |
| - Issues where the slow span is an external API call we can't control | |
| - Issues that need more data to evaluate | |
| - Performance issues where multiple related Sentry issues point to the same code path — group them mentally and only create ONE issue for the root cause, not one per symptom | |
| ## Step 5: Create GitHub issues | |
| For **error** issues: | |
| ```bash | |
| gh issue create \ | |
| --title "Sentry: [error title]" \ | |
| --label "sentry" \ | |
| --label "[backend|frontend]" \ | |
| --body "## Sentry Error | |
| **Project:** [ethernal-backend|ethernal-frontend] | |
| **Level:** [error|warning] | |
| **Events (24h):** [count] | |
| **Regression:** [Yes/No] | |
| **Link:** https://sentry.tryethernal.com/organizations/sentry/issues/[ID]/ | |
| ### Error | |
| \`\`\` | |
| [error message and key stack trace frames] | |
| \`\`\` | |
| ### Context | |
| [any relevant tags, transaction name, or URL] | |
| --- | |
| *Created by Sentry Scanner*" | |
| ``` | |
| For **performance** issues: | |
| ```bash | |
| gh issue create \ | |
| --title "Perf: [concise description]" \ | |
| --label "sentry" \ | |
| --label "performance" \ | |
| --label "[backend|frontend]" \ | |
| --body "## Performance Issue | |
| **Project:** [ethernal-backend|ethernal-frontend] | |
| **Type:** [N+1 Query | Slow DB | Slow Transaction | Regression | ...] | |
| **Impact:** [events in 24h] events | |
| **Regression:** [Yes/No] | |
| **Transaction:** \`[transaction name]\` | |
| **Link:** https://sentry.tryethernal.com/organizations/sentry/issues/[ID]/ | |
| ### Problem | |
| [Clear description of the bottleneck — what's slow and why] | |
| ### Suggested Fix | |
| [Concrete suggestion: add eager loading, use Promise.all, add index, batch queries, etc.] | |
| ### Evidence | |
| \`\`\` | |
| [Key spans, query patterns, or timing data] | |
| \`\`\` | |
| --- | |
| *Created by Sentry Scanner*" | |
| ``` | |
| For regressions, prefix the title with "(regression)" e.g. `Sentry (regression): [title]` or `Perf (regression): [title]`. | |
| **IMPORTANT: Stagger issue creation.** After each `gh issue create`, sleep 30 seconds before creating the next: | |
| ```bash | |
| sleep 30 | |
| ``` | |
| This prevents concurrent workflow storms. | |
| **Limit to 3 issues per scan.** If more than 3 issues qualify, create only the top 3 by priority (regressions first, then by event count). The rest will be picked up in the next hourly scan. | |
| ## Step 6: Notify dashboard | |
| For each issue created, notify the dashboard webhook: | |
| ```bash | |
| curl -s -X POST "$APP_URL/webhooks/github-actions" \ | |
| -H "Authorization: Bearer $ETHERNAL_WEBHOOK_SECRET" \ | |
| -H "Content-Type: application/json" \ | |
| -d "{ | |
| \"githubIssueNumber\": ISSUE_NUMBER, | |
| \"sentryIssueId\": \"SENTRY_ID\", | |
| \"sentryProject\": \"PROJECT\", | |
| \"sentryTitle\": \"TITLE\", | |
| \"sentryLevel\": \"LEVEL\", | |
| \"sentryEventCount\": COUNT, | |
| \"sentryLink\": \"https://sentry.tryethernal.com/organizations/sentry/issues/SENTRY_ID/\", | |
| \"status\": \"discovered\", | |
| \"currentStep\": \"Discovered by scanner\" | |
| }" | |
| ``` | |
| ## Step 7: Print summary | |
| At the end, print a summary: | |
| ``` | |
| === Sentry Scanner Summary === | |
| Errors scanned: X | |
| Performance issues scanned: Y | |
| Regressions found: Z | |
| GitHub issues created: A (B errors, C performance) | |
| Auto-resolved: D | |
| Skipped (already tracked): E | |
| Skipped (not actionable): F | |
| ``` | |
| ## Rules | |
| - NEVER create duplicate GitHub issues — always check first | |
| - Be conservative: only create issues for clear code bugs or significant performance problems | |
| - When resolving in Sentry, always include a reason | |
| - Keep issue descriptions concise but include enough for the auto-fix agent | |
| - For N+1 queries, always identify the model/relation involved | |
| - For slow queries, suggest specific indexes or query optimizations | |
| - Regressions are ALWAYS high priority — create issues for them even with low event counts | |
| env: | |
| SENTRY_API_TOKEN: ${{ secrets.SENTRY_API_TOKEN }} | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| APP_URL: ${{ secrets.APP_URL }} | |
| ETHERNAL_WEBHOOK_SECRET: ${{ secrets.ETHERNAL_WEBHOOK_SECRET }} |