Skip to content

Commit 8ea9a9b

Browse files
joseph-isaacsjoseph-isaacs
committed
feat: automated fuzzing issue creation with Claude analysis
Signed-off-by: Joe Isaacs <[email protected]>
1 parent 0e12233 commit 8ea9a9b

File tree

1 file changed

+180
-134
lines changed

1 file changed

+180
-134
lines changed

‎.github/workflows/fuzz.yml‎

Lines changed: 180 additions & 134 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ jobs:
2020
crashes_found: ${{ steps.check.outputs.crashes_found }}
2121
crash_count: ${{ steps.check.outputs.crash_count }}
2222
first_crash_name: ${{ steps.check.outputs.first_crash_name }}
23+
artifact_url: ${{ steps.upload_artifacts.outputs.artifact-url }}
2324
steps:
2425
- uses: runs-on/action@v2
2526
with:
@@ -75,6 +76,7 @@ jobs:
7576
echo "No crashes found"
7677
fi
7778
- name: Archive crash artifacts
79+
id: upload_artifacts
7880
if: steps.check.outputs.crashes_found == 'true'
7981
uses: actions/upload-artifact@v4
8082
with:
@@ -110,152 +112,196 @@ jobs:
110112
permissions:
111113
issues: write
112114
contents: read
115+
id-token: write # Required for Claude Code Action OIDC authentication
116+
pull-requests: read # For gh api commands
113117
steps:
114118
- name: Checkout repository
115119
uses: actions/checkout@v5
116120

117-
- name: Download crash artifacts
118-
uses: actions/download-artifact@v4
119-
with:
120-
name: io-fuzzing-crash-artifacts
121-
path: ./crash_artifacts
122-
123121
- name: Download fuzzer logs
124122
uses: actions/download-artifact@v4
125123
with:
126124
name: io-fuzzing-logs
127125
path: ./logs
128126

129-
- name: Extract first crash only
130-
run: |
131-
# Only keep the first crash file for analysis
132-
FIRST_CRASH="${{ needs.io_fuzz.outputs.first_crash_name }}"
133-
echo "Processing only first crash: $FIRST_CRASH"
134-
135-
# Create a clean directory with just the first crash
136-
mkdir -p ./first_crash
137-
if [ -f "crash_artifacts/$FIRST_CRASH" ]; then
138-
cp "crash_artifacts/$FIRST_CRASH" ./first_crash/
139-
echo "Copied first crash to ./first_crash/"
140-
else
141-
echo "Warning: Could not find crash file $FIRST_CRASH"
142-
echo "Available files:"
143-
ls -la crash_artifacts/ || echo "No files in crash_artifacts"
144-
fi
145-
146-
- name: Show crash info
147-
run: |
148-
echo "=== Crash Summary ==="
149-
echo "Total crashes found: ${{ needs.io_fuzz.outputs.crash_count }}"
150-
echo "Processing first crash: ${{ needs.io_fuzz.outputs.first_crash_name }}"
151-
echo ""
152-
echo "=== First crash file size ==="
153-
ls -lh first_crash/ || echo "No crash file"
154-
echo ""
155-
echo "=== Fuzzer log preview (last 100 lines, where crashes are reported) ==="
156-
tail -100 logs/fuzz_output.log || echo "No log file"
157-
158-
- name: Create GitHub issue
127+
# Single Claude Code Action that does everything:
128+
# - Analyzes crash from logs
129+
# - Checks for duplicate issues
130+
# - Creates new issue OR comments on existing issue
131+
- name: Analyze and report crash with Claude
159132
env:
160-
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
161-
run: |
162-
# Extract crash type from filename
163-
CRASH_FILE="${{ needs.io_fuzz.outputs.first_crash_name }}"
164-
if [[ "$CRASH_FILE" == crash-* ]]; then
165-
CRASH_TYPE="Crash"
166-
elif [[ "$CRASH_FILE" == leak-* ]]; then
167-
CRASH_TYPE="Memory Leak"
168-
elif [[ "$CRASH_FILE" == timeout-* ]]; then
169-
CRASH_TYPE="Timeout"
170-
elif [[ "$CRASH_FILE" == oom-* ]]; then
171-
CRASH_TYPE="Out of Memory"
172-
else
173-
CRASH_TYPE="Unknown"
174-
fi
175-
176-
# Create issue with gh CLI
177-
gh issue create \
178-
--repo ${{ github.repository }} \
179-
--title "Fuzzing $CRASH_TYPE: file_io - $(date -u +%Y-%m-%d)" \
180-
--label "bug,fuzzing,file_io-fuzz,needs-triage" \
181-
--body-file - <<'EOF'
182-
## Fuzzing Crash Report
183-
184-
The `file_io` fuzzing target detected a $CRASH_TYPE during a scheduled fuzzing run.
185-
186-
### Summary
187-
188-
- **Crash Type**: $CRASH_TYPE
189-
- **Target**: `file_io`
190-
- **Crash File**: `${{ needs.io_fuzz.outputs.first_crash_name }}`
191-
- **Total Crashes Found**: ${{ needs.io_fuzz.outputs.crash_count }}
192-
- **Workflow Run**: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
193-
- **Timestamp**: $(date -u +"%Y-%m-%d %H:%M:%S UTC")
194-
- **Branch**: ${{ github.ref_name }}
195-
- **Commit**: ${{ github.sha }}
196-
197-
### Crash Artifacts
198-
199-
Download crash artifacts from the workflow run:
200-
**https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}**
201-
202-
Artifacts available:
203-
- `io-fuzzing-crash-artifacts` - All crash files found (includes ${{ needs.io_fuzz.outputs.crash_count }} crashes)
204-
- `io-fuzzing-logs` - Complete fuzzer output with stack traces
205-
206-
### Reproduction Steps
207-
208-
1. Download the `io-fuzzing-crash-artifacts` from the workflow run above
209-
2. Extract the crash file to your local `fuzz/artifacts/file_io/` directory
210-
3. Reproduce the crash locally:
211-
212-
```bash
213-
cargo +nightly fuzz run file_io fuzz/artifacts/file_io/${{ needs.io_fuzz.outputs.first_crash_name }}
214-
```
215-
216-
4. Get full backtrace:
217-
218-
```bash
219-
RUST_BACKTRACE=full cargo +nightly fuzz run file_io fuzz/artifacts/file_io/${{ needs.io_fuzz.outputs.first_crash_name }}
220-
```
221-
222-
5. Minimize the test case (optional):
223-
224-
```bash
225-
cargo +nightly fuzz tmin file_io fuzz/artifacts/file_io/${{ needs.io_fuzz.outputs.first_crash_name }}
226-
```
227-
228-
### Investigation Checklist
229-
230-
- [ ] Download crash artifacts from workflow run
231-
- [ ] Reproduce crash locally with full backtrace
232-
- [ ] Analyze stack trace and identify root cause
233-
- [ ] Determine severity (security vs stability)
234-
- [ ] Check if this is a duplicate of an existing issue
235-
- [ ] Minimize test case if needed
236-
- [ ] Create fix PR with reference to this issue
237-
- [ ] Add regression test
238-
- [ ] Verify fix with: `cargo +nightly fuzz run file_io <crash-file>`
239-
240-
### Environment
241-
242-
- **Runner**: ubuntu24-full-arm64
243-
- **Rust Toolchain**: nightly
244-
- **Fuzz Duration**: 7200 seconds (2 hours)
245-
- **Fuzzer**: cargo-fuzz (libFuzzer)
246-
247-
### Additional Context
248-
249-
This issue was automatically created by the fuzzing workflow. If multiple crashes were found, this issue represents the first crash detected. All crash artifacts are available for download.
250-
251-
**Note**: If this issue is a duplicate of an existing bug, please close it and reference the original issue.
252-
253-
---
133+
CRASH_FILE: ${{ needs.io_fuzz.outputs.first_crash_name }}
134+
CRASH_COUNT: ${{ needs.io_fuzz.outputs.crash_count }}
135+
WORKFLOW_RUN: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
136+
ARTIFACT_URL: ${{ needs.io_fuzz.outputs.artifact_url }}
137+
BRANCH: ${{ github.ref_name }}
138+
COMMIT: ${{ github.sha }}
139+
uses: anthropics/claude-code-action@v1
140+
with:
141+
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
142+
github_token: ${{ secrets.GITHUB_TOKEN }}
143+
show_full_output: true
144+
prompt: |
145+
# Fuzzer Crash Analysis and Reporting
146+
147+
A fuzzing run for the `file_io` target has detected a crash. Analyze it and report by creating or updating a GitHub issue.
148+
149+
## Step 1: Analyze the Crash
150+
151+
1. Read the fuzzer log: `logs/fuzz_output.log`
152+
2. Extract:
153+
- Stack trace (lines with `#0`, `#1`, etc.)
154+
- Error message ("panicked at" or "ERROR:")
155+
- Crash location (top user code frame, not std/core/libfuzzer)
156+
- Debug output (look for "Output of `std::fmt::Debug`:" section before the crash)
157+
3. Read the source code at the crash location to understand root cause
158+
159+
## Step 2: Check for Duplicates
160+
161+
1. List existing fuzzer issues:
162+
```bash
163+
gh issue list --repo ${{ github.repository }} --label fuzzer --state open --json number,title,body --limit 50
164+
```
165+
166+
2. For each existing issue, compare:
167+
- **Crash location**: Same file + function? (line numbers can differ)
168+
- **Error pattern**: Same error after normalizing values?
169+
- "index 5 out of bounds" = "index 12 out of bounds" (SAME)
170+
- "len is 100" = "len is 5" (SAME)
171+
- Read source code if needed to verify same root cause
172+
173+
3. Determine duplication level:
174+
- **EXACT DUPLICATE**: Same crash location + same error pattern → Add reaction
175+
- **SIMILAR**: Same general area but unclear → Add comment
176+
- **NEW BUG**: Different location or different error type → Create new issue
177+
178+
## Step 3: Take Action
179+
180+
### If EXACT DUPLICATE (high confidence):
181+
Update or create a tracking comment to count occurrences:
182+
183+
1. First, check if there's already a tracking comment (look for a comment starting with "<!-- occurrences: ")
184+
2. If found, extract the count, increment it, and edit the comment
185+
3. If not found, create a new comment
186+
187+
Comment format:
188+
```markdown
189+
<!-- occurrences: N -->
190+
**Crash seen N time(s)**
191+
192+
Latest occurrence:
193+
- Crash file: $CRASH_FILE
194+
- Artifact: $ARTIFACT_URL
195+
- Branch: $BRANCH
196+
- Commit: $COMMIT
197+
```
198+
199+
Use:
200+
- `gh api repos/${{ github.repository }}/issues/ISSUE_NUM/comments` to list comments
201+
- `gh api repos/${{ github.repository }}/issues/comments/COMMENT_ID -X PATCH` to update
202+
- `gh issue comment ISSUE_NUM` to create new
203+
204+
### If SIMILAR (medium confidence):
205+
Comment on the existing issue with analysis:
206+
```bash
207+
gh issue comment ISSUE_NUM --repo ${{ github.repository }} --body "..."
208+
```
209+
210+
Include in comment:
211+
- Note that another crash was detected
212+
- Your confidence level and reasoning
213+
- Key differences if any
214+
- Crash file: $CRASH_FILE
215+
- Workflow: $WORKFLOW_RUN
216+
217+
### If NEW BUG (not a duplicate):
218+
Create a new issue with `gh issue create`:
219+
```bash
220+
gh issue create --repo ${{ github.repository }} \
221+
--title "Fuzzing Crash: [brief description]" \
222+
--label "bug,fuzzer" \
223+
--body "..."
224+
```
225+
226+
Issue body must include:
227+
```markdown
228+
## Fuzzing Crash Report
229+
230+
### Analysis
231+
232+
**Crash Location**: `file.rs:function_name`
233+
234+
**Error Message**:
235+
```
236+
[error message]
237+
```
238+
239+
**Stack Trace**:
240+
```
241+
[top 5-7 frames - keep in code block to prevent markdown rendering issues]
242+
```
243+
244+
Note: Keep stack traces in code blocks to prevent `#0`, `#1` from being interpreted as markdown headers.
245+
246+
**Root Cause**: [Your analysis]
247+
248+
**Debug Output** (if available):
249+
```
250+
[Include any Debug: output or structured debug information from the fuzzer log]
251+
```
252+
253+
### Summary
254+
255+
- **Target**: `file_io`
256+
- **Crash File**: `$CRASH_FILE`
257+
- **Branch**: $BRANCH
258+
- **Commit**: $COMMIT
259+
- **Crash Artifact**: $ARTIFACT_URL
260+
261+
### Reproduction
262+
263+
1. Download the crash artifact:
264+
- **Direct download**: $ARTIFACT_URL
265+
- Or find `io-fuzzing-crash-artifacts` at: $WORKFLOW_RUN
266+
- Extract the zip file
267+
268+
2. Reproduce locally:
269+
```bash
270+
# The artifact contains file_io/$CRASH_FILE
271+
cargo +nightly fuzz run --sanitizer=none file_io file_io/$CRASH_FILE
272+
```
273+
274+
3. Get full backtrace:
275+
```bash
276+
RUST_BACKTRACE=full cargo +nightly fuzz run --sanitizer=none file_io file_io/$CRASH_FILE
277+
```
278+
279+
---
280+
*Auto-created by fuzzing workflow with Claude analysis*
281+
```
282+
283+
## Important Guidelines
284+
285+
- Be conservative: when unsure, prefer creating a new issue over marking as duplicate
286+
- Focus on ROOT CAUSE, not specific values in error messages
287+
- You have full repo access - read source code to understand crashes
288+
- Only use +1 reaction for truly identical crashes
289+
290+
## Environment Variables
291+
292+
- CRASH_FILE: $CRASH_FILE
293+
- CRASH_COUNT: $CRASH_COUNT
294+
- WORKFLOW_RUN: $WORKFLOW_RUN
295+
- ARTIFACT_URL: $ARTIFACT_URL (direct link to crash artifact)
296+
- BRANCH: $BRANCH
297+
- COMMIT: $COMMIT
298+
299+
Start by reading `logs/fuzz_output.log`.
300+
claude_args: |
301+
--model claude-sonnet-4-5-20250929
302+
--max-turns 25
303+
--allowedTools "Read,Write,Bash(gh issue list:*),Bash(gh issue view:*),Bash(gh issue create:*),Bash(gh issue comment:*),Bash(gh api:*),Bash(echo:*),Bash(cat:*),Bash(jq:*),Bash(grep:*),Bash(cargo +nightly fuzz run:*),Bash(RUST_BACKTRACE=* cargo +nightly fuzz run:*)"
254304
255-
*Automatically created by fuzzing workflow*
256-
*Workflow file: [fuzz.yml](https://github.com/${{ github.repository }}/blob/${{ github.ref_name }}/.github/workflows/fuzz.yml)*
257-
*Run: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}*
258-
EOF
259305
260306
ops_fuzz:
261307
name: "Array Operations Fuzz"

0 commit comments

Comments
 (0)