Skip to content

Commit 06649ef

Browse files
committed
feat: add automated crash reporting for array_ops fuzzing
Add Claude-powered crash analysis and GitHub issue creation for the array_ops fuzzing target, matching the existing file_io fuzzing setup. Changes: - Extract shared crash reporting logic into reusable composite action - Add crash detection and reporting to ops_fuzz job - Add report-ops-fuzz-failures job using shared action - Refactor report-io-fuzz-failures to use shared action This reduces code duplication and makes it easier to add fuzzing crash reporting for additional targets in the future. Signed-off-by: Joe Isaacs <[email protected]>
1 parent 5ee3983 commit 06649ef

File tree

5 files changed

+303
-185
lines changed

5 files changed

+303
-185
lines changed
Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
name: "Report Fuzz Failure"
2+
description: "Analyzes fuzzer crashes and creates or updates GitHub issues using Claude"
3+
inputs:
4+
fuzz_target:
5+
description: "Name of the fuzz target (e.g., file_io, array_ops)"
6+
required: true
7+
crash_file:
8+
description: "Name of the crash file"
9+
required: true
10+
artifact_url:
11+
description: "URL to the crash artifact"
12+
required: true
13+
artifact_name:
14+
description: "Name of the artifact (e.g., io-fuzzing-crash-artifacts)"
15+
required: true
16+
logs_artifact_name:
17+
description: "Name of the logs artifact"
18+
required: true
19+
branch:
20+
description: "Branch name"
21+
required: true
22+
commit:
23+
description: "Commit SHA"
24+
required: true
25+
claude_code_oauth_token:
26+
description: "Claude Code OAuth token"
27+
required: true
28+
github_token:
29+
description: "GitHub token"
30+
required: true
31+
32+
runs:
33+
using: "composite"
34+
steps:
35+
- name: Checkout repository
36+
uses: actions/checkout@v5
37+
38+
- name: Download fuzzer logs
39+
uses: actions/download-artifact@v4
40+
with:
41+
name: ${{ inputs.logs_artifact_name }}
42+
path: ./logs
43+
44+
- name: Analyze and report crash with Claude
45+
env:
46+
CRASH_FILE: ${{ inputs.crash_file }}
47+
ARTIFACT_URL: ${{ inputs.artifact_url }}
48+
BRANCH: ${{ inputs.branch }}
49+
COMMIT: ${{ inputs.commit }}
50+
FUZZ_TARGET: ${{ inputs.fuzz_target }}
51+
ARTIFACT_NAME: ${{ inputs.artifact_name }}
52+
uses: anthropics/claude-code-action@v1
53+
with:
54+
claude_code_oauth_token: ${{ inputs.claude_code_oauth_token }}
55+
github_token: ${{ inputs.github_token }}
56+
show_full_output: true
57+
prompt: |
58+
# Fuzzer Crash Analysis and Reporting
59+
60+
A fuzzing run for the `$FUZZ_TARGET` target has detected a crash. Analyze it and report by creating or updating a GitHub issue.
61+
62+
## Step 1: Analyze the Crash
63+
64+
1. Read the fuzzer log: `logs/fuzz_output.log`
65+
2. Extract:
66+
- Stack trace (lines with `#0`, `#1`, etc.)
67+
- Error message ("panicked at" or "ERROR:")
68+
- Crash location (top user code frame, not std/core/libfuzzer)
69+
- Debug output (look for "Output of `std::fmt::Debug`:" section before the crash)
70+
3. Read the source code at the crash location to understand root cause
71+
72+
## Step 2: Check for Duplicates
73+
74+
1. List existing fuzzer issues:
75+
```bash
76+
gh issue list --repo ${{ github.repository }} --label fuzzer --state open --json number,title,body --limit 50
77+
```
78+
79+
2. For each existing issue, compare:
80+
- **Crash location**: Same file + function? (line numbers can differ)
81+
- **Error pattern**: Same error after normalizing values?
82+
- "index 5 out of bounds" = "index 12 out of bounds" (SAME)
83+
- "len is 100" = "len is 5" (SAME)
84+
- Read source code if needed to verify same root cause
85+
86+
3. Determine duplication level:
87+
- **EXACT DUPLICATE**: Same crash location + same error pattern → Add reaction
88+
- **SIMILAR**: Same general area but unclear → Add comment
89+
- **NEW BUG**: Different location or different error type → Create new issue
90+
91+
## Step 3: Take Action
92+
93+
### If EXACT DUPLICATE (high confidence):
94+
Update or create a tracking comment to count occurrences:
95+
96+
1. First, check if there's already a tracking comment (look for a comment starting with "<!-- occurrences: ")
97+
2. If found, extract the count, increment it, and edit the comment
98+
3. If not found, create a new comment
99+
100+
Comment format:
101+
```markdown
102+
<!-- occurrences: N -->
103+
**Crash seen N time(s)**
104+
105+
Latest occurrence:
106+
- Crash file: $CRASH_FILE
107+
- Artifact: $ARTIFACT_URL
108+
- Branch: $BRANCH
109+
- Commit: $COMMIT
110+
```
111+
112+
Use:
113+
- `gh api repos/${{ github.repository }}/issues/ISSUE_NUM/comments` to list comments
114+
- `gh api repos/${{ github.repository }}/issues/comments/COMMENT_ID -X PATCH` to update
115+
- `gh issue comment ISSUE_NUM` to create new
116+
117+
### If SIMILAR (medium confidence):
118+
Comment on the existing issue with analysis:
119+
```bash
120+
gh issue comment ISSUE_NUM --repo ${{ github.repository }} --body "..."
121+
```
122+
123+
Include in comment:
124+
- Note that another crash was detected
125+
- Your confidence level and reasoning
126+
- Key differences if any
127+
- Crash file: $CRASH_FILE
128+
- Workflow: $WORKFLOW_RUN
129+
130+
### If NEW BUG (not a duplicate):
131+
Create a new issue with `gh issue create`:
132+
```bash
133+
gh issue create --repo ${{ github.repository }} \
134+
--title "Fuzzing Crash: [brief description]" \
135+
--label "bug,fuzzer" \
136+
--body "..."
137+
```
138+
139+
Issue body must include:
140+
```markdown
141+
## Fuzzing Crash Report
142+
143+
### Analysis
144+
145+
**Crash Location**: `file.rs:function_name`
146+
147+
**Error Message**:
148+
```
149+
[error message]
150+
```
151+
152+
**Stack Trace**:
153+
```
154+
[top 5-7 frames - keep in code block to prevent markdown rendering issues]
155+
```
156+
157+
Note: Keep stack traces in code blocks to prevent `#0`, `#1` from being interpreted as markdown headers.
158+
159+
**Root Cause**: [Your analysis]
160+
161+
<details>
162+
<summary>Debug Output</summary>
163+
164+
```
165+
[Include the complete "Output of std::fmt::Debug:" section from the fuzzer log]
166+
```
167+
</details>
168+
169+
### Summary
170+
171+
- **Target**: `$FUZZ_TARGET`
172+
- **Crash File**: `$CRASH_FILE`
173+
- **Branch**: $BRANCH
174+
- **Commit**: $COMMIT
175+
- **Crash Artifact**: $ARTIFACT_URL
176+
177+
### Reproduction
178+
179+
1. Download the crash artifact:
180+
- **Direct download**: $ARTIFACT_URL
181+
- Or find `$ARTIFACT_NAME` at: $WORKFLOW_RUN
182+
- Extract the zip file
183+
184+
2. Reproduce locally:
185+
```bash
186+
# The artifact contains $FUZZ_TARGET/$CRASH_FILE
187+
cargo +nightly fuzz run --sanitizer=none $FUZZ_TARGET $FUZZ_TARGET/$CRASH_FILE
188+
```
189+
190+
3. Get full backtrace:
191+
```bash
192+
RUST_BACKTRACE=full cargo +nightly fuzz run --sanitizer=none $FUZZ_TARGET $FUZZ_TARGET/$CRASH_FILE
193+
```
194+
195+
---
196+
*Auto-created by fuzzing workflow with Claude analysis*
197+
```
198+
199+
## Important Guidelines
200+
201+
- Be conservative: when unsure, prefer creating a new issue over marking as duplicate
202+
- Focus on ROOT CAUSE, not specific values in error messages
203+
- You have full repo access - read source code to understand crashes
204+
- Only use +1 reaction for truly identical crashes
205+
206+
## Environment Variables
207+
208+
- CRASH_FILE: $CRASH_FILE
209+
- ARTIFACT_URL: $ARTIFACT_URL (direct link to crash artifact)
210+
- BRANCH: $BRANCH
211+
- COMMIT: $COMMIT
212+
- FUZZ_TARGET: $FUZZ_TARGET
213+
- ARTIFACT_NAME: $ARTIFACT_NAME
214+
215+
Start by reading `logs/fuzz_output.log`.
216+
claude_args: |
217+
--model claude-sonnet-4-5-20250929
218+
--max-turns 25
219+
--allowedTools "Read,Write,Bash(gh issue list:*),Bash(gh issue view:*),Bash(gh issue create:*),Bash(gh issue comment:*),Bash(gh api:*),Bash(echo:*),Bash(cat:*),Bash(jq:*),Bash(grep:*),Bash(cargo +nightly fuzz run:*),Bash(RUST_BACKTRACE=* cargo +nightly fuzz run:*)"

.github/workflows/claude.yml

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,15 +45,21 @@ jobs:
4545
additional_permissions: |
4646
actions: read
4747
48+
# Optional: Specify model (defaults to Claude Sonnet 4, uncomment for Claude Opus 4)
49+
# model: "claude-opus-4-20250514"
50+
4851
# Optional: Customize the trigger phrase (default: @claude)
4952
# trigger_phrase: "/claude"
5053

5154
# Optional: Trigger when specific user is assigned to an issue
5255
# assignee_trigger: "claude-bot"
5356

54-
claude_args: |
55-
--allowedTools "Bash(cargo nextest:*),Bash(cargo check:*),Bash(cargo clippy:*),Bash(cargo fmt:*),Bash(uv run:*)"
56-
--system-prompt "You have been granted tools for editing files and running cargo commands (cargo nextest, cargo check, cargo clippy, cargo fmt) and uv for running pytest (e.g. via uv run --all-packages pytest)"
57+
allowed_tools: "Bash(cargo nextest:*),Bash(cargo check:*),Bash(cargo clippy:*),Bash(cargo fmt:*),Bash(uv run:*)"
58+
59+
# Optional: Add custom instructions for Claude to customize its behavior for your project
60+
custom_instructions: "You have also been granted tools:
61+
- for editing files and running cargo commands (cargo nextest, cargo check, cargo clippy, cargo fmt).
62+
- uv for running pytest (e.g. via uv run --all-packages pytest)"
5763

5864
# Optional: Custom environment variables for Claude
5965
# claude_env: |

0 commit comments

Comments
 (0)