Skip to content

Commit 39b26ff

Browse files
committed
feat: add automated crash reporting for array_ops fuzzing
Add Claude-powered crash analysis and GitHub issue creation for the array_ops fuzzing target, matching the existing file_io fuzzing setup. Changes: - Add crash detection and reporting to ops_fuzz job - Add report-ops-fuzz-failures job with Claude analysis - Use environment variables to parameterize shared Claude prompt logic This enables automated issue creation for array_ops fuzzing crashes, matching the existing file_io crash reporting functionality. Signed-off-by: Joe Isaacs <[email protected]>
1 parent 5ee3983 commit 39b26ff

File tree

1 file changed

+247
-8
lines changed

1 file changed

+247
-8
lines changed

.github/workflows/fuzz.yml

Lines changed: 247 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,8 @@ jobs:
129129
ARTIFACT_URL: ${{ needs.io_fuzz.outputs.artifact_url }}
130130
BRANCH: ${{ github.ref_name }}
131131
COMMIT: ${{ github.sha }}
132+
FUZZ_TARGET: file_io
133+
ARTIFACT_NAME: io-fuzzing-crash-artifacts
132134
uses: anthropics/claude-code-action@v1
133135
with:
134136
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
@@ -137,7 +139,7 @@ jobs:
137139
prompt: |
138140
# Fuzzer Crash Analysis and Reporting
139141
140-
A fuzzing run for the `file_io` target has detected a crash. Analyze it and report by creating or updating a GitHub issue.
142+
A fuzzing run for the `$FUZZ_TARGET` target has detected a crash. Analyze it and report by creating or updating a GitHub issue.
141143
142144
## Step 1: Analyze the Crash
143145
@@ -248,7 +250,7 @@ jobs:
248250
249251
### Summary
250252
251-
- **Target**: `file_io`
253+
- **Target**: `$FUZZ_TARGET`
252254
- **Crash File**: `$CRASH_FILE`
253255
- **Branch**: $BRANCH
254256
- **Commit**: $COMMIT
@@ -258,18 +260,18 @@ jobs:
258260
259261
1. Download the crash artifact:
260262
- **Direct download**: $ARTIFACT_URL
261-
- Or find `io-fuzzing-crash-artifacts` at: $WORKFLOW_RUN
263+
- Or find `$ARTIFACT_NAME` at: $WORKFLOW_RUN
262264
- Extract the zip file
263265
264266
2. Reproduce locally:
265267
```bash
266-
# The artifact contains file_io/$CRASH_FILE
267-
cargo +nightly fuzz run --sanitizer=none file_io file_io/$CRASH_FILE
268+
# The artifact contains $FUZZ_TARGET/$CRASH_FILE
269+
cargo +nightly fuzz run --sanitizer=none $FUZZ_TARGET $FUZZ_TARGET/$CRASH_FILE
268270
```
269271
270272
3. Get full backtrace:
271273
```bash
272-
RUST_BACKTRACE=full cargo +nightly fuzz run --sanitizer=none file_io file_io/$CRASH_FILE
274+
RUST_BACKTRACE=full cargo +nightly fuzz run --sanitizer=none $FUZZ_TARGET $FUZZ_TARGET/$CRASH_FILE
273275
```
274276
275277
---
@@ -289,6 +291,8 @@ jobs:
289291
- ARTIFACT_URL: $ARTIFACT_URL (direct link to crash artifact)
290292
- BRANCH: $BRANCH
291293
- COMMIT: $COMMIT
294+
- FUZZ_TARGET: $FUZZ_TARGET
295+
- ARTIFACT_NAME: $ARTIFACT_NAME
292296
293297
Start by reading `logs/fuzz_output.log`.
294298
claude_args: |
@@ -307,6 +311,10 @@ jobs:
307311
- disk=large
308312
- extras=s3-cache
309313
- tag=ops-fuzz
314+
outputs:
315+
crashes_found: ${{ steps.check.outputs.crashes_found }}
316+
first_crash_name: ${{ steps.check.outputs.first_crash_name }}
317+
artifact_url: ${{ steps.upload_artifacts.outputs.artifact-url }}
310318
steps:
311319
- uses: runs-on/action@v2
312320
with:
@@ -335,13 +343,47 @@ jobs:
335343
AWS_ENDPOINT_URL: "https://01e9655179bbec953276890b183039bc.r2.cloudflarestorage.com"
336344
- name: Run fuzzing target
337345
id: fuzz
338-
run: RUST_BACKTRACE=1 cargo +nightly fuzz run --release --debug-assertions array_ops -- -max_total_time=7200
346+
run: |
347+
RUST_BACKTRACE=1 cargo +nightly fuzz run --release --debug-assertions array_ops -- -max_total_time=7200 2>&1 | tee fuzz_output.log
339348
continue-on-error: true
349+
- name: Check for crashes
350+
id: check
351+
run: |
352+
if [ -d "fuzz/artifacts" ] && [ "$(ls -A fuzz/artifacts 2>/dev/null)" ]; then
353+
echo "crashes_found=true" >> $GITHUB_OUTPUT
354+
355+
# Get the first crash file only
356+
FIRST_CRASH=$(find fuzz/artifacts -type f \( -name "crash-*" -o -name "leak-*" -o -name "timeout-*" -o -name "oom-*" \) | head -1)
357+
358+
if [ -n "$FIRST_CRASH" ]; then
359+
echo "first_crash=$FIRST_CRASH" >> $GITHUB_OUTPUT
360+
echo "first_crash_name=$(basename $FIRST_CRASH)" >> $GITHUB_OUTPUT
361+
362+
# Count all crashes for reporting
363+
CRASH_COUNT=$(find fuzz/artifacts -type f \( -name "crash-*" -o -name "leak-*" -o -name "timeout-*" -o -name "oom-*" \) | wc -l)
364+
echo "crash_count=$CRASH_COUNT" >> $GITHUB_OUTPUT
365+
echo "Found $CRASH_COUNT crash(es), will process first: $(basename $FIRST_CRASH)"
366+
fi
367+
else
368+
echo "crashes_found=false" >> $GITHUB_OUTPUT
369+
echo "crash_count=0" >> $GITHUB_OUTPUT
370+
echo "No crashes found"
371+
fi
340372
- name: Archive crash artifacts
373+
id: upload_artifacts
374+
if: steps.check.outputs.crashes_found == 'true'
341375
uses: actions/upload-artifact@v5
342376
with:
343377
name: operations-fuzzing-crash-artifacts
344378
path: fuzz/artifacts
379+
retention-days: 30
380+
- name: Archive fuzzer output log
381+
if: steps.check.outputs.crashes_found == 'true'
382+
uses: actions/upload-artifact@v4
383+
with:
384+
name: ops-fuzzing-logs
385+
path: fuzz_output.log
386+
retention-days: 30
345387
- name: Persist corpus
346388
shell: bash
347389
run: |
@@ -353,5 +395,202 @@ jobs:
353395
AWS_REGION: "us-east-1"
354396
AWS_ENDPOINT_URL: "https://01e9655179bbec953276890b183039bc.r2.cloudflarestorage.com"
355397
- name: Fail job if fuzz run found a bug
356-
if: steps.fuzz.outcome == 'failure'
398+
if: steps.check.outputs.crashes_found == 'true'
357399
run: exit 1
400+
401+
report-ops-fuzz-failures:
402+
name: "Report Array Operations Fuzz Failures"
403+
needs: ops_fuzz
404+
if: always() && needs.ops_fuzz.outputs.crashes_found == 'true'
405+
runs-on: ubuntu-latest
406+
permissions:
407+
issues: write
408+
contents: read
409+
id-token: write
410+
pull-requests: read
411+
steps:
412+
- name: Checkout repository
413+
uses: actions/checkout@v5
414+
415+
- name: Download fuzzer logs
416+
uses: actions/download-artifact@v4
417+
with:
418+
name: ops-fuzzing-logs
419+
path: ./logs
420+
421+
- name: Analyze and report crash with Claude
422+
env:
423+
CRASH_FILE: ${{ needs.ops_fuzz.outputs.first_crash_name }}
424+
ARTIFACT_URL: ${{ needs.ops_fuzz.outputs.artifact_url }}
425+
BRANCH: ${{ github.ref_name }}
426+
COMMIT: ${{ github.sha }}
427+
FUZZ_TARGET: array_ops
428+
ARTIFACT_NAME: operations-fuzzing-crash-artifacts
429+
uses: anthropics/claude-code-action@v1
430+
with:
431+
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
432+
github_token: ${{ secrets.GITHUB_TOKEN }}
433+
show_full_output: true
434+
prompt: |
435+
# Fuzzer Crash Analysis and Reporting
436+
437+
A fuzzing run for the `$FUZZ_TARGET` target has detected a crash. Analyze it and report by creating or updating a GitHub issue.
438+
439+
## Step 1: Analyze the Crash
440+
441+
1. Read the fuzzer log: `logs/fuzz_output.log`
442+
2. Extract:
443+
- Stack trace (lines with `#0`, `#1`, etc.)
444+
- Error message ("panicked at" or "ERROR:")
445+
- Crash location (top user code frame, not std/core/libfuzzer)
446+
- Debug output (look for "Output of `std::fmt::Debug`:" section before the crash)
447+
3. Read the source code at the crash location to understand root cause
448+
449+
## Step 2: Check for Duplicates
450+
451+
1. List existing fuzzer issues:
452+
```bash
453+
gh issue list --repo ${{ github.repository }} --label fuzzer --state open --json number,title,body --limit 50
454+
```
455+
456+
2. For each existing issue, compare:
457+
- **Crash location**: Same file + function? (line numbers can differ)
458+
- **Error pattern**: Same error after normalizing values?
459+
- "index 5 out of bounds" = "index 12 out of bounds" (SAME)
460+
- "len is 100" = "len is 5" (SAME)
461+
- Read source code if needed to verify same root cause
462+
463+
3. Determine duplication level:
464+
- **EXACT DUPLICATE**: Same crash location + same error pattern → Add reaction
465+
- **SIMILAR**: Same general area but unclear → Add comment
466+
- **NEW BUG**: Different location or different error type → Create new issue
467+
468+
## Step 3: Take Action
469+
470+
### If EXACT DUPLICATE (high confidence):
471+
Update or create a tracking comment to count occurrences:
472+
473+
1. First, check if there's already a tracking comment (look for a comment starting with "<!-- occurrences: ")
474+
2. If found, extract the count, increment it, and edit the comment
475+
3. If not found, create a new comment
476+
477+
Comment format:
478+
```markdown
479+
<!-- occurrences: N -->
480+
**Crash seen N time(s)**
481+
482+
Latest occurrence:
483+
- Crash file: $CRASH_FILE
484+
- Artifact: $ARTIFACT_URL
485+
- Branch: $BRANCH
486+
- Commit: $COMMIT
487+
```
488+
489+
Use:
490+
- `gh api repos/${{ github.repository }}/issues/ISSUE_NUM/comments` to list comments
491+
- `gh api repos/${{ github.repository }}/issues/comments/COMMENT_ID -X PATCH` to update
492+
- `gh issue comment ISSUE_NUM` to create new
493+
494+
### If SIMILAR (medium confidence):
495+
Comment on the existing issue with analysis:
496+
```bash
497+
gh issue comment ISSUE_NUM --repo ${{ github.repository }} --body "..."
498+
```
499+
500+
Include in comment:
501+
- Note that another crash was detected
502+
- Your confidence level and reasoning
503+
- Key differences if any
504+
- Crash file: $CRASH_FILE
505+
- Workflow: $WORKFLOW_RUN
506+
507+
### If NEW BUG (not a duplicate):
508+
Create a new issue with `gh issue create`:
509+
```bash
510+
gh issue create --repo ${{ github.repository }} \
511+
--title "Fuzzing Crash: [brief description]" \
512+
--label "bug,fuzzer" \
513+
--body "..."
514+
```
515+
516+
Issue body must include:
517+
```markdown
518+
## Fuzzing Crash Report
519+
520+
### Analysis
521+
522+
**Crash Location**: `file.rs:function_name`
523+
524+
**Error Message**:
525+
```
526+
[error message]
527+
```
528+
529+
**Stack Trace**:
530+
```
531+
[top 5-7 frames - keep in code block to prevent markdown rendering issues]
532+
```
533+
534+
Note: Keep stack traces in code blocks to prevent `#0`, `#1` from being interpreted as markdown headers.
535+
536+
**Root Cause**: [Your analysis]
537+
538+
<details>
539+
<summary>Debug Output</summary>
540+
541+
```
542+
[Include the complete "Output of std::fmt::Debug:" section from the fuzzer log]
543+
```
544+
</details>
545+
546+
### Summary
547+
548+
- **Target**: `$FUZZ_TARGET`
549+
- **Crash File**: `$CRASH_FILE`
550+
- **Branch**: $BRANCH
551+
- **Commit**: $COMMIT
552+
- **Crash Artifact**: $ARTIFACT_URL
553+
554+
### Reproduction
555+
556+
1. Download the crash artifact:
557+
- **Direct download**: $ARTIFACT_URL
558+
- Or find `$ARTIFACT_NAME` at: $WORKFLOW_RUN
559+
- Extract the zip file
560+
561+
2. Reproduce locally:
562+
```bash
563+
# The artifact contains $FUZZ_TARGET/$CRASH_FILE
564+
cargo +nightly fuzz run --sanitizer=none $FUZZ_TARGET $FUZZ_TARGET/$CRASH_FILE
565+
```
566+
567+
3. Get full backtrace:
568+
```bash
569+
RUST_BACKTRACE=full cargo +nightly fuzz run --sanitizer=none $FUZZ_TARGET $FUZZ_TARGET/$CRASH_FILE
570+
```
571+
572+
---
573+
*Auto-created by fuzzing workflow with Claude analysis*
574+
```
575+
576+
## Important Guidelines
577+
578+
- Be conservative: when unsure, prefer creating a new issue over marking as duplicate
579+
- Focus on ROOT CAUSE, not specific values in error messages
580+
- You have full repo access - read source code to understand crashes
581+
- Only use +1 reaction for truly identical crashes
582+
583+
## Environment Variables
584+
585+
- CRASH_FILE: $CRASH_FILE
586+
- ARTIFACT_URL: $ARTIFACT_URL (direct link to crash artifact)
587+
- BRANCH: $BRANCH
588+
- COMMIT: $COMMIT
589+
- FUZZ_TARGET: $FUZZ_TARGET
590+
- ARTIFACT_NAME: $ARTIFACT_NAME
591+
592+
Start by reading `logs/fuzz_output.log`.
593+
claude_args: |
594+
--model claude-sonnet-4-5-20250929
595+
--max-turns 25
596+
--allowedTools "Read,Write,Bash(gh issue list:*),Bash(gh issue view:*),Bash(gh issue create:*),Bash(gh issue comment:*),Bash(gh api:*),Bash(echo:*),Bash(cat:*),Bash(jq:*),Bash(grep:*),Bash(cargo +nightly fuzz run:*),Bash(RUST_BACKTRACE=* cargo +nightly fuzz run:*)"

0 commit comments

Comments
 (0)