Skip to content

Associate crashes with last started test#58

Merged
alexey1312 merged 1 commit intomasterfrom
feat/crash-to-test-association
Feb 28, 2026
Merged

Associate crashes with last started test#58
alexey1312 merged 1 commit intomasterfrom
feat/crash-to-test-association

Conversation

@alexey1312
Copy link
Collaborator

Description

  • Track Test Case '...' started. lines to associate crashes (SIGTRAP, SIGABRT, Fatal error) with the specific test that caused them
  • Previously, when a test crashed on assert()/precondition(), xcsift reported failed_tests: 0 with a generic error — making it impossible for AI agents or developers to identify which test failed
  • Three-level crash detection:
    • Restarting after... with signal code creates a FailedTest entry with signal info
    • Fatal error with a tracked started test creates a FailedTest with file/line from the error
    • Safety net: if a test started but never completed and TEST FAILED appears, a FailedTest is created at end-of-parse
  • Supports both XCTest (Test Case '...' started.) and Swift Testing (◇ Test "..." started.) formats
  • lastStartedTestName is correctly cleared on passed or failed to prevent false associations

Additional notes

  • 8 new tests + 1 updated existing test covering all crash association scenarios
  • All 286 tests pass with zero regressions

When a test crashes on assert()/precondition() (SIGTRAP, signal 5, etc.),
xcsift now tracks the last "Test Case '...' started." line and associates
the crash with that test in failed_tests.

Three-level crash detection:
- "Restarting after..." with signal code → FailedTest with signal info
- Fatal error with lastStartedTest → FailedTest with file/line from error
- Safety net: started without completion + TEST FAILED → FailedTest

Supports both XCTest and Swift Testing "started" formats.
@alexey1312 alexey1312 requested a review from Copilot February 28, 2026 17:45
@alexey1312 alexey1312 merged commit a1723d8 into master Feb 28, 2026
4 checks passed
@alexey1312 alexey1312 deleted the feat/crash-to-test-association branch February 28, 2026 17:47
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the OutputParser to associate crash indicators in xcodebuild output (signal exits and fatal errors) with the most recently started test, so failedTests is populated with actionable per-test failures instead of only generic run-level failure state.

Changes:

  • Track the last started test name and signal code while parsing output, and create FailedTest entries on crash confirmation / fatal error.
  • Add an end-of-parse “safety net” to emit a FailedTest when a test started but never completed and the run failed.
  • Expand ParsingTests to cover crash association scenarios for both XCTest and Swift Testing formats.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
Sources/OutputParser.swift Adds crash association state + parsing logic to map crashes/fatal errors to the last started test.
Tests/ParsingTests.swift Adds test coverage for signal crashes, fatal errors, Swift Testing started format, and safety-net behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +723 to 727
|| line.contains("' started") // XCTest: "Test Case '...' started."
|| line.contains("\" started") // Swift Testing: Test "..." started
|| line.contains("signal code ") // "Exited with [unexpected] signal code N"
|| line.hasPrefix("Restarting after") // crash confirmation

Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new containsRelevant filter checks line.contains("' started") / line.contains("\" started"), which will also mark unrelated lines like Test Suite '...' started at ... as relevant. That defeats the purpose of this fast-path (it forces the parser to run the more expensive parsing chain on many suite-start lines) and could regress performance on large logs. Consider tightening these conditions to the specific formats you actually parse (e.g., hasPrefix("Test Case '") || hasPrefix("Test case '") || hasPrefix("◇ Test ")) rather than substring matches.

Suggested change
|| line.contains("' started") // XCTest: "Test Case '...' started."
|| line.contains("\" started") // Swift Testing: Test "..." started
|| line.contains("signal code ") // "Exited with [unexpected] signal code N"
|| line.hasPrefix("Restarting after") // crash confirmation
|| line.hasPrefix("Test Case '") // XCTest: "Test Case '...' started."
|| line.hasPrefix("Test case '") // variant casing
|| line.hasPrefix("◇ Test ") // Swift Testing formatted tests
|| line.hasPrefix("Test \"") // Swift Testing: Test "..." started
|| line.contains("signal code ") // "Exited with [unexpected] signal code N"
|| line.hasPrefix("Restarting after") // crash confirmation

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants