Associate crashes with last started test#58
Conversation
When a test crashes on assert()/precondition() (SIGTRAP, signal 5, etc.), xcsift now tracks the last "Test Case '...' started." line and associates the crash with that test in failed_tests. Three-level crash detection: - "Restarting after..." with signal code → FailedTest with signal info - Fatal error with lastStartedTest → FailedTest with file/line from error - Safety net: started without completion + TEST FAILED → FailedTest Supports both XCTest and Swift Testing "started" formats.
There was a problem hiding this comment.
Pull request overview
Updates the OutputParser to associate crash indicators in xcodebuild output (signal exits and fatal errors) with the most recently started test, so failedTests is populated with actionable per-test failures instead of only generic run-level failure state.
Changes:
- Track the last started test name and signal code while parsing output, and create
FailedTestentries on crash confirmation / fatal error. - Add an end-of-parse “safety net” to emit a
FailedTestwhen a test started but never completed and the run failed. - Expand
ParsingTeststo cover crash association scenarios for both XCTest and Swift Testing formats.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| Sources/OutputParser.swift | Adds crash association state + parsing logic to map crashes/fatal errors to the last started test. |
| Tests/ParsingTests.swift | Adds test coverage for signal crashes, fatal errors, Swift Testing started format, and safety-net behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| || line.contains("' started") // XCTest: "Test Case '...' started." | ||
| || line.contains("\" started") // Swift Testing: Test "..." started | ||
| || line.contains("signal code ") // "Exited with [unexpected] signal code N" | ||
| || line.hasPrefix("Restarting after") // crash confirmation | ||
|
|
There was a problem hiding this comment.
The new containsRelevant filter checks line.contains("' started") / line.contains("\" started"), which will also mark unrelated lines like Test Suite '...' started at ... as relevant. That defeats the purpose of this fast-path (it forces the parser to run the more expensive parsing chain on many suite-start lines) and could regress performance on large logs. Consider tightening these conditions to the specific formats you actually parse (e.g., hasPrefix("Test Case '") || hasPrefix("Test case '") || hasPrefix("◇ Test ")) rather than substring matches.
| || line.contains("' started") // XCTest: "Test Case '...' started." | |
| || line.contains("\" started") // Swift Testing: Test "..." started | |
| || line.contains("signal code ") // "Exited with [unexpected] signal code N" | |
| || line.hasPrefix("Restarting after") // crash confirmation | |
| || line.hasPrefix("Test Case '") // XCTest: "Test Case '...' started." | |
| || line.hasPrefix("Test case '") // variant casing | |
| || line.hasPrefix("◇ Test ") // Swift Testing formatted tests | |
| || line.hasPrefix("Test \"") // Swift Testing: Test "..." started | |
| || line.contains("signal code ") // "Exited with [unexpected] signal code N" | |
| || line.hasPrefix("Restarting after") // crash confirmation |
Description
Test Case '...' started.lines to associate crashes (SIGTRAP, SIGABRT, Fatal error) with the specific test that caused themassert()/precondition(), xcsift reportedfailed_tests: 0with a generic error — making it impossible for AI agents or developers to identify which test failedRestarting after...with signal code creates aFailedTestentry with signal infoFatal errorwith a tracked started test creates aFailedTestwith file/line from the errorTEST FAILEDappears, aFailedTestis created at end-of-parseTest Case '...' started.) and Swift Testing (◇ Test "..." started.) formatslastStartedTestNameis correctly cleared onpassedorfailedto prevent false associationsAdditional notes