Skip to content

Commit 3c84623

Browse files
kubafloCopilot
andcommitted
Fix: trust test results over XHarness exit code
XHarness can report exit code 83 (APP_LAUNCH_FAILURE) even when tests ran successfully (57 passed, 0 failed). This is a teardown/ cleanup issue, not a real test failure. The parser was checking env error patterns (exit code 83) BEFORE checking actual test results (Passed: 57). This caused the gate to report ENV ERROR when tests actually passed. Fix: check for actual test results (Passed: N where N > 0) FIRST. If tests produced real results, trust them over the exit code. Env error patterns only apply when zero tests ran. Uses the LAST Passed:/Failed: counts in the log to handle cases where Run-DeviceTests.ps1 retries internally and the log contains multiple result blocks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 2dc6a10 commit 3c84623

File tree

1 file changed

+33
-2
lines changed

1 file changed

+33
-2
lines changed

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -551,10 +551,41 @@ function Get-TestResultFromOutput {
551551

552552
$content = Get-Content $LogFile -Raw
553553

554-
# Check for environment/infrastructure errors (not real test failures)
554+
# ── First, check if tests actually ran and produced results ──
555+
# This must come BEFORE env error checks because xharness can report
556+
# exit code 83 (APP_LAUNCH_FAILURE) even when tests ran successfully
557+
# (e.g., due to cleanup/teardown issues after test completion).
558+
559+
# Device test output: check Passed/Failed counts from Run-DeviceTests.ps1
560+
# Format: " Passed: 57\n Failed: 0"
561+
$hasDeviceResults = $false
562+
if ($content -match "Passed:\s*(\d+)" -and $content -match "Failed:\s*(\d+)") {
563+
$lastPassMatch = [regex]::Matches($content, "Passed:\s*(\d+)") | Select-Object -Last 1
564+
$lastFailMatch = [regex]::Matches($content, "Failed:\s*(\d+)") | Select-Object -Last 1
565+
$devicePassCount = [int]$lastPassMatch.Groups[1].Value
566+
$deviceFailCount = [int]$lastFailMatch.Groups[1].Value
567+
$deviceTotal = $devicePassCount + $deviceFailCount
568+
569+
# If tests actually ran (total > 0), trust the results over exit codes
570+
if ($deviceTotal -gt 0) {
571+
$hasDeviceResults = $true
572+
if ($deviceFailCount -gt 0) {
573+
return @{
574+
Passed = $false; FailCount = $deviceFailCount; Failed = $deviceFailCount
575+
PassCount = $devicePassCount; Total = $deviceTotal; Skipped = 0
576+
FailureReason = "Device tests: $deviceFailCount of $deviceTotal failed"
577+
}
578+
}
579+
return @{
580+
Passed = $true; PassCount = $devicePassCount; Failed = 0
581+
FailCount = 0; Total = $deviceTotal; Skipped = 0
582+
}
583+
}
584+
}
585+
586+
# ── Environment/infrastructure errors (only if no real test results above) ──
555587
$envErrorPatterns = @(
556588
@{ Pattern = "error ADB0010.*InstallFailedException"; Message = "App install failed (ADB broken pipe)" }
557-
@{ Pattern = "APP_LAUNCH_FAILURE|exit code:?\s*83"; Message = "App failed to launch (XHarness exit 83)" }
558589
@{ Pattern = "XHarness exit code:\s*83"; Message = "App failed to launch (XHarness exit 83)" }
559590
@{ Pattern = "Application test run crashed"; Message = "App crashed during test run" }
560591
@{ Pattern = "SIGABRT.*load_aot_module"; Message = "App crashed during AOT loading" }

0 commit comments

Comments
 (0)