-
Notifications
You must be signed in to change notification settings - Fork 32
Description
Context:
While we export allure-results in our TMS, we recieve only Failed and Passed test results for iOS UI Tests, but on Android side we have third results which shows up as "Broken".
Goal:
distinguish failed (assertion failures) from broken (infrastructure/runtime issues) when converting xcresult → allure-results for iOS UI tests.
Proposal:
Map Allure status based on the content of xcresult:
Classify as failed if there are clear assertion indicators, e.g. in testFailureSummaries / failure messages:
-Message contains: Assertion Failure, XCTAssert, XCTFail, expected:<...> but was:<...>, etc.
Classify as broken if there are signs of runtime/infrastructure problems:
-errorSummaries present (not tied to an explicit assertion).
Crash/termination/runner problems in messages or logs, e.g. phrases like:
-App crashed, Application crashed, App terminated,
UI Testing Failure:
-App terminated,
Lost connection to test process,
Test runner exited / failed to launch,
Timed out waiting for UI Testing to start,
Application quit unexpectedly,
Test did not run, Failed to establish test session.
-Presence of diagnostic/crash artifacts in the bundle (e.g. .crash, .ips, crash logs referenced via diagnosticsRef/attachments).
-Failures occurring during setUp / tearDown phases without an explicit assertion failure (environment/setup errors).