feat: add test_step_outcome input to detect silent test failures#169
feat: add test_step_outcome input to detect silent test failures#169
Conversation
Allow users to pass the outcome of their test runner step so the CLI can detect cases where the runner crashed but the JUnit report looks clean. Maps success→0, failure/cancelled→1, omitted→empty (ignored). Change-Id: Ibab2d1e5546eadafe040e01c4a7483735a8bcce7
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🔴 🤖 Continuous IntegrationThis rule is failing.
🔴 👀 Review RequirementsThis rule is failing.
🔴 🔎 ReviewsThis rule is failing.
🟢 Enforce conventional commitWonderful, this rule succeeded.Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 📕 PR descriptionWonderful, this rule succeeded.
|
There was a problem hiding this comment.
Pull request overview
Adds a new optional action input that lets workflows pass the test runner step outcome so the junit-process CLI invocation can detect “silent” test runner crashes even when JUnit output looks clean.
Changes:
- Document a new
test_step_outcomeinput in the action docs. - Add
test_step_outcomeinput toaction.yml. - Map the input to
MERGIFY_TEST_EXIT_CODEfor thejunit-processstep via a GitHub Actions expression.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
README.md |
Documents the new test_step_outcome input in the auto-generated Inputs section. |
action.yml |
Adds the test_step_outcome input and derives MERGIFY_TEST_EXIT_CODE from it for junit-process. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| MERGIFY_API_URL: ${{ inputs.mergify_api_url }} | ||
| MERGIFY_TOKEN: ${{ inputs.token }} | ||
| MERGIFY_JOB_NAME: ${{ inputs.job_name }} | ||
| MERGIFY_TEST_EXIT_CODE: ${{ inputs.test_step_outcome == 'success' && '0' || inputs.test_step_outcome && '1' || '' }} |
There was a problem hiding this comment.
The expression sets MERGIFY_TEST_EXIT_CODE to 1 for any non-empty test_step_outcome that isn't exactly success. If users pass steps.<id>.outcome, GitHub can return skipped, which would currently be treated as a failure. Consider explicitly mapping only failure/cancelled to 1 and treating skipped (and empty) as '' (ignored), and/or add an input validation check that errors on unexpected values to avoid silent misclassification.
| MERGIFY_TEST_EXIT_CODE: ${{ inputs.test_step_outcome == 'success' && '0' || inputs.test_step_outcome && '1' || '' }} | |
| MERGIFY_TEST_EXIT_CODE: ${{ inputs.test_step_outcome == 'success' && '0' || (inputs.test_step_outcome == 'failure' || inputs.test_step_outcome == 'cancelled') && '1' || '' }} |
| Outcome of the test runner step (e.g. steps.<id>.outcome). | ||
| Pass this to detect silent failures where the test runner crashed | ||
| but the JUnit report appears clean. | ||
| Values: 'success', 'failure', 'cancelled', or omit entirely. |
There was a problem hiding this comment.
The documented allowed values omit skipped, but steps.<id>.outcome can be skipped. If skipped is expected to be ignored (similar to omitting the input), it should be documented here and handled consistently by the env var mapping.
| Values: 'success', 'failure', 'cancelled', or omit entirely. | |
| Values: 'success', 'failure', 'cancelled', 'skipped', or omit entirely. 'skipped' is treated the same as omitting this input. |
| # Outcome of the test runner step (e.g. steps.<id>.outcome). | ||
| # Pass this to detect silent failures where the test runner crashed | ||
| # but the JUnit report appears clean. | ||
| # Values: 'success', 'failure', 'cancelled', or omit entirely. |
There was a problem hiding this comment.
This input documentation lists success, failure, and cancelled, but steps.<id>.outcome can also be skipped. If users pass steps.<id>.outcome directly, skipped would currently be treated as a failure by the action’s mapping. Please document how skipped should be handled and keep it consistent with action.yml.
| # Values: 'success', 'failure', 'cancelled', or omit entirely. | |
| # Values: 'success', 'failure', 'cancelled', 'skipped', or omit entirely. | |
| # 'skipped' is supported when passing steps.<id>.outcome directly. |
jd
deleted the
devs/jd/worktree-test-code-exit/Ibab2d1e5546eadafe040e01c4a7483735a8bcce7
branch
Allow users to pass the outcome of their test runner step so the CLI
can detect cases where the runner crashed but the JUnit report looks
clean. Maps success→0, failure/cancelled→1, omitted→empty (ignored).