-
Notifications
You must be signed in to change notification settings - Fork 740
feat(cli): categorize execution results with resultType for better failure tracking #1265
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
✅ Deploy Preview for midscene ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@copilot A type should be added to categorize these errors instead of directly counting them under failures. |
Added
The execution summary now displays:
This provides better visibility into what happened during execution. Changes implemented in commit b2ee934. |
- Added 'resultType' field to MidsceneYamlConfigResult to distinguish between: - 'success': All tasks completed successfully - 'failed': Complete execution failure (player error) - 'partialFailed': Some tasks failed with continueOnError=true - 'notExecuted': Not executed due to previous failures - Updated BatchRunner to track and display partial failures separately - Added 'Partial failed' count to execution summary - Updated tests to verify correct categorization This provides better visibility into what actually happened during execution, distinguishing between complete failures and partial failures where tasks failed but execution continued due to continueOnError setting. Co-authored-by: quanru <[email protected]>
b2ee934
to
da82da4
Compare
Problem
When
continueOnError: true
is specified in YAML configuration, assertion failures were not being properly counted in the execution summary's "Failed" count. The execution summary would incorrectly show "Failed: 0" even when some tasks contained failed assertions.Root Cause
The
BatchRunner.processResults()
method only checkedplayer.status !== 'error'
to determine if a file execution was successful. However, whencontinueOnError: true
is used:status === 'error'
(e.g., failed assertions)'done'
because execution completedSolution
Added a
resultType
field to categorize execution results, providing better visibility into what actually happened during execution:Result Type Categories
success
: All tasks completed successfullyfailed
: Complete execution failure (player status = 'error')partialFailed
: Some tasks failed but execution continued (continueOnError = true)notExecuted
: Not executed due to previous failuresThis categorization allows distinguishing between complete failures and partial failures where tasks failed but execution continued.
Changes
resultType
field: New field inMidsceneYamlConfigResult
interface to categorize execution resultsprocessResults()
to determine and set appropriate result types based on player and task statusesgetFailedFiles()
,getSuccessfulFiles()
, etc. to useresultType
for filteringcontinueOnError: true
Example
With this fix, a YAML configuration like:
Will now show:
Before:
After:
The execution still continues due to
continueOnError: true
, and the statistics now provide clear categorization of different failure types.Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.