Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Sep 29, 2025

Problem

When continueOnError: true is specified in YAML configuration, assertion failures were not being properly counted in the execution summary's "Failed" count. The execution summary would incorrectly show "Failed: 0" even when some tasks contained failed assertions.

Root Cause

The BatchRunner.processResults() method only checked player.status !== 'error' to determine if a file execution was successful. However, when continueOnError: true is used:

  • Individual tasks can fail with status === 'error' (e.g., failed assertions)
  • But the overall player status remains 'done' because execution completed
  • This caused files with failed assertion tasks to be incorrectly counted as successful

Solution

Added a resultType field to categorize execution results, providing better visibility into what actually happened during execution:

Result Type Categories

  • success: All tasks completed successfully
  • failed: Complete execution failure (player status = 'error')
  • partialFailed: Some tasks failed but execution continued (continueOnError = true)
  • notExecuted: Not executed due to previous failures

This categorization allows distinguishing between complete failures and partial failures where tasks failed but execution continued.

Changes

  • Added resultType field: New field in MidsceneYamlConfigResult interface to categorize execution results
  • Updated BatchRunner logic: Modified processResults() to determine and set appropriate result types based on player and task statuses
  • Enhanced execution summary: Added separate "Partial failed" count and display section
  • Improved error reporting: Clear distinction between complete failures (❌) and partial failures (⚠️)
  • Updated helper methods: Modified getFailedFiles(), getSuccessfulFiles(), etc. to use resultType for filtering
  • Added test coverage: New test case specifically validates that failed tasks are correctly categorized with continueOnError: true

Example

With this fix, a YAML configuration like:

tasks:
  - name: failing_assertion
    continueOnError: true
    flow:
      - aiAssert: this page shows weather information  # This fails
  - name: passing_task
    flow:
      - aiAssert: this page shows example content      # This passes

Will now show:

Before:

📊 Execution Summary:
   Total files: 3
   Successful: 2
   Failed: 1        ← All failures counted together
   Not executed: 0

After:

📊 Execution Summary:
   Total files: 3
   Successful: 1
   Failed: 0        ← Complete failures only
   Partial failed: 1 ← Partial failures (continueOnError)
   Not executed: 1

⚠️  Partial failed files (some tasks failed with continueOnError)
   test-with-assertion-failure.yml

The execution still continues due to continueOnError: true, and the statistics now provide clear categorization of different failure types.

Original prompt

This section details on the original issue you should resolve

<issue_title>[Bug]: After an assertion failure, the test case still shows successful execution.</issue_title>
<issue_description>### Version

System:
    OS: Windows 11 10.0.26100
    CPU: (12) x64 AMD Ryzen 5 5500U with Radeon Graphics
    Memory: 1.43 GB / 13.83 GB
  Browsers:
    Edge: Chromium (140.0.3485.54)
    Internet Explorer: 11.0.26100.1882
v0.28.9 | qwen-vl-max-latest(qwen-vl mode)

Details

Image

Reproduce link

增加断言失败的统计

Reproduce Steps

在yaml中指定了continueOnError,虽然有的断言失败了,但是Execution Summary中Failed还为0</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #1237

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copy link

netlify bot commented Sep 29, 2025

Deploy Preview for midscene ready!

Name Link
🔨 Latest commit da82da4
🔍 Latest deploy log https://app.netlify.com/projects/midscene/deploys/68e34c133bd4fc0008a49879
😎 Deploy Preview https://deploy-preview-1265--midscene.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Copilot Copilot AI changed the title [WIP] [Bug]: After an assertion failure, the test case still shows successful execution. fix(cli): track assertion failures correctly with continueOnError=true Sep 29, 2025
@Copilot Copilot AI requested a review from quanru September 29, 2025 12:07
Copilot finished work on behalf of quanru September 29, 2025 12:07
@quanru
Copy link
Collaborator

quanru commented Sep 30, 2025

@copilot A type should be added to categorize these errors instead of directly counting them under failures.

Copy link
Contributor Author

Copilot AI commented Sep 30, 2025

@copilot A type should be added to categorize these errors instead of directly counting them under failures.

Added resultType field to categorize execution results instead of counting everything under "failed". The categorization now includes:

  • success: All tasks completed successfully
  • failed: Complete execution failure (player error)
  • partialFailed: Some tasks failed with continueOnError=true ← NEW category
  • notExecuted: Not executed due to previous failures

The execution summary now displays:

📊 Execution Summary:
   Total files: 3
   Successful: 1
   Failed: 0
   Partial failed: 1    ← Separate count
   Not executed: 1

This provides better visibility into what happened during execution. Changes implemented in commit b2ee934.

@Copilot Copilot AI changed the title fix(cli): track assertion failures correctly with continueOnError=true feat(cli): categorize execution results with resultType for better failure tracking Sep 30, 2025
@quanru quanru marked this pull request as ready for review October 6, 2025 04:56
Copilot AI and others added 3 commits October 6, 2025 12:56
- Added 'resultType' field to MidsceneYamlConfigResult to distinguish between:
  - 'success': All tasks completed successfully
  - 'failed': Complete execution failure (player error)
  - 'partialFailed': Some tasks failed with continueOnError=true
  - 'notExecuted': Not executed due to previous failures

- Updated BatchRunner to track and display partial failures separately
- Added 'Partial failed' count to execution summary
- Updated tests to verify correct categorization

This provides better visibility into what actually happened during execution,
distinguishing between complete failures and partial failures where tasks
failed but execution continued due to continueOnError setting.

Co-authored-by: quanru <[email protected]>
@quanru quanru force-pushed the copilot/fix-1f4697d3-9016-4c2a-9e1e-018b477fde64 branch from b2ee934 to da82da4 Compare October 6, 2025 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: After an assertion failure, the test case still shows successful execution.
3 participants