Skip to content

Conversation

@cuihantao
Copy link
Contributor

Previously, TestItemRunner used try/finally without catch, causing any error in a testitem to crash the entire test run. This made it impossible to see results from other testitems when one failed with a load error.

Changes:

  • Add catch block to record errors as Test.Error(:nontest_error, ...)
  • Matches behavior of Julia's @testset macro error handling
  • Fix compute_line_column testitem to properly import the function

Now when a testitem throws an error (e.g., UndefVarError from missing import), the error is recorded and the test run continues to execute remaining testitems.

🤖 Generated with Claude Code

I have reviewed the changes and tested it locally. The changes to compute_line_column is to fix the failing tests. It was not related to my changes.

Previously, TestItemRunner used try/finally without catch, causing any
error in a testitem to crash the entire test run. This made it impossible
to see results from other testitems when one failed with a load error.

Changes:
- Add catch block to record errors as Test.Error(:nontest_error, ...)
- Matches behavior of Julia's @testset macro error handling
- Fix compute_line_column testitem to properly import the function

Now when a testitem throws an error (e.g., UndefVarError from missing
import), the error is recorded and the test run continues to execute
remaining testitems.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@codecov
Copy link

codecov bot commented Dec 23, 2025

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 92.02%. Comparing base (7ad26e1) to head (b1d8261).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/TestItemRunner.jl 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #121      +/-   ##
==========================================
- Coverage   92.59%   92.02%   -0.57%     
==========================================
  Files           2        2              
  Lines         135      138       +3     
==========================================
+ Hits          125      127       +2     
- Misses         10       11       +1     
Flag Coverage Δ
unittests 92.02% <66.66%> (-0.57%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@davidanthoff davidanthoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@davidanthoff davidanthoff merged commit b5feb7c into julia-vscode:main Dec 23, 2025
48 of 61 checks passed
@cuihantao
Copy link
Contributor Author

Thanks!!

I also have a question about Test.finish(ts_3) in the finally block. It seems to re-throw the complete error information. In my integration test with complex structures, it immediately blows up the AI context window....

The brief versions of errors are shown as they appear before the summary, so I wonder what would be the best way to hide the full error output. Should it be tied to verbose? Any better options?

@moble
Copy link

moble commented Jan 5, 2026

Thanks very much for this! But it did cause one surprising new problem.

One thing to note is that when tests are run via a script, the InterruptException doesn't actually do anything by default. So, for example, I had a whole lot of replicates of a test that was failing, so the script sat there trying every one, even though I knew they would all fail. I tried to CTRL+C out of it, but nothing happened; I had to kill the process.

This is from the help text for InterruptException:

in Julia script started without -i (interactive) option, InterruptException is not thrown by
default. Calling Base.exit_on_sigint(false) in the script can recover the behavior of the REPL.
Alternatively, a Julia script can be started with

julia -e "include(popfirst!(ARGS))" script.jl

to let InterruptException be thrown by CTRL+C during the execution.

I found that when I added Base.exit_on_sigint(false) to my runtests.jl, interruption could actually work correctly.

@cuihantao
Copy link
Contributor Author

Thank you for reporting the issue. Sorry about the issue my PR has caused. I'm not sure about the correct solution and rely on the maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants