Suite setup and setup failures are poorly handled

[jamesagardner]

Repurposing this issue to capture the following behavior:

If a suiteSetup.sjs file throws an error, the tests still run and can pass successfully. But the suiteSetup error is reported as a failed test, which doesn't make sense.

If a setup.sjs file throws an error, the tests still run but will always be marked as having failed - even if all their assertions succeeded.

[rjrudin]

Keeping this open - I'm not sure the solution is the right approach here, but this behavior is definitely buggy in that the failed suite setup is reported as a failed test, which doesn't make sense.

I do agree with the approach of "If the suite setup or setup fails, mark every test as having failed". But that could be considered a breaking change.

[rjrudin]

Needs a little more research to identify exactly what all should be changed.

[rjrudin]

For 1.4.0, I'm thinking we should at a minimum make sure that failures in setup/suiteSetup are not counted as test failures, as that's misleading.

At least for consistency with things like JUnit / pytest, I think that a failure in either should result in each associated test counting as a failure. So:

• If suiteSetup fails, don't invoke setup or any test, and treat each test as a failure.
• If setup fails, don't invoke any test, and treat each test as a failure.

That seems like the most expected behavior to me.

I'll find out what's up with teardown modules as well, though I believe that generally in other frameworks - if an error occurs there, it does not cause the test to fail, but is logged.

[jamesagardner]

I agree with this direction. I think another important aspect for either a setupSetup or a setup failure is what's displayed in the UI. It should be obvious to the user that they need to investigate suiteSetup or setup for the particular test failures and not be misled into thinking there is an issue within the individual test files. I don't remember what happens off the top of my head right now when the setup fails. But it's non obvious to a casual unit test user that they need to look at the suiteSetup or setup files and not the test itself.

…

On Thu, Aug 10, 2023 at 9:14 AM Rob Rudin ***@***.***> wrote: For 1.4.0, I'm thinking we should at a minimum make sure that failures in setup/suiteSetup are not counted as test failures, as that's misleading. At least for consistency with things like JUnit / pytest, I think that a failure in either should result in each associated test counting as a failure. So: - If suiteSetup fails, don't invoke setup or any test, and treat each test as a failure. - If setup fails, don't invoke any test, and treat each test as a failure. That seems like the most expected behavior to me. — Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBA7LUEKCK2RLAGECX4KLLXUTNCVANCNFSM4IY3FI7Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dmcassel]

This direction sounds good. An added note: when a setup or test script fails, I'd like to see the stack trace in the UI like we do for a test case. That would make it a lot easier to resolve those problems.

[brent-hartwig]

I bumped into this when setup wasn't able to load a test file. I ended up:

1. Adding a try/catch in setup to ensure details were captured.
2. Having my first assert check that the file is available. When it isn't, a clear message showed up in the UI.

Not terribly practical should there be a lot of setup.