Question: stop tests on first failure (--fail-fast)

[schoettl]

What's the best way to cancel the test runner on the first failure (like bash's -e/-o errexit)? Is it even possible?

I looked in RunnerOptions, but I didn't find it here.

In my case, the Tests are actually HUnit Tests which are IO actions (-> shelltestrunner).

I can't even just use die or fail in the IO action because the test runner continues on error:

I'd be glad if an experienced Haskeller could give me a hint!

[sol]

Hspec provides this functionality with --fail-fast. If converting your tests to Hspec is not feasible, you can try to run your existing tests with Hspec by using

• either hspec-test-framework (not used much, not sure if it compiles)
• or better Test.Hspec.Contrib.HUnit, if you can get hold of the actual HUnit test tree

(not sure how well either of these options will work in the context of shelltestrunner, though)

I can't even just use die or fail in the IO action because the test runner continues on error:

Alternatively: If your are set on test-framework, try throwIO UserInterrupt (from Control.Exception). At least HUnit will not catch these. So provided that test-framework behaves sane, it should terminate the process.

[schoettl]

Thanks for your answer! Oh, you are the author of hspec, what an honor :) I used it (and also QuickCheck2) a lot in an early project.

Hspec provides this functionality with --fail-fast. If converting your tests to Hspec is not feasible, you can try to run your existing tests with Hspec by using

• either hspec-test-framework (not used much, not sure if it compiles)

• or better Test.Hspec.Contrib.HUnit, if you can get hold of the actual HUnit test tree

That looks good. As far as I understand, I would have to convert the tests to Hspec and then just call hspec convertedTests in the shelltestrunner's main function. It should work but yield different output than current shelltestrunner.

I've found the options page, some are similar to what shelltestrunner/test-framework use.

But I fear it would be much refactoring work in shelltestrunner, for me. It would probably be less work to add a --fail-fast option in test-framework.

I have to think about if or how shelltestrunner could combine or allow both alternatives: hspec and test-framework. Maybe a --use-hspec option for shelltestrunner which then uses hspec instead and accepts some hspec options.

Alternatively: If your are set on test-framework, try throwIO UserInterrupt (from Control.Exception). At least HUnit will not catch these. So provided that test-framework behaves sane, it should terminate the process.

Good idea, that would be a simple approach. shelltestrunner reports shelltest: user interrupt and stops, but it does not exit until I press Ctrl-C. Don't know if that is a test-framework or shelltestrunner problem. From the output, it looks like test-framework passes the exception to shelltestrunner. Pressing Ctrl-C directly works and does not print the shelltest: user interrupt. Anyway, it's not perfect when shelltestrunner cannot distinguish between user interrupt and first failed test...

[sol]

Good idea, that would be a simple approach. shelltestrunner reports shelltest: user interrupt and stops, but it does not exit until I press Ctrl-C.

I think the issue here is that test-framework runs your tests on a worker thread and does not propagate the exception to the main thread. So my suggestion won't work without modifications to test-framework.