Alternative (safer) Test.Runner API?

[kofigumbs]

TLDR: Test.Runner.Exploration

---

Test.Runner has a fairly "query-driven" module which doesn't enforce some of the rules that Test discusses. Here are a couple example:

[todo] tests always fail, but test runners will only include them in their output if there are no other failures.

[skip] your entire test suite will fail, even if each of the individual tests pass

Rules like this could enforced internally to elm-test, which would simplify and protect call sites.
Additionally, an alternative API could let simplify the labeling by enforcing that formatLabels is actually used.

formatFailures : { label : String -> a, error : String -> a } -> Runner -> List (List a)

I suspect an API change could simplify some elm-test internals as well, but I haven't looked through this code base.
I've started to incorporate some of these concerns in html-test-runner's Test.Runner.Exploration.
It's mostly to serve as a discussion point, since there are some obvious changes that need to happen:

• pull in formatLabel as discussed above (aka, don't expose labels : List String)
• add a wrapper : (() -> List Expectation) -> () -> List Expectation param so that node-test-runner can catch exceptions

Test.Runner is currently labeled as "experts only" module, but I don't think it has to be!

[mgold]

This is between @rtfeldman and you. I don't have a huge preference other than that you don't break things.

I think it would be reasonable to add the new functions in a minor release and deprecate (but not remove) the old ones. We're looking at some breaking changes in #183 so it would be good to bundle the removal of the old interface together with that.

[rtfeldman]

I like this idea!

Another thing I think is worth incorporating: I talked about this with @mgold at some point, but I can't remember where it's written down - exposing a union type for the different built-in expectation types, e.g. type ExpectationType = Equal String String | NotEqual String String | Custom String so different runners can render things differently depending on the expectation.

Examples of what this breaking change would let us do:

• For the HTML runner, showing something prettier than the "ASCII art" pipes next to Expect.equal that is currently the only option for displaying the result of Expect.equal
• For the CLI, rendering things differently for JUnit XML output than for console output - the ASCII art looks bizarre in JUnit
• In any test runner, rendering a diff on the two sides of Expect.equal but not on Expect.lessThan

[kofigumbs]

@rtfeldman glad you like the direction. We're actually working on changing the awkward ASCII art in the html runner now 😄.

Thinking about how to move this forward leaves me with a couple of questions:

1. Would this actually work for all of the cli runners in the node implementation? I wonder if it's worth copying the Test.Runner.Exploration into that project and see what tweaks make things more convenient.
2. The new API could be dropped into this project without any modification to current runner by moving the old Test.Runner to something like Test.Runner.Internal. That lets maintainers here work on combining the two without changing the API. Is there a better way to make the change though?

I think this API actually solves #194 outright, which is a nice consequence.

[kofigumbs]

I think html-test-runner is going to folded into Richard's runner, so this issue seems finished.