Improve testset logs?

[christiangnrd]

The way our tests are set up, whenever a a test fails, the error gets attributed to the top-level testset. It would be quite helpful if we could have the specific testset get reported.

For example, in this run, instead of

mps/copy: Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-aarch64-5.0/build/default-macmini-aarch64-5-0/julialang/metal-dot-jl/test/mps/copy.jl:39

it would say

UInt32: Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-aarch64-5.0/build/default-macmini-aarch64-5-0/julialang/metal-dot-jl/test/mps/copy.jl:39

.

Since we already know which file and line to look for, it would be more helpful (assuming appropriately named testsets) to report the deepest testset.

I've looked into it a bit, and with multi-threaded tests, the testset structure seems to get lost when returning. Is there a way to preserve that information so the test log is more descriptive?

[maleadt]

Is there a way to preserve that information so the test log is more descriptive?

I'm not sure, I'm actually not too familiar with the Test stdlib design. Much of the multprocess test runner came from the Base implementation of runtests there, so maybe see if there's been an improvement over there since we forked off that code?

[christiangnrd]

I spent a while digging into this yesterday and it seems like the testset functions behave differently (and more importantly return different things) if they're run from a top-level testset or not. Therefore, by running in a different process, each testset is technically a top-level testset, and then the object recovered from remotecall_fetch doesn't even contain the information required to report the low-level testset that failed.

My current best idea is to basically define our own AbstractTestSet that always behaves as if it weren't top-level, that way all the data gets returned from the worker and then no information is lost.

I'll have a look at Base to see if they've fixed this. I just have to take the time to find a CI run that failed instead of errored.

[christiangnrd]

I do think this is worth the effort because we seem to have lots of hard to reproduce bugs that come up and having the testset tell us right away which type or the array size that failed without having to reproduce would be extremely helpful.
IE this, these, this, etc.

[maleadt]

Sounds good, thanks for taking a look.

FWIW, it's been long on my TODO list to factor this functionality out into (yet another) testing package, hopefully (as opposed to existing packages like ReTestItems.jl or TestItems.jl) trying to be more lightweight by supporting plain @testset and simply running tests as they are defined. So feel free to take the liberty of doing more radical changes to make this code more reusable; it'd be great to share it across our GPU back-ends.

[christiangnrd]

Ok so the POC hack I tried seemed to work amazingly, but my attempt at legitimizing it is proving to be much more complicated than I have the bandwidth for at the moment, and it honestly doesn't really feel all that more legitimate as I'm adding methods to unexported Test.jl functions.

I may revisit at some point but the Test.jl interface does not seem like it was made to be extended.

[christiangnrd]

Pushed current work to the testreporting branch in case anyone wants to take a look/finish it.