Console output should clarify when a test is unfinished

[briancroom]

Motivation

When running a large set of tests from the command line, if there is a single test that is long-running or has even hung it can be difficult to tell what is happening from visually scanning the console output. The "test started" event for the long-pole test may be located somewhere early in the output, followed by lots of output about other tests and suites beginning and ending. Then, once all those other tests have finished, the console output will typically go quiet, without a clear indication of the state of the run. It would be helpful if we had a way to more clearly indicate where things stand once this state has been reached.

Proposed solution

Some ideas of things we could consider that might help:

• Emit some indicator of overall progress together with other events like tests/suites beginning and ending.
• Emit a periodic "heartbeat" or progress update after a period of time where no console output has been produced.
• Respond to external requests for emitting a progress update (e.g. SIGINFO, SIGUSR1)

Alternatives considered

Test timeouts are one important tool that's related, however they are not in effect by default, and timeouts are generous, still leaving significant gap in time between the point when the console output goes quiet and when the timeout fires.

Additional information

No response

[grynspan]

This is trivially solved by us actually setting the default test timeout via swift test.

[stmontgomery]

This is trivially solved by us actually setting the default test timeout via swift test.

I’m not sure that’s quite the same though. This request is about showing a user which test(s) are still in-flight/running, regardless of whether they're subject to a time limit.

[stmontgomery]

Tracked by rdar://156413505

[grynspan]

There's not really a mechanism in place for us to say at some arbitrary point that the only running tests are ones that have overrun. I imagine we could come up with something but I'm not entirely certain what it would look like.

Setting the default timeout will cause the library to log when a test overruns its time limit. That's at least a minimum step forward, although the logging will be intermixed with other logging.

We could set a signal handler for Ctrl+T (SIGINFO) that logs info about in-flight tests to stderr. That might not be a bad quality-of-life change anyway, although writing safe signal handlers is "fun".

[grynspan]

Alas, because swift test spawns a child process without creating a PTY or signal forwarding or anything like that, we can't implement SIGINFO support for typical workflows without first making significant changes to SwiftPM.