Skip to content

Test suite failure hard to distinguish from test failure #250

@achingbrain

Description

@achingbrain

The JSON output is:

  1. Test suite started:
{"Time":"...","Action":"start","Package":"Gateway Tests"}
  1. A test started running:
{"Time":"...","Action":"run","Package":"Gateway Tests","Test":"TestMetadata"}
  1. The test emitted some output:
{"Time":"...","Action":"output","Package":"Gateway Tests","Test":"TestMetadata","Output":"=== RUN   TestMetadata\n"}
  1. The test passed:
{"Time":"...,"Action":"pass","Package":"Gateway Tests","Test":"TestMetadata","Elapsed":0}
  1. The test failed:
{"Time":"...","Action":"fail","Package":"Gateway Tests","Test":"TestMetadata","Elapsed":0}

So far, so good, but:

  1. The test suite failed:
{"Time":"...","Action":"fail","Package":"Gateway Tests","Elapsed":0.002}

So the "fail" action is duplicated for both individual failure and test suite failure. You have to infer that the event refers to the overall test suite failure by the absence of the "Test" property or some other method.

The results would be more straightforward to consume if the test suite failure/success event had a different "Action" name to the success/failure of individual tests.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions