TRX reporter is not reporting the hook failures

[goelankit]

The TRX reporter is not handling the hook failures correctly.

The condition here in "suite end" handler seems to be incorrect.
runner.on('suite end', (suite) => {
if (failedHook && failedHook.parent === suite) {

This is what I am seeing in debug console.

failedHook.parent
{root: false, title: '#test1()'}
root:false
title:'#test1()'
proto:Object

suite
{_bail: false, root: false, title: '#test1()', fullTitle: ƒ}
_bail:false
fullTitle:() => value
root:false
title:'#test1()'
proto:Object

failedHook.parent === suite
false

Since the failedHook.parent is not equal to suite, hook failures doesn't get reported in the trx reports.

[pmoleri]

Hi @goelankit.
I think that condition is to prevent reporting it wrong in some nesting scenarios.
There's a unit test with failing hooks which is passing, but perhaps it's different to your case.

Could you provide more details about your case?

• mocha version
• which kind of hook you are using

If you could attach a minimum reproducible example it would be great.

Thanks.

[goelankit]

Hi @pmoleri ,

I think it might not be working as expected in the mocha parallel mode. I see that if I run the test in sequenced mode the report come up fine.
BTW, the same issue if the test is marked as skipped in parallel mode, the test is marked as "Inconclusive" in the output report, that also breaks our upstream workflows.

Mocha version: 8.0.1
Using "before" hook.

Please find attached the minimal repro for this.

1. Run "npm install"
2. Run "npm run test" to run test in parallel mode, observe Result.trx doesn't have failure entry (Not OK)
3. Run "npm run test-seq" to run test in sequenced mode, observe Result.trx has failure entry (OK)

Similar for Skipped test showing up as "Inconclusive" in parallel mode.

1. Uncomment this.skip (line 10) in test1.js.
2. Run "npm run test" to run test in parallel mode, observe Result.trx doesn't have test marked as pending (Not OK)
3. Run "npm run test-seq" to run test in sequenced mode, observe Result.trx has the test marked as pending (OK)

mochatest-trx.zip

We have our tests configured to run in parallel mode, and really looking forward to be able to consume mocha-trx-reporter in parallel mode as well.

[pmoleri]

@goelankit
I see there are some caveats in parallel mode.
It seems it's giving us a simplified object, i.e. instead of Suite class object we get a plain JS object with only some properties.
That means that hook.parent and suite are different objects and that we don't get suite.eachTest() method, so we don't have any way to get the list of tests that were skipped due to a failed "before all" hook.

Also consider that reporting failed hooks is something that I included in this reporter, but I haven't seen in other reporters.

To sum up:
For "before each" hook, we can set the error in each test.
For "before all" hook, I don't know if there's any way to report them in TRX, because we don't have any information about the tests that did not run.

[goelankit]

@pmoleri , I think the issue is happening due to how the pending information for the test is surfaced between sequenced and parallel execution (which is perhaps a mocha issue). In parallel run, test.isPending() is how the info is available.

If I change line 97 in test-to-trx.js to,

if (test.pending === true || test.isPending() === true) {

I see the output trx report getting generated fine for for parallel runs as well now with this change. The test gets marked correctly as pending now instead of inconclusive.

I was not able to push a branch to the repo, any chance you can take a look at the change and get it in. Thanks in advance!

test-to-trx.zip

[goelankit]

Hi @pmoleri , Gentle reminder on this. Can you please check my previous reply and incorporate the change (zip file attached) in the code. Thanks!

[v-hep]

Hi @pmoleri , Gentle reminder on this. Can you please check @goelankit 's previous reply and incorporate the change (zip file attached) in the code. Thanks!