Jest CTRF Fails to mark test as Failed if there is a syntax issue in test

[simontao18]

In a jest test file, if there are compiler issues in the test itself (i.e. missing imports, missing arguments to a function, etc.), the output summary states that the number of tests = 0 and failed = 0. Would be great to include the "test suite failed to run" tests in the summary as a failed test.

[Ma11hewThomas]

Hello, I can't recreate this issue.

I've created a basic javascript app with two test files containing the same 4 tests. In one of the files, I've removed an import.

The jest default reporter output shows 8 total and 4 failed:

Test Suites: 1 failed, 1 passed, 2 total
Tests:       4 failed, 4 passed, 8 total
Snapshots:   0 total
Time:        0.222 s, estimated 1 s

Which matches the CTRF report summary:

 "summary": {
      "tests": 8,
      "passed": 4,
      "failed": 4,
      "pending": 0,
      "skipped": 0,
      "other": 0,
      "start": 1722890532054,
      "stop": 1722890532275
    }

And the test contains the error message:

{
        "name": "should not remove a task if index is out of bounds",
        "duration": 0,
        "status": "failed",
        "message": "ReferenceError: TaskManager is not defined\n",
}

Please can you provide more details that might help with investigation

Thanks

[simontao18]

It seems the test suite is still run in your test, I'm more referring to the cases where jest itself doesn't execute the test and returns Test suite failed to run due to syntax or other issues. In those cases, the jest summary would output tests = 0, failed = 0, and no testResults array since the test didn't run.

A simple fix would just be to augment theonTestResult function:

if (testResult.testResults.length === 0) {
            // Add to testResults that the test failed to run
            testResult.testResults.push({
                ancestorTitles: [],
                duration: 0,
                failureMessages: [`Test suite failed to run`],
                fullName: test.path,
                location: null,
                invocations: 1,
                numPassingAsserts: 0,
                retryReasons: [],
                status: "failed",
                title: "Test suite failed to run",
            });
        }
        // then call the normal onTestResult summary code