Description

every other testing suite that i've come across in my career (including, since it's a peer-ish library to mocha, jest) will consistently report results for all tests on each run - more specifically, it will show a test failure (or error, which is distinct from a "failure", depending on the framework) for each test in the suite(s) you're trying to execute, regardless of whether that failure occurred in a setup phase or in the test itself. mocha, however, appears to roll test setup failures that occur in before/beforeEach hooks into a single "failure", effectively swallowing counts of all the tests that were sidelined by the setup failure. this makes it rather frustrating to compare outputs between runs, understand the number of tests impacted by that setup failure at a glance, and can make reporting metrics from test runs somewhat meaningless due to the variance in the number of tests actually being reported on.

additionally, it seems that a failure in beforeEach will just circuit-break the execution of that setup function entirely, instead of retrying for subsequent tests, which honestly contravenes the name and intent of the function in the context of software testing conventions.

Steps to Reproduce

• on any test suite with a before or beforeEach hook, introduce an intentional failure in the hook.
  • for your reference, i've thrown together a simple demonstration
    • with failing before
    • with failing beforeEach
• run npm test
• --reporter options don't appear to have any affect on this

Expected behavior:

• a suite containing N tests (regardless of how they're nested in subsuites) should report on the outcome of N tests, even if there is a failure in a before or beforeEach hook.
• examples (modified from output of failing runs) of what i'd expect to see:
  • before
  • beforeEach

Actual behavior: [What actually happens]

• a suite with a failure in a before or beforeEach hook will only report a single failure.
  • before
  • beforeEach

Reproduces how often: 100% - it seems to be by design..? so perhaps this should be a feature request instead?

Versions

• mocha --version: 8.0.1
• node node_modules/.bin/mocha --version: 8.0.1
• node --version: v12.18.1
• Operating system
  • name and version: osX Catalina 10.15.5
  • architecture (32 or 64-bit): x64
• Shell (e.g., bash, zsh, PowerShell, cmd): zsh 5.7.1 (x86_64-apple-darwin19.0)
• browser and version: N/A
• Any third-party Mocha-related modules (and their versions): N/A
• Any code transpiler (e.g., TypeScript, CoffeeScript, Babel) being used (and its version): none