Support Code Coverage Reporting

[shamrt-v]

• cli-testing-library version: 2.0.2
• Testing Framework and version: Vitest@1.6.1
• DOM Environment: jsdom@25.0.1

Relevant code or config:

> nx run tools:test --coverage


 RUN  v1.6.1 /Users/smartin/code/workspaces/v/vantage/libs/tools
      Coverage enabled with v8

 ✓ src/lib/utils/writeTiersFeatures.spec.ts (12) 1034ms
 ✓ src/bin/tiers-features/addItem.spec.ts (9) 4412ms
 ✓ src/bin/tiers-features/addFeatureToTier.spec.ts (4) 1344ms
 ✓ src/lib/utils/getTiersFeatures.spec.ts (1)
 ✓ src/bin/tiers-features/removeItem.spec.ts (2) 982ms
 ✓ src/lib/utils/getNextPk.spec.ts (6)
 ✓ src/lib/tools.spec.ts (1)

 Test Files  7 passed (7)
      Tests  35 passed (35)
   Start at  13:55:17
   Duration  8.50s (transform 89ms, setup 18ms, collect 356ms, tests 7.77s, environment 157ms, prepare 41ms)

 % Coverage report from v8
-------------------------|---------|----------|---------|---------|-------------------
File                     | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-------------------------|---------|----------|---------|---------|-------------------
All files                |   35.63 |    80.43 |   53.33 |   35.63 |
 src                     |       0 |        0 |       0 |       0 |
  index.ts               |       0 |        0 |       0 |       0 | 1
 src/bin/tiers-features  |       0 |        0 |       0 |       0 |
  addFeatureToTier.ts    |       0 |        0 |       0 |       0 | 1-99
  addItem.ts             |       0 |        0 |       0 |       0 | 1-92
  manage.ts              |       0 |        0 |       0 |       0 | 1-42
  removeItem.ts          |       0 |        0 |       0 |       0 | 1-50
 src/lib                 |   91.66 |       50 |     100 |   91.66 |
  constants.ts           |   90.47 |    33.33 |     100 |   90.47 | 13-14
  tools.ts               |     100 |      100 |     100 |     100 |
 src/lib/utils           |   92.25 |    94.59 |      75 |   92.25 |
  getNextPk.ts           |     100 |      100 |     100 |     100 |
  getTiersFeatures.ts    |     100 |      100 |     100 |     100 |
  includesNameOrValue.ts |       0 |        0 |       0 |       0 | 1-2
  isUniqueKey.ts         |       0 |        0 |       0 |       0 | 1-10
  writeTiersFeatures.ts  |     100 |      100 |     100 |     100 |
-------------------------|---------|----------|---------|---------|-------------------

What you did:

Ran Vitest with --coverage flag (via Nx, which shouldn't affect reports), and do not see coverage for binary test modules (which thoroughly test a CLI wizard). Note that I'm using V8 as my coverage provider.

What happened:

Coverage reports are incomplete/non-existent for cli-testing-library spec modules.

Reproduction:

I'd expect any cli-testing-library code wouldn't provide coverage stats. I don't see anything in the source code that would handle this 👀

Problem description:

This is a gap in the tool, I'd say. Maybe I'm using the wrong tooling or coverage provider. Would istanbul work?

Suggested solution:

🙂 — perhaps this is a feature request

[crutchcorn]

I actually don't know how we'd support this 😅 Any help here would be great, but definitely a feature request

[timofei-iatsenko]

I think currently it running a CLI command using a spawn, which is creating a separate nodejs process. Niether Vitest or Jest can hijack this with code coverage instrumentation. What would work, instead of spawning a separate process run it in the same. So in psedudocode it could be something like that

function execute(progamName: string, cliArguments: string) {
  mockArgv(cliArguments)
  {stdout, stderr} = mockStdOutErr()

  const fileName = resolveBinary(progamName)
  import(fileName) // or require(fileName)

  return {stdout, stderr}
}

For sure it will require some changes on the program side, for example something like require.main === module oftenly used in cli entrypoints would not work. And in general that might be less accurate than launching it using spawn, but it seems there is no other easy way of doing that.

Links:

Vitest is not supporting gathering a coverage from spawned processes, the issue
NYC standalone cli tool actually can do that, maybe it can be used somehow