Test Coverage is Empty when Files are not in lib/ directory

[stefanhk31]

After upgrading a Dart Frog project to version 1.28.0 of test, I noticed that coverage was not getting outputted for files in my routes directory--running coverage would produce an empty lcov.info file. When I rolled back to version 1.25.8, the lcov.info once again had data.

Dart SDK version: 3.9.0

Here is a repository with a minimal reproduction.

Steps:

1. Download repository
2. Run dart test -j 4 --coverage=coverage --platform=vm && dart pub global run coverage:format_coverage --lcov --in=coverage --out=coverage/lcov.info --report-on=foo and verify that coverage/lcov.info has data
3. In pubspec.yaml, change test from 1.25.8 to 1.28.0
4. Run dart pub get
5. Run dart test -j 4 --coverage=coverage --platform=vm && dart pub global run coverage:format_coverage --lcov --in=coverage --out=coverage/lcov.info --report-on=foo, and see that coverage/lcov.info is now empty

[mrverdant13]

I think this is no longer an issue in test 1.29.0

[felangel]

The problem is 1.29.0 is incompatible with the latest stable version of Flutter (3.41.0) because flutter_test is pinned to test_api 0.7.8

[felangel]

This also affects non Dart Frog projects so it seems like coverage reporting broke when going from test_api: 0.7.7 to test_api: 0.7.8 and there is an inconsistency between flutter test --coverage and dart test --coverage with flutter_test being significantly more reliable.

[felangel]

Related to #2570

[felangel]

This is still an issue in test 1.29.0 :(
cc @kevmoo @camsim99

Looking into it more now. Seems like the bug was introduced in 1.27.0 of package:test.

[camsim99]

Thanks for looking into it @felangel. I'll see if I can escalate the triage of this issue!

[felangel]

@camsim99 I looked into it a bit more and it looks like the line that causes the regression is

test/pkgs/test_core/lib/src/runner/vm/platform.dart

Line 430 in fe41d54

await _filterCoveragePackages(config.coveragePackages),

It seems like it was originally introduced in https://github.com/dart-lang/test/pull/2517/changes#r2806436414 by @liamappelbe.

Previously collect was called without scopedOutput and the newly introduced filter now excludes files that are outside of the lib directory which were previously being included.

The following one line change restores coverage reporting in the reproduction sample above:

Future<Map<String, dynamic>> _gatherCoverage(
  Environment environment,
  Configuration config,
) async {
  final isolateId =
      Uri.parse(
        environment.observatoryUrl!.fragment,
      ).queryParameters['isolateId'];
  return await collect(
    environment.observatoryUrl!,
    false,
    false,
    false,
-   await _filterCoveragePackages(config.coveragePackages),
+  {},
    isolateIds: {isolateId!},
    branchCoverage: config.branchCoverage,
  );
}

[felangel]

I did confirm that the breaking change was introduced in #2517 as part of pkg:test_core and I do think that just reverting the one line above resolves the issue.

[felangel]

@camsim99 @liamappelbe @kevmoo I opened a PR with a fix.