coverage: Allow specifying coverage flags via a yaml file

[Dhruv-Maradiya]

PR Description

Overview

• Fixes Allow specifying coverage flags via a yaml file #532
• Implemented functionality to specify coverage flags via a YAML file.
• Deprecated flags are ignored and not used from the YAML file.
• Functionality only allows specifying common flags.

Concerns

• parseArgs Handling:
  The parseArgs function is defined as private, preventing it from being imported into test files. As a workaround, I created a minimal version for testing purposes. I am wondering if there is a more efficient way to handle this.

• Test Coverage:
  I have added test cases to validate the new functionality. I'm unsure if I have organized the tests properly and would appreciate feedback on their coverage and structure.

---

• I’ve reviewed the contributor guide and applied the relevant portions to this PR.

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:coverage	1.12.0	ready to publish	coverage-v1.12.0
package:graphs	2.3.3-wip	WIP (no publish necessary)
package:boolean_selector	2.1.2	already published at pub.dev
package:browser_launcher	1.1.3	already published at pub.dev
package:cli_config	0.2.1-wip	WIP (no publish necessary)
package:cli_util	0.4.2	already published at pub.dev
package:clock	1.1.2	already published at pub.dev
package:code_builder	4.10.1	already published at pub.dev
package:benchmark_harness	2.3.1	already published at pub.dev
package:csslib	1.0.2	already published at pub.dev
package:extension_discovery	2.1.0	already published at pub.dev
package:file	7.0.2-wip	WIP (no publish necessary)
package:file_testing	3.1.0-wip	WIP (no publish necessary)
package:bazel_worker	1.1.3-wip	WIP (no publish necessary)
package:html	0.15.5	already published at pub.dev
package:io	1.0.5	already published at pub.dev
package:json_rpc_2	3.0.3	already published at pub.dev
package:markdown	7.3.1-wip	WIP (no publish necessary)
package:mime	2.0.0	already published at pub.dev
package:oauth2	2.0.4-wip	WIP (no publish necessary)
package:package_config	2.1.1	already published at pub.dev
package:pool	1.5.2-wip	WIP (no publish necessary)
package:pub_semver	2.1.5	already published at pub.dev
package:pubspec_parse	1.5.0	already published at pub.dev
package:source_map_stack_trace	2.1.3-wip	WIP (no publish necessary)
package:yaml_edit	2.2.2	already published at pub.dev
package:source_span	1.10.1	already published at pub.dev
package:sse	4.1.7	already published at pub.dev
package:stack_trace	1.12.1	already published at pub.dev
package:stream_channel	2.1.4	already published at pub.dev
package:stream_transform	2.1.2-wip	WIP (no publish necessary)
package:string_scanner	1.4.1	already published at pub.dev
package:term_glyph	1.2.3-wip	WIP (no publish necessary)
package:test_reflective_loader	0.2.3	already published at pub.dev
package:timing	1.0.2	already published at pub.dev
package:unified_analytics	7.0.1	already published at pub.dev
package:watcher	1.1.1	already published at pub.dev
package:yaml	3.1.3	already published at pub.dev
package:source_maps	0.10.14-wip	WIP (no publish necessary)

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[github-actions]

PR Health

Breaking changes ✔️

Package	Change	Current Version	New Version	Needed Version	Looking good?
coverage	Non-Breaking	1.11.1	1.12.0-wip	1.12.0	✔️

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage ✔️

File	Coverage
pkgs/coverage/bin/collect_coverage.dart	💚 56 %
pkgs/coverage/bin/format_coverage.dart	💚 49 % ⬆️ 831 %
pkgs/coverage/bin/test_with_coverage.dart	💚 36 %
pkgs/coverage/lib/src/coverage_options.dart	💚 100 %

This check for test coverage is informational (issues shown here will not fail the PR).

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbols

License Headers ⚠️

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
pkgs/coverage/lib/src/coverage_options.dart
pkgs/coverage/test/collect_coverage_config_test.dart
pkgs/coverage/test/config_file_locator_test.dart

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/bazel_worker/benchmark/benchmark.dart
pkgs/bazel_worker/example/client.dart
pkgs/bazel_worker/example/worker.dart
pkgs/benchmark_harness/integration_test/perf_benchmark_test.dart
pkgs/boolean_selector/example/example.dart
pkgs/clock/lib/clock.dart
pkgs/clock/lib/src/clock.dart
pkgs/clock/lib/src/default.dart
pkgs/clock/lib/src/stopwatch.dart
pkgs/clock/lib/src/utils.dart
pkgs/clock/test/clock_test.dart
pkgs/clock/test/default_test.dart
pkgs/clock/test/stopwatch_test.dart
pkgs/clock/test/utils.dart
pkgs/html/example/main.dart
pkgs/html/lib/dom.dart
pkgs/html/lib/dom_parsing.dart
pkgs/html/lib/html_escape.dart
pkgs/html/lib/parser.dart
pkgs/html/lib/src/constants.dart
pkgs/html/lib/src/encoding_parser.dart
pkgs/html/lib/src/html_input_stream.dart
pkgs/html/lib/src/list_proxy.dart
pkgs/html/lib/src/query_selector.dart
pkgs/html/lib/src/token.dart
pkgs/html/lib/src/tokenizer.dart
pkgs/html/lib/src/treebuilder.dart
pkgs/html/lib/src/utils.dart
pkgs/html/test/dom_test.dart
pkgs/html/test/parser_feature_test.dart
pkgs/html/test/parser_test.dart
pkgs/html/test/query_selector_test.dart
pkgs/html/test/selectors/level1_baseline_test.dart
pkgs/html/test/selectors/level1_lib.dart
pkgs/html/test/selectors/selectors.dart
pkgs/html/test/support.dart
pkgs/html/test/tokenizer_test.dart
pkgs/pubspec_parse/test/git_uri_test.dart
pkgs/stack_trace/example/example.dart
pkgs/watcher/test/custom_watcher_factory_test.dart
pkgs/yaml_edit/example/example.dart

This check can be disabled by tagging the PR with skip-license-check.

[liamappelbe]

Looks like there's some test failures. See if you can reproduce them locally.

https://github.com/dart-lang/tools/actions/runs/12686464325/job/35499098457?pr=1954

[Dhruv-Maradiya]

Hi @liamappelbe

I've implemented the mentioned changes, and all test cases passed successfully locally. I look forward to your review. Thank you!

[coveralls]

Pull Request Test Coverage Report for Build 13134770139

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 43 of 43 (100.0%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.4%) to 93.454%

---

Totals
Change from base Build 13093867902:	0.4%
Covered Lines:	671
Relevant Lines:	718

---

💛 - Coveralls

[liamappelbe]

Almost there. Just a few more comments.

[liamappelbe]

Implementation looks good. Reviewing the tests now.

There's one more interesting bit of logic that deserves to be better tested: _findOptionsFilePath. Make _findOptionsFilePath public for testing, then write a unit test (in a separate test file) that tests all its edge cases. You'll also need to make some more directory structures and yaml files (maybe in a new directory under the test/ directory to keep things simple):

• Typical case: walk up a few directories to find the pubspec.yaml, and find the coverage_options.yaml file next to it
• No config case: walk up a few directories to find the pubspec.yaml, but there's no coverage_options.yaml file
• Edge case: when there's no pubspec.yaml file at all (this is tricky to test because any walk in the test directory is going to find package:coverage's pubspec, so maybe start in Directory.systemTemp)

The other issue with testing this function is that it starts in Directory.current, which is not safe to alter in a test. So you'll need to change the currentDir variable into an argument, and pass Directory.current when invoking it from _getOptionsFile. Then you can pass whatever you need to in the unit tests.

[liamappelbe]

Almost done. Looks like the analyzer bot wants some fields in the test pubspecs. I think at a minimum you'll need the name and environment fields. Make sure to dart analyze them before you upload, so you don't have to wait for the GitHub bots to verify you did it right.

[liamappelbe]

Looks good. I'm ready to approve. But it looks like some tests are failing on CI. I can see you're passing recursive: true when you create the output file, so I'm not sure why this is failing. You might need to do some debugging. Does it repro locally?

[Dhruv-Maradiya]

Hey @liamappelbe,
I understand the issue. Before, we weren't using the actual argument parser from /bin, but now we are, and it’s creating files if they don’t exist (bin/collect_coverage.dart 146:40). For example: test/test_coverage_options/coverage. However, the directory is read-only.

Here’s what we can do to fix it:

1. Add these files and push them to the repository.
2. Move the syncing logic to a higher layer in /bin.

Let me know your thoughts.

[liamappelbe]

@Dhruv-Maradiya You shouldn't need to add the files, and it would undermine the test slightly. I definitely have tests that create files on github CI. Maybe it just doesn't like creating directories? Git doesn't track empty directories, so even if they exist in your local repo, they won't exist on the bot. Try adding some empty files to those directories and commit them so that git tracks those directories. See if that fixes the issue.

[Dhruv-Maradiya]

Hey @liamappelbe
I agree that adding those files will slightly undermine the test cases, I'm aware that git doesn't track empty directories but also it was using coverage as directly which is ignored by the .gitignore file, so I renamed it to coverage_data.

Running test cases for coverage package will create some files which are not ignored by git, do we need to add them in gitignore?

[liamappelbe]

Running test cases for coverage package will create some files which are not ignored by git, do we need to add them in gitignore?

Yeah, best to gitignore them. Otherwise if you run the tests locally you'll end up with a bunch of files that git wants to track.

[Dhruv-Maradiya]

Hey @liamappelbe
I see test cases still failing on macos and ubuntu with PathAccessException. I think problem is that you cannot modify subdirectories like test, bin or lib but I see some test cases create var directory and modify it so I changed our test data directory to var/coverage_data. I think it should work now.

[liamappelbe]

@Dhruv-Maradiya looks like the analyzer bot is failing, which prevents the tests from running, so not sure if your fix works. But I definitely have tests in other repos that create files in subdirectories of test/ (eg dart-lang/native/ffigen), so I think something else is going on. Maybe the dart-lang/tools repo is more locked down than the dart-lang/native repo?

@mosuem Do you know if there's anything special about the dart-lang/tools repo that would prevent a test from creating files?

[mosuem]

Do you know if there's anything special about the dart-lang/tools repo that would prevent a test from creating files?

No, there is nothing set up here different to any other repo AFAIK. Weird. I have also never heard of not being able to modify any specific subdirectories..

Maybe something about relative/absolute paths? Just a wild guess, but this has bitten me many times before :D Or path separators if it only fails on some OSs.

[Dhruv-Maradiya]

Okay, I'm currently using Windows and unable to reproduce the issue. Additionally, all test cases have successfully executed on Windows on the bot. I'll run them on macOS locally next to try and find the issue.

[Dhruv-Maradiya]

Maybe something about relative/absolute paths? Just a wild guess, but this has bitten me many times before :D Or path separators if it only fails on some OSs.

You are right, @mosuem! The issue was that I specified the path as /var/coverage_data. On Ubuntu and macOS, this defaults to an absolute path. Removing the leading / fixed the issue.

Thanks! 🙌