From db58ba646ef154ed1c1d5bd4c7307f188e99acfe Mon Sep 17 00:00:00 2001 From: Bassam Khouri Date: Wed, 10 Dec 2025 13:27:10 -0500 Subject: [PATCH] SE-0501: Update proposal base on review Update the SE-0501 proposal based on feedback from the review --- .../0501-swiftpm-html-coverage-report.md | 244 ++++++++++++------ 1 file changed, 165 insertions(+), 79 deletions(-) diff --git a/proposals/0501-swiftpm-html-coverage-report.md b/proposals/0501-swiftpm-html-coverage-report.md index c896e031b2..4ab40859ab 100644 --- a/proposals/0501-swiftpm-html-coverage-report.md +++ b/proposals/0501-swiftpm-html-coverage-report.md @@ -5,7 +5,7 @@ * Review Manager: [David Cummings](https://github.com/daveyc123) * Status: **Active Review (December 7 - December 22, 2025)** * Implementation: [swiftlang/swift-package-manager#9076](https://github.com/swiftlang/swift-package-manager/pull/9076) -* Review: +* Review: * [Pitch](https://forums.swift.org/t/pitch-adding-html-coverage-support/82358) * [Review](https://forums.swift.org/t/se-0501-html-coverage-report/83601/1) @@ -14,26 +14,25 @@ Currently, `swift test` supports generating a JSON coverage report, which is great for ingesting into various systems. The JSON, however, is not very -"human-readable" while iterating at-desk. +"human-readable" during iterative development at-desk. -This proposes adding an additional command line argument to `swift test` that -would allow the caller to select the generation of an HTML coverage report. +This proposal introduces an additional command line argument for `swift test` that +enables users to generate HTML coverage reports. ## Motivation -JSON coverage report is great for ingesting into external tools that post-process -the coverage data. If SwiftPM could generate an HTML coverage report: - - said report can be uploaded to CI systems for visual inspection - - developer can generate the report at-desk, giving faster feedback to determine - if the current changes are sufficiently covered to their liking. +JSON coverage reports are well-suited for ingestion into external tools that post-process +coverage data. The ability for SwiftPM to generate HTML coverage reports would provide: + - Reports that can be uploaded to CI systems for visual inspection + - Immediate feedback for developers during development, enabling rapid assessment of test coverage adequacy for current changes ## Proposed solution -If users currently want an HTML report, the user must manually construct the -`llvm-cov` binary directly with the correct command line arguments. +Currently, users requiring an HTML report must manually invoke the +`llvm-cov` binary with the appropriate command line arguments. -e.g.: +For example: ```sh ❯ swift test --enable-code-coverage @@ -48,28 +47,26 @@ e.g.: "Sources" ``` -Since SwiftPM currently orchestrates the JSON coverage data, the solution adds a -new command line argument, e.g.: `--coverage-format` to `swift test` which can -be specified multiple times, to generate multiple coverage report type from a +Since SwiftPM currently orchestrates JSON coverage data generation, this proposal introduces +a new command line argument `--coverage-format` for `swift test`. This option can +be specified multiple times to generate multiple coverage report types from a single test execution. -While processing the coverage data, SwiftPM will loop on all the unique coverage +While processing the coverage data, SwiftPM will loop through all the unique coverage format options to generate the specified reports. -Unless otherwise specified, this proposal applies only to the HTML Coverage -report. The generation of the JSON Coverage report is unchanged and is out of -scope. +Unless otherwise specified, this proposal addresses only HTML coverage report +generation. The existing JSON coverage report functionality remains unchanged and +is outside the scope of this proposal. ## Detailed design -Existing tools in LLVM have been around for several years (and maybe even decades), -and provide robust tools for code coverage analysis. The LLVM tools are -well-documented, and have been used in production for many years. SwiftPM will -make use of LLVM's tools and construct the proper command line arguments to the -`llvm-cov show` utility, which will generate the HTML report. - -The proposted command line changes are as follows: +LLVM provides mature, well-documented tools for code coverage analysis that have been +extensively used in production environments. SwiftPM will leverage these existing +LLVM tools by constructing the appropriate command line arguments for the +`llvm-cov show` utility to generate HTML reports. +The proposed command line changes are as follows: ### Format Selection @@ -81,68 +78,82 @@ specified. The command line option will be similar to: ```sh - --codecov-format, --code-coverage-format, --coverage-format + --coverage-format Format of the code coverage output. Can be specified multiple times. (default: json) json - Produces a JSON coverage report. html - Produces an HTML report produced by llvm-cov. ``` -### Coverage Report configuration - -`llvm-cov show` has several report configurability options. In order to -prevent a "command line arguments" explosion to `swift test`, the configuration -options will be read from a response file. The optional response file will be -located in `/.swiftpm/configuration/coverage.html.report.args.txt`. The -response file will be supported. +### Coverage report configuration -The user can include `--format=text`, or a variation thereof, in the response -file. In order to ensure SwiftPM will always generate an HTML report, SwiftPM -will add `--format=html` after the response file argument to ensure `llvm-cov` -will generate an HTML report. +The `llvm-cov show` utility provides extensive configurability options. To enable +HTML coverage report customization while avoiding command line argument proliferation +in `swift test`, a `-Xcov` command line option will be introduced. These +arguments will be passed through directly to the underlying `llvm-cov` executable in the +order specified. +The `-Xcov` arguments will be supported for all coverage formats. -SwiftPM will not perform any validation on the response file contents, except -to determine the output location. +Since multiple coverage report formats can be specified in a single `swift test` invocation, +a mechanism must be provided to specify which arguments apply to specific formats. -### Coverage report location +Consider the following example: -By default, the HTML report will be created in location under the scratch path -(ie: the build directory). However, this can be overridden using the response file. +``` +swift test --enable-coverage --coverage-format html --coverage-format json -Xcov --title -Xcov "My title" +``` -Some CI system, such as [Jenkins](https://www.jenkins.io), only allow archiving -contents files/directories that belong in a "sandbox" location. It can be a safe -assumption that the CI system will have a copy of the repository in the "sandbox" -location, allowing this system to upload the HTML report. + +The value of `-Xcov` follows this syntax: ``` - --show-codecov-path [mode], --show-code-coverage-path [mode], --show-coverage-path [mode] - Print the path of the exported code coverage files. The mode specifies how to - display the paths of the selected code coverage file formats. (default: text) - json - Display the output in JSON format. - text - Display the output as plain text. - --show-codecov-path-mode, --show-code-coverage-path-mode, --show-coverage-path-mode - --enable-codecov, --enable-code-coverage, --enable-coverage/--disable-codecov, --disable-code-coverage, --disable-coverage - Enable code coverage. (default: --disable-codecov) - --codecov-format, --code-coverage-format, --coverage-format - Format of the code coverage output. Can be specified multiple times. (values: json, html; default: Produces a JSON coverage report.) +-Xcov [=] ``` +Here are some examples: + +- `-Xcov html=--title`: argument `--title` is only sent to the HTML coverage report generation +- `-Xcov json=myarg`: argument `myarg` is only sent to the JSON coverage report +- `-Xcov commonArg`: the argument `commonArg` is sent to all coverage format reports +- `-Xcov notASupportedFormat=value`: the argument `notASupportedFormat=value` is sent to all coverage format reports as the `` is an unsupported format + + +### Coverage report location + +By default, the HTML report will be created in a location under the scratch path +(i.e.: the build directory). However, this can be overridden using the `-Xcov` argument. + +Certain CI systems, such as [Jenkins](https://www.jenkins.io), restrict archiving +to content files and directories within designated sandbox locations. Since CI systems +typically maintain repository copies within these sandbox environments, this constraint +allows for HTML report uploading while maintaining security boundaries. + ### Show coverage path -Prior to this proposal `swift test --show-coverage-path` would display a single -absolute path location to the JSON coverage report. -Since `--coverage-format` can be specified multiple times, it's output must be +Prior to this proposal, `swift test --show-coverage-path` displays a single +absolute path to the JSON coverage report location. + +Since `--coverage-format` can be specified multiple times, its output must be changed to reflect the new functionality. If the `--coverage-format` option is specified on the `swift test` command line a single time (or is not specified at all), there is no change to the output. -if `--coverage-format` is specified multiple times, the output must reflect this. -The `--show-coverage-path` command line argument will be modified to be an optional -with a default value. The supported values are `json` or `text`, with the default -being `text` to preverse existing behaviour. +When `--coverage-format` is specified multiple times, the output must reflect this capability. +The `--show-coverage-path` command line argument will be enhanced to accept an optional +parameter with a default value. The supported values are `json` or `text`, with the default +being `text` to preserve existing behavior. + +The help text for the `--show-coverage-path` option with its default flag is: +``` + --show-coverage-path [] + Print the path of the exported code coverage files. The mode specifies how to + display the paths of the selected code coverage file formats. (default: text) + json - Display the output in JSON format. + text - Display the output as plain text. +``` A value of `json` will output a JSON object with the key representing the format, and the value representing the output location of said format. @@ -177,13 +188,70 @@ Build of product 'swift-test' complete! (0.40s) } ``` +### Consolidate coverage options to use same argument style + +Prior to this feature, there are 2 coverage option formats: + +``` + --show-codecov-path, --show-code-coverage-path, --show-coverage-path + Print the path of the exported code coverage JSON + --enable-code-coverage/--disable-code-coverage + Enable code coverage. (default: + --disable-code-coverage) +``` + +Currently, there are 3 ways to display the coverage path. This proposal recommends +consolidating all coverage command line options into a single, more comprehensive option: + +``` + --show-coverage-path [] + Print the path of the exported code coverage files. The mode specifies how to + display the paths of the selected code coverage file formats. (default: text) + json - Display the output in JSON format. + text - Display the output as plain text. + --enable-coverage/--disable-coverage + Enable code coverage. (default: --disable-coverage) +``` + +This change requires a graceful deprecation path for the previous options in favor of the new unified approach. + +### Coverage command line options + +The following represents the complete coverage command line options: + +``` +COVERAGE OPTIONS: + --show-coverage-path [] + Print the path of the exported code coverage files. (values: json, text; default as flag: text) + --show-codecov-path, --show-code-coverage-path + Print the path of the exported code coverage files. (deprecated. use `--show-coverage-path []` instead) + --enable-coverage/--disable-coverage + Enable code coverage. (default: --disable-coverage) + --enable-code-coverage/--disable-code-coverage + Enable code coverage. (deprecated. use '--enable-coverage/--disable-coverage' instead) + --coverage-format + Format of the code coverage output. Can be specified multiple times. (default: json) + json - Produces a JSON coverage report. + html - Produces an HTML report produced by llvm-cov. + -Xcov Pass flag, with optional format specification, through to the underlying coverage report tool. Syntax: '[=]'. Can be specified multiple times. +``` + +In addition, for consistency, the `swift build` coverage option help will be modified to the following: + +``` + --enable-coverage/--disable-coverage + Enable code coverage. (default: --disable-coverage) + --enable-code-coverage/--disable-code-coverage + Enable code coverage. (deprecated. use '--enable-coverage/--disable-coverage' instead) +``` + ## Security -Since SwiftPM will use `llvm-cov show`, there may be security implications from -the `llvm-cov` utility, but this is outside of the Swift organizations control. +SwiftPM's use of `llvm-cov show` may inherit security implications from the underlying +utility, which falls outside the Swift organization's direct control. -The LLVM project has a [LLVM Security Response Group](https://llvm.org/docs/Security.html), -which has a process for handling security vulnerabilities. +The LLVM project maintains an [LLVM Security Response Group](https://llvm.org/docs/Security.html) +with established processes for addressing security vulnerabilities. ## Impact on existing packages @@ -191,17 +259,35 @@ No impact is expected. ## Alternatives considered -- In addition to the response file, the coverage report generation can support - a command line argument similar to `-Xlinker`, `-Xcc` and others, which will - pass the arguments to `llvm-cov show` and override the values in the response - file. This has _not_ been implemented in the [PR]. -- Instead of having a `--show-codecov-path` as a tri-state, we could preserve - `--show-codecov-path` original behaviour and add an additional command line - argument to indicate the output mode. The comand line argument would be - `--show-codecov-path-mode `, where `` is either `text` or `json`. - This was not favours as `--show-codecov-path-mode` would have a dependency on - `--show-codecov-path` argument, and may lead to some confusion. + +### `--show-coverage-path` alternative -[PR]: https://github.com/swiftlang/swift-package-manager/pull/9076 +An alternative approach would preserve the original `--show-coverage-path` behavior +while introducing an additional command line argument to specify output mode. This +command line argument would be `--show-coverage-path-mode `, where `` is either `text` or `json`. +This approach was rejected because `--show-coverage-path-mode` would create a dependency on +the `--show-coverage-path` argument, potentially leading to user confusion.