Skip to content

SE-0501: Update proposal base on review #3052

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
bkhouri wants to merge 2 commits into
base: main
Choose a base branch
from
+112 −39
Draft

SE-0501: Update proposal base on review #3052

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3e7eec0
SE-0501: Update proposal base on review
bkhouri Dec 10, 2025
a8b81a8
updates.. still need to clean up
bkhouri Jan 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 112 additions & 39 deletions proposals/0501-swiftpm-html-coverage-report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* Review Manager: [David Cummings](https://github.com/daveyc123)
* Status: **Active Review (December 7 - December 22, 2025)**
* Implementation: [swiftlang/swift-package-manager#9076](https://github.com/swiftlang/swift-package-manager/pull/9076)
* Review:
* Review:
* [Pitch](https://forums.swift.org/t/pitch-adding-html-coverage-support/82358)
* [Review](https://forums.swift.org/t/se-0501-html-coverage-report/83601/1)

Expand Down Expand Up @@ -70,7 +70,6 @@ make use of LLVM's tools and construct the proper command line arguments to the

The proposted command line changes are as follows:


### Format Selection

The `swift test` command line will have an option named `--coverage-format`,
Expand All @@ -81,7 +80,7 @@ specified.
The command line option will be similar to:

```sh
--codecov-format, --code-coverage-format, --coverage-format <format>
--coverage-format <format>
Format of the code coverage output. Can be specified multiple times. (default: json)
json - Produces a JSON coverage report.
html - Produces an HTML report produced by llvm-cov.
Expand All @@ -90,45 +89,47 @@ The command line option will be similar to:

### Coverage Report configuration

`llvm-cov show` has several report configurability options. In order to
prevent a "command line arguments" explosion to `swift test`, the configuration
options will be read from a response file. The optional response file will be
located in `<repo>/.swiftpm/configuration/coverage.html.report.args.txt`. The
response file will be supported.
`llvm-cov show` has several report configurability options. In order to allow
for HTML coverage report customization, and to prevent a "command linearguments"
explosion to `swift test`, a `-Xcov` command line option will be added. These
argument will be passed directly to the underlying `llvm-cov` executable in the
order they appear.

Since multiple coverage report can be specified in a single `swift test` invocation,
we must provide a way to ensure which argument is passed to one format, and not the other.

Take this example

```
swift test --enable-coverage --coverage-format html --coverage-format json -Xcov --title -Xcov "My title"
```

The `-Xcov` arguments are ony supported when generating the HTML report. The value of `-Xcov` will have the following syntax

The user can include `--format=text`, or a variation thereof, in the response
file. In order to ensure SwiftPM will always generate an HTML report, SwiftPM
will add `--format=html` after the response file argument to ensure `llvm-cov`
will generate an HTML report.
```
-Xcov [<coverage-format>=]<value>
```

Here are some examples:

- `-Xcov html=--title`: argument `--title` is only sent to the HTML coverage report generation
- `-Xcov json=myarg`: argument `myarg` is only sent to the JSON coverage report
- `-Xcov commonArg`: the argument `commonArg` is sent to all coverage format reports.
- `-Xcov notASupportedFormat=value`: the argument `notASupportedFormat=value` is sent to all coverage format reports.

SwiftPM will not perform any validation on the response file contents, except
to determine the output location.

### Coverage report location

By default, the HTML report will be created in location under the scratch path
(ie: the build directory). However, this can be overridden using the response file.
(ie: the build directory). However, this can be overridden using `-Xcov` argument.

Some CI system, such as [Jenkins](https://www.jenkins.io), only allow archiving
contents files/directories that belong in a "sandbox" location. It can be a safe
assumption that the CI system will have a copy of the repository in the "sandbox"
location, allowing this system to upload the HTML report.

```
--show-codecov-path [mode], --show-code-coverage-path [mode], --show-coverage-path [mode]
Print the path of the exported code coverage files. The mode specifies how to
display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
--show-codecov-path-mode, --show-code-coverage-path-mode, --show-coverage-path-mode <show-codecov-path-mode>
--enable-codecov, --enable-code-coverage, --enable-coverage/--disable-codecov, --disable-code-coverage, --disable-coverage
Enable code coverage. (default: --disable-codecov)
--codecov-format, --code-coverage-format, --coverage-format <format>
Format of the code coverage output. Can be specified multiple times. (values: json, html; default: Produces a JSON coverage report.)
```

### Show coverage path

Prior to this proposal `swift test --show-coverage-path` would display a single
absolute path location to the JSON coverage report.

Expand All @@ -144,6 +145,15 @@ The `--show-coverage-path` command line argument will be modified to be an optio
with a default value. The supported values are `json` or `text`, with the default
being `text` to preverse existing behaviour.

The help test of the `--show-coverage-path` default as flage optional is:
```
--show-coverage-path [<mode>]
Print the path of the exported code coverage files. The mode specifies how to
display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
```

A value of `json` will output a JSON object with the key representing the format,
and the value representing the output location of said format.

Expand Down Expand Up @@ -177,6 +187,50 @@ Build of product 'swift-test' complete! (0.40s)
}
```

### Consolidate coverage option to use same argument "style"

Prior to this feature, there are 2 coverage options

```
--show-codecov-path, --show-code-coverage-path, --show-coverage-path
Print the path of the exported code coverage JSON
--enable-code-coverage/--disable-code-coverage
Enable code coverage. (default:
--disable-code-coverage)
```

There are 3 ways to show the coverage path. This proposal recommends to modify all
coverage command line option to be a single, more useful option

```
--show-coverage-path [<mode>]
Print the path of the exported code coverage files. The mode specifies how to
display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
--enable-coverage/--disable-coverage
Enable code coverage. (default: --disable-coverage)
```

This requires gracefully phasing out the previous option in favour of the new one.

### Coverage command line options

TODO: update this command line options.
```
COVERAGE OPTIONS:
--show-coverage-path [<mode>]
Print the path of the exported code coverage files. The mode specifies how to
display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
--enable-coverage/--disable-coverage
Enable code coverage. (default: --disable-coverage)
--coverage-format <format>
Format of the code coverage output. Can be specified multiple times. (values: json, html; default: Produces a JSON coverage report.)
-Xcov <Xcov> Pass flag through to the underlying coverage report tool. Syntax is `[<format>=]<value>`
```

## Security

Since SwiftPM will use `llvm-cov show`, there may be security implications from
Expand All @@ -191,17 +245,36 @@ No impact is expected.

## Alternatives considered

- In addition to the response file, the coverage report generation can support
a command line argument similar to `-Xlinker`, `-Xcc` and others, which will
pass the arguments to `llvm-cov show` and override the values in the response
file. This has _not_ been implemented in the [PR].

- Instead of having a `--show-codecov-path` as a tri-state, we could preserve
`--show-codecov-path` original behaviour and add an additional command line
argument to indicate the output mode. The comand line argument would be
`--show-codecov-path-mode <mode>`, where `<mode>` is either `text` or `json`.
This was not favours as `--show-codecov-path-mode` would have a dependency on
`--show-codecov-path` argument, and may lead to some confusion.
<!-- ### Using `-Xcov`-style argument

In addition to the response file, the coverage report generation can support
a command line argument similar to `-Xlinker`, `-Xcc` and others, which will
pass the arguments to `llvm-cov show` and override the values in the response
file.

Once benefit of having a response file in the repository is the ability of
generating a repeatable HTML report. In addition, since `llvm-cov` has many
subcommands, we would need careful considering on how to handle the case where
we demand JSON and HTML report, but the associated `llvm-cov` subcommand does
not support all the `-Xcov` arguments provided via the `swift test` command line
option.

The intent is that when demand an HTML coverage report, the same HTML report is
produced for all users. Granted, this does not prevent a user from modifying the
response file arguments and create a "temporary HTML" report, but the repository
will be the source of truth for these HTML report command options.

As a result, it was decided to only support a response file, where said response
file is in a given location relative to the repository root.

Support for `-Xcov` (or similar) is to be made in a subsequent proposal.
-->
### `--show-coverage-path` alternative

[PR]: https://github.com/swiftlang/swift-package-manager/pull/9076
Instead of having a `--show-coverage-path` as a tri-state, we could preserve
`--show-coverage-path` original behaviour and add an additional command line
argument to indicate the output mode. The comand line argument would be
`--show-coverage-path-mode <mode>`, where `<mode>` is either `text` or `json`.
This was not favoured as `--show-coverage-path-mode` would have a dependency on
`--show-coverage-path` argument, and may lead to some confusion.