Approach to test coverage

[mzabaluev]

Test coverage is a commonly used metric for quality of code and testing. It can be used as an OKR.

Methodology

Decide what we want to collect coverage on: e2e only, any integration-style tests, or also including the unit tests? There are tradeoffs.

• Coverage of E2E only:

  • ➕ Highlights the code you can really reach in realistic scenarios.
  • ➖ Simulation of some conditions can be difficult.

• Coverage of unit tests:

  • ➖ Potential for testing dead code and irrelevant scenarios, considered good by the coverage metric.
  • ➕ Easier to reach unexpected and hard-to-reproduce conditions.

• Coverage of library "integration" tests (exercising the public API):

  • ➕ Proves quality of the SDK offering.
  • ➖ Does not help with the YAGNI problems, similarly to covering unit tests.

Decision: instrument unit and integration tests (i.e. roughly anything run by cargo test --workspace with no external setup).

Short-term goals

Start with some coverage testing to work out the technical details and set up workflows.

Quality standards and OKR

Would it make sense to commit to a target coverage figure, or would it be shambolic?

A non-decrease in coverage can certainly be a PR quality gate.

[l-monninger]

For integration and unit tests, we can start with https://lib.rs/crates/cargo-llvm-cov to get line and region coverage. Branch and MC/DC are both experimental.

The notion of coverage in e2e tests is generally going to be too abstract I think.

[l-monninger]

@mzabaluev Can we go forth on the above? would you be able to spec this out in an issue? Presumably, we also want coverage badges.

[mzabaluev]

You get #407