add option to generate test coverage report without codecov

[zacharyburnett]

using the guide here: https://hynek.me/articles/ditch-codecov-python/

This does not create coverage diffs or report to PRs, just creates a markdown summary table

closes #189

EDIT: here's what the summary table looks like:

[Cadair]

I tried this out on sunpy and it seems to explode. sunpy/sunpy#7668

[Cadair]

It seems that it's not picking up the coverage.xml to upload here: https://github.com/sunpy/sunpy/actions/runs/9445798095/job/26014155532?pr=7668#step:14:16 🤔

[Cadair]

Thanks so much for taking a swing at this!

[Cadair]

I think it would be nice to add the ability to fail this job based on either total coverage % or diff coverage %.

[pllim]

tl;dr -- this does not generate patch coverage, right?

[Cadair]

Not yet, but I found a way to make it.

[zacharyburnett]

tl;dr -- this does not generate patch coverage, right?

Not yet, but I found a way to make it.

For patch coverage we'd need to store the coverage from main somewhere, to compare it to; some people I've seen online have done that by storing it in a hidden page in the repository's wiki, reading and writing it with a github token with ${{ secrets.GITHUB_TOKEN }}

[pllim]

I am trying this out too at spacetelescope/acstools#205 and it still crashes:

https://github.com/spacetelescope/acstools/actions/runs/9463279566/job/26068438067?pr=205

[zacharyburnett]

I am trying this out too at spacetelescope/acstools#205 and it still crashes:

https://github.com/spacetelescope/acstools/actions/runs/9463279566/job/26068438067?pr=205

oh whoops, I fixed the artifact issue:

but didn't pass any filenames to coverage combine 😅

[pllim]

Even when CI passes, it failed with the same error.

Coverage XML written to file coverage.xml

mv: cannot stat '.coverage': No such file or directory

[zacharyburnett]

Even when CI passes, it failed with the same error.

Coverage XML written to file coverage.xml

mv: cannot stat '.coverage': No such file or directory

it should be uploading / downloading the .coverage database, to be able to combine multiple machine types, so if that's not present then something is messed up

[zacharyburnett]

it looks like --cov-report=xml is what's creating the coverage.xml; however, it should still be creating a .coverage database file... I'll so some local testing to see what files it spits out
https://github.com/spacetelescope/acstools/actions/runs/9463279566/job/26118526244#step:10:105

pytest --pyargs acstools --cov-report=xml --cov=acstools /home/runner/work/acstools/acstools/doc --cov --remote-data -v

[zacharyburnett]

running pytest --pyargs acstools --cov-report=xml --cov=acstools --cov --remote-data -v locally does appear to create the .coverage file:

❯ ls -a
.							CITATION.md
..							CODE_OF_CONDUCT.md
.bandit.yaml						LICENSE.md
.coverage						MANIFEST.in
.coverage.urelialia2022.local.stsci.edu.35749.XCMKciAx	README.rst
.git							__pycache__
.github							acstools
.gitignore						acstools.egg-info
.mypy_cache						conftest.py
.pytest_cache						coverage.xml
.readthedocs.yaml					doc
.tmp							pyproject.toml
.tox							tox.ini

[pllim]

What is the difference between .coverage and coverage.xml?

[zacharyburnett]

What is the difference between .coverage and coverage.xml?

I'm not intimately familiar with it, but as far as I know .coverage is a SQLite database file with support for parallel-mode (storing coverage for multiple machine configurations / runs) while coverage.xml is report-focused and so is expected to only contain results from a single run.

We could use coverage.xml, but then we'd need to have a summary table of coverage for each and every run, instead of a single table collating all runs

[pllim]

Tried to debug at #209 but no luck. I see the coverage ran but the the upload failed: https://github.com/spacetelescope/acstools/actions/runs/9483014271/job/26134042811?pr=205

[zacharyburnett]

Tried to debug at #209 but no luck. I see the coverage ran but the the upload failed: https://github.com/spacetelescope/acstools/actions/runs/9483014271/job/26134042811?pr=205

could you also add -a to the ls?

[pllim]

Ah, yes, added. New log: https://github.com/spacetelescope/acstools/actions/runs/9483014271/job/26138000458?pr=205

[zacharyburnett]

@neutrinoceros try now, it should be rebased

[neutrinoceros]

thank you ! now zacharyburnett#1 is much easier to read !
FTR, I did iterate overate it on my own fork, so it should be safe to merge, but feel free to run CI on yours first !

[neutrinoceros]

well of course it just had to fail, right ?
I swear I did check my patch... before the rebase... let me look into this again then

[neutrinoceros]

ok it's coming from tox_matrix.py, which assumes that pytest-cov is used and just appends pytest flags, but that isn't compatible with my approach of just using coverage directly (note that, in contrast, using coverage will work for anyone that assumes pytest-cov is installed)

[Cadair]

Can you fork on the value of the coverage option?

[neutrinoceros]

This is already the case. However it looks to me that I can't drop the assumption that pytest-cov is used without a significant change, because then I cannot simply generate a command by appending arguments. I need to think this over for a bit.

[neutrinoceros]

I'm going to lay out the design constraints I'm seeing:

• tox.yml can only append arguments to a command via matrix.pytest_flag, which tox redirects as posargs
• in order to automatically form a coverage command dynamically, I need to be able to prepend coverage run -m before the pytest invoke
• I would also need to somehow be able to add dynamically generated commands to invoke coverage report or coverage xml ...

This just seems impossible to do in the most general case. I can see why pytest-cov solves these issues but I'm frustrated that it's a de facto requirement. My prefered solution would be to renounce doing this stuff dynamically and let users define their own commands (using pytest-cov or coverage directly, at their discretion); but of course that'd be a breaking change so now I'm kinda stuck.

Anyway, this problem I'm reporting isn't specific to this PR, it's pre-existing. All things considered, it seems reasonable to preserve the assumption that pytest-cov is needed for the time being, and I can revisit this adjacent issue later. I'll open a new PR-to-PR to restore functionality.

[neutrinoceros]

zacharyburnett#2 👀

[neutrinoceros]

I don't understand why it's still failing now. I can spend some more time digging it tommorow. In the mean time, I would suggest adding a debug step to run ls -R to see exactly where artifacts get downloaded

[neutrinoceros]

(my rant about pytest-cov now has its own issue #330)

[neutrinoceros]

Phew, took me a while to get there, but I finally have something that works again: zacharyburnett#3

important notes: I couldn't, for the life of me, figure out where pytest-cov would write coverage data¹, nor how it overwrites (as the documentation claims it does) coverage's --parallel-mode, so I had to resort to cutting the middle man again and invoke coverage directly. The obvious drawback is that users need to do the same thing (coverage run --parallel-mode -m pytest ...) from their tox.ini file to leverage this. But since it's a new feature, there's no backward incompatibility, so it should be acceptable, right ?

Footnotes

1. to some extent, I suspect it might actively delete them ↩

[neutrinoceros]

Oh and I just realized another important limitation : combining coverage data from different platforms is really tricky (TBH, I could never quite figure it out, but I did fiddle with this a couple times and the author claims it's possible). To the extent I understand how it's supposed to work, it should be possible to configure it on the user side.

If we're going to release this in its current state, it should be advertised very clearly.

[neutrinoceros]

Whoops. Guess why I didn't run pre-commit hooks locally ? (a clue: #328)

[neutrinoceros]

Also, there's a job stuck because it's trying to use macos-12, which isn't available anymore. This is already fixed on main though.