Feature Request: Coverage Results Feature

[aschwanden1]

We're talking about Code coverage. this would be on the run.html page.

[hategan]

The basic idea would be something that would allow us to visualize merged coverage results.

The interface should be a simplified version of what is available on codeconv.io. For example, we could do an explorer-like interface, with a filesystem tree on the left and a content display in the main area, such as in the picture below:

Initially, this can display coverage results without merging. The results would be uploaded from the sites running the tests, possibly in json format (see https://coverage.readthedocs.io/en/6.4.4/cmd.html#json-reporting-coverage-json).

A second iteration would take all coverage files for a specific git commit and combine them together to generate a single result and display that.

A prototype implementation of the interface can be done without support from the service, since we can simply serve a json file with coverage results.

[hategan]

It turns out that if you run pytest with coverage, the plugin does not support json output, so we might have to use XML.

To get coverage xml output when running tests, edit the Makefile in psij-python and add --cov-report=xml after --cov in the coverage-tests target (or maybe create a copy of that target first).

[aschwanden1]

Question: would this pane show up when you click a new Icon at the top?

[hategan]

Question: would this pane show up when you click a new Icon at the top?

It should be associated with the latest commit of a PR/branch, so no. It should be linked somewhere in the test results. Best place seems to be in run.html under each branch heading.

However, I think this can be prototyped without linking it in a first iteration.

[aschwanden1]

Ok, that makes sense because afterall we're going to drill down to run.html

[aschwanden1]

Now lets talk about location. I assume you want it to be above the branch boxes but below the breadcrumbs. is that right?

[hategan]

Now lets talk about location. I assume you want it to be above the branch boxes but below the breadcrumbs. is that right?

I meant under the actual branch heading, right before the "Environment" expander. We would have a link saying something like "Coverage: x%" that takes you to the coverage page for that branch.

[aschwanden1]

ok, so we'll have to return some coverage percent to put into the page. and when the user clicks on the coverage link it opens a new tab with the coverage page for that branch.

i'm currently looking for some Vue component to do the file tree. i don't know if we'll find one that's ideal.

as for the right side of the coverage page: the code section, I don't think we need a component to do that, just color the divs green, red or blank.

[aschwanden1]

I assume green means that code was tested and passed. Red means it was tested and failed and white means it wasn't tested at all.

[aschwanden1]

It turns out that if you run pytest with coverage, the plugin does not support json output, so we might have to use XML.

To get coverage xml output when running tests, edit the Makefile in psij-python and add --cov-report=xml after --cov in the coverage-tests target (or maybe create a copy of that target first).

I didn't see "cov" inside the Makefile. Perhaps, you could post the coverage result as a file somewhere, so we know exactly what output we're dealing with.

[hategan]

It turns out that if you run pytest with coverage, the plugin does not support json output, so we might have to use XML.
To get coverage xml output when running tests, edit the Makefile in psij-python and add --cov-report=xml after --cov in the coverage-tests target (or maybe create a copy of that target first).

I didn't see "cov" inside the Makefile. Perhaps, you could post the coverage result as a file somewhere, so we know exactly what output we're dealing with.

https://github.com/ExaWorks/psij-python/blob/main/Makefile#L17-L20

[aschwanden1]

"hits" below as represented in the coverage xml file.
hits=0 is red,
hits > 0 is green,
and everything else is white

[aschwanden1]

I want to clarify, we're running this code coverage on the code base psij-python, correct?

[hategan]

I want to clarify, we're running this code coverage on the code base psij-python, correct?

In the psij dashboard, yes. In the sdk dashboard, there shouldn't be this feature (for now).