Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.69%. Comparing base (a829f0f) to head (82cd494).
⚠️ Report is 9 commits behind head on main.

@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
- Coverage   97.69%   97.69%   -0.01%     
==========================================
  Files          21       21              
  Lines        1127     1126       -1     
==========================================
- Hits         1101     1100       -1     
  Misses         26       26              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

I'm happy with the code, but I'm wondering whether this is the right thing to do now: before cementing all the generated docs and figures, we should rather take more care that the examples provide the right messages and that we're happy with the scope.

I want some tests, for example, because a modification to the API can have a strong effect and change totally the example without I notice it.
I need to improve it a bit more inclusion into the CI to determine if it raises just a warning or a blocking the CI.

I'm happy with the code, but I'm wondering whether this is the right thing to do now: before cementing all the generated docs and figures, we should rather take more care that the examples provide the right messages and that we're happy with the scope.

I want some tests, for example, because a modification to the API can have a strong effect and change totally the example without I notice it. I need to improve it a bit more inclusion into the CI to determine if it raises just a warning or a blocking the CI.

But you will notice it easily by eyeballing the examples. There are only 10 atm.

It's not so obvious because before I started the tests, I didn't see the small difference in the example due to the randomisation.

To me, the costs outweigh the benefits at the moment.

It seems to be more sensitive to changes in the dependencies than other tests and thus more maintenance work. But maybe that's the price to pay to have plotting functions.

yeah I would definitely only run those tests with one set of dependency version (like the oldest supported matploblib version) otherwise it's gonna be hell to maintain

Originally posted by @Remi-Gau in #439 (comment)

I think testing the plots of the examples is not desirable due to the randomness of the methods implemented in hidimstat.
The comment of @Remi-Gau applies to testing plotting functions of the library, which can be done reliably by providing deterministic values before calling .plot() (hard-coding the importance values). However, if the values plotted are computed using a stochastic method (like knockoffs), the plots may look slightly different.

I agree that variability is not good but we put some effort in it, it's possible to have full control of it, for example.

The more I think about it, the more I realise that this can be used only to help reviewers to catch the change in the figures.

I agree that variability is not good but we put some effort in it, it's possible to have full control of it, for example.

The more I think about it, the more I realise that this can be used only to help reviewers to catch the change in the figures.

I don't remember: are you building your doc with pinned versions of your dependencies? if not it's probably better to do that to avoid figures changing for reasons outside of your control

We don't pin the version for building the examples.
This should be done at some point, especially if we want to do some testing on the figures.