Testing across a range of config options

[liz-is]

It would be good to have a thorough set of end-to-end tests that use different config parameters, to make it easier to spot accidental breaking changes and/or weird interactions between options. See tests/_explanation_test.py for an example, that uses snapshot testing to ensure that code changes don't introduce unintended changes in the output, as well as checking that the full workflow runs with the given set of config parameters.

What parameters and parameter values should we focus on for this? Some initial thoughts:

• different model input approaches: onnx, pytorch
• different explanation strategies: global, spatial
• check that --analyse option works for both onnx and pytorch (requires bugfix, see Analyse broken for onnx format  #1 )
• check that all plot types work (but may be difficult to ensure that results are consistent)
• different distribution types for splitting boxes - at the moment probably only uniform works

[liz-is]

• test models with channels at the beginning vs end - C * H * W vs H * W * C
  • this is partly handled by testing onnx vs pytorch
• test different model image sizes
  • find a model that resizes to e.g. 512*512 - but needs something with a quick inference time
• different batch sizes - probably sufficient to test dynamic vs set batch size
  • find an appropriate model with a fixed batch size
• different devices - set up CI to run on Mac and Windows?

[liz-is]

Tests implemented:

• different model inputs
  • onnx
  • pytorch via custom script
• different explanation strategies
  • spatial
  • global
• analyze
  • for pytorch models
  • for onnx models
• plotting
  • heatmap
  • surface
  • save image
• models with channels at the beginning vs end
• different model image sizes
• different batch sizes
• models with fixed vs dynamic batch sizes
• different devices
• different distributions for splitting boxes
  • uniform (default)
  • betabinomial
• spectral data

[liz-is]

See #32 - this PR addresses many, but not all, of the scenarios described here.

[liz-is]

It would be great to have one or more input images where we know roughly where in the image the explanation should be (e.g., top left quadrant). This could be something like the "chimera" images with the red panda, as used in the first ReX paper. This would allow us to write tests where we have a reasonable expectation of the "correct" result, rather than just comparing to a snapshot of the previous results.