pytest-pyvista

Plugin to test PyVista plot outputs.

---

This pytest plugin was generated with Cookiecutter along with @hackebrot's cookiecutter-pytest-plugin template.

Features

This plugin facilitates the comparison of the images produced by PyVista. It generates a cache of images from the tests, using the PyVista plotting function in its first execution. Then, further executions will compare its results against this cache, so if there are any changes in the code that break the image generation, the comparison against the cache will notice it. Note that there is an error tolerance in the comparison, so minor differences won't fail.

Requirements

You must have a Python version >= 3.9, as well as PyVista installed in your environment.

pyvista version >=0.37.0 and vtk version >=9.0.0 required.

Installation

You can install "pytest-pyvista" via pip from PyPI

pip install pytest-pyvista

Alternatively, you can also install via conda or mamba from conda-forge

mamba install -c conda-forge pytest-pyvista

Usage

The plugin has two main use cases:

1. Evaluate images generated by calling pl.show() in unit tests
2. Evaluate images generated by the Sphinx PyVista Plot Directive when building documentation.

Specifying multiple cache images

The cache directory is typically flat with no sub-directories. However, it is possible to specify multiple cache images for a single test by including a sub-directory with the same name as the test, and including multiple "valid" cache images in the sub-directory. For example, a single cached image:

cache/my_image.jpg

Can be replaced with multiple versions of the image:

cache/my_image/0.jpg
cache/my_image/1.jpg

Note

• The sub-directory name should match the name of the test.
• The image names in sub-directories can be arbitrary, e.g. 0.jpg or foo.jpg.
• Nested sub-directories are also supported, and their names can also be arbitrary.
• Use the --generate_subdirs flag to automatically generate test images in a sub-directory format.

When there are multiple images, the test will initially compare the test image to the first cached image. If that comparison fails, the test image is then compared to all other cached images for that test. The test is successful if one of the comparisons is successful, though a warning is still issued if it initially failed.

Both use cases (i.e. unit tests and documentation tests) support specifying multiple cache images.

Unit tests

Once installed, you only need to use the command pl.show() in your test. The plugin will automatically manage the cache generation if it does not exist, and the image comparison itself. Make sure you enable pv.OFF_SCREEN when loading PyVista, so the pl.show() doesn't pop up any window while testing. By default, the verify_image_cache fixture should be used for each test for image comparison:

import pyvista as pv

pv.OFF_SCREEN = True


def test_succeeds(verify_image_cache):
    pl = pyvista.Plotter()
    pl.add_mesh(pyvista.Sphere(), show_edges=True)
    pl.show()

If most tests utilize this functionality, possibly restricted to a module, a wrapped version could be used

@pytest.fixture(autouse=True)
def wrapped_verify_image_cache(verify_image_cache):
    return verify_image_cache

If you need to use any flag inside the tests, you can modify the verify_image_cache object in the test

import pyvista as pv

pv.OFF_SCREEN = True


def test_succeeds(verify_image_cache):
    verify_image_cache.windows_skip_image_cache = True
    pl = pyvista.Plotter()
    pl.add_mesh(pyvista.Sphere(), show_edges=True)
    pl.show()

Global flags

These are the flags you can use when calling pytest in the command line:

• --reset_image_cache creates a new image for each test in tests/plotting/test_plotting.py and is not recommended except for testing or for potentially a major or minor release.
• You can use --ignore_image_cache if you want to temporarily ignore regression testing, e.g. on a particular CI action.
• --generated_image_dir <DIR> dumps all generated test images into the provided directory, relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.
• --generate_subdirs saves generated test images in separate sub-directories instead of saving them directly to the generated_image_dir. Without this option, generated images are saved as generated_image_dir/<test_name>.png; with this option enabled, they are instead saved as <generated_image_dir>/<test_name>/<image_name>.png, where the image name has the format <os-version>_<machine>_<gpu-vendor>_<python-version>_<pyvista-version>_<vtk-version>_<using-ci>. This can be useful for providing context about how an image was generated. See the Test specific flags section for customizing the info.
• --failed_image_dir <DIR> dumps copies of cached and generated test images when there is a warning or error raised. This directory is useful for reviewing test failures. It is relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.
• --add_missing_images adds any missing images from the test run to the cache.
• --image_cache_dir <DIR> sets the image cache directory, relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.
• --reset_only_failed reset the image cache of the failed tests only.
• Use --allow_unused_generated to prevent an error from being raised when a test image is generated but not used. A test image is considered "used" if it has a corresponding cached image to compare against, or is used to reset or update the cache (e.g. if using --add_missing_images). Otherwise, an error is raised by default.
• --disallow_unused_cache report test failure if there are any images in the cache which are not compared to any generated images.
• Use --allow_useless_fixture to prevent test failure when the verify_image_cache fixture is used but no images are generated. If no images are generated (i.e. there are no calls made to Plotter.show() or mesh.plot()), then these tests will fail by default. Set this CLI flag to allow this globally, or use the test-specific flag by the same name below to configure this on a per-test basis.
• Use --image_format to save test images in either png or jpg format. png files are saved by default. Use jpg to reduce the image file size. This will override any configuration, see below.

Test specific flags

These are attributes of verify_image_cache. You can set them as True if needed in the beginning of your test function.

• high_variance_test: If necessary, the threshold for determining if a test will pass or not is incremented to another predetermined threshold. This is currently done due to the use of an unstable version of VTK, in stable versions this shouldn't be necessary.

• windows_skip_image_cache: For test where the plotting in Windows is different from MacOS/Linux.

• macos_skip_image_cache: For test where the plotting in MacOS is different from Windows/Linux.

• skip: If you have a test that plots a figure, but you don't want to compare its output against the cache, you can skip it with this flag.

• allow_useless_fixture: Set this flag to True to prevent test failure when the verify_image_cache fixture is used but no images are generated. The value of this flag takes precedence over the global flag by the same name (see above).

• env_info: Dataclass for controlling the environment info used to name the generated test image(s) when the --generate_dirs option is used. The info can be test-specific or can be modified globally by wrapping the verify_image_cache fixture, e.g.:

  @pytest.fixture(autouse=True)
  def wrapped_verify_image_cache(verify_image_cache):
      # Customize the environment info (NOTE: Default values are shown)
      info = verify_image_cache.env_info
      info.prefix: str = ""  # Add a custom prefix
      info.os: bool = True  # Show/hide the os version (e.g. ubuntu, macOS, Windows)
      info.machine: bool = True  # Show/hide the machine info (e.g. arm64)
      info.gpu: bool = True  # Show/hide the gpu vendor (e.g. NVIDIA)
      info.python: bool = True  # Show/hide the python version
      info.pyvista: bool = True  # Show/hide the pyvista version
      info.vtk: bool = True  # Show/hide the vtk version
      info.ci: bool = True  # Show/hide if generated in CI
      info.suffix: str = ""  # Add a custom suffix
  
      # Alternatively, set a custom string
      verify_image_cache.env_info = 'my_custom_string'
  
      return verify_image_cache

Documentation image tests

Unlike the unit tests, which use the verify_image_cache fixture to evaluate test images during each test's teardown, the documentation tests do not use a fixture. Instead, the documentation tests are invoked with the --doc_mode flag, and requires specifying:

1. A target directory which contains all images to be tested. The directory is specified using the --doc_images_dir flag.
2. A cache directory containing all reference images to compare with. The directory is specified using the --doc_image_cache_dir flag.

Since all images must be available for testing, the documentation tests are typically executed after building documentation with Sphinx or some other build process. To test build images against images in a cache directory use:

pytest --doc_mode --doc_images_dir images --doc_image_cache_dir cache

where images is the target directory of images to test, and cache is the cache directory.

When executed, the test will first pre-process the build images. The images are:

1. Collected from the images directory (including images in nested directories).

2. Resized to a maximum of 400x400 pixels.

3. Renamed so that each file's parent directories are included in the name.

4. Saved as JPEG images with in a temporary directory. The directory is flat with no sub-directories.

   Note

   These temporary images may be saved using the --doc_generated_image_dir flag.

Next, the pre-processed images are compared to the cached images. The tests have three main modes of failure:

1. An image is in the cache but is missing from the build.
2. An image is in the build but is missing from the cache.
3. The error threshold when comparing two images is exceeded.

Note

Use the --doc_failed_image_dir flag to save copies of the images for failed tests.

Global flags

These are the flags you can use when calling pytest in the command line:

• --doc_mode is a required flag for testing documentation images. It configures pytest to only collect tests relevant for the image testing.

• --doc_images_dir <DIR>> sets the target directory of images to be tested.

  Note

  With Sphinx, build images are typically saved to doc/_build/html/_images.

• --doc_image_cache_dir <DIR> sets the doc image cache directory, relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.

• --doc_generated_image_dir <DIR> dumps all doc generated test images into the provided directory, relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.

  Note

  These are the pre-processed images generated for the tests. They are not the images generated by a documentation build (use --doc_images_dir for specifying that).

• --doc_failed_image_dir <DIR> dumps copies of cached and generated test images when there is a warning or error raised. This directory is useful for reviewing test failures. It is relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.

• Use --doc_image_format to save test images in either png or jpg format. png files are saved by default. Use jpg to reduce the image file size. This will override any configuration, see below.

Configuration

If using pyproject.toml or any other pytest configuration section, consider configuring your test directory location to avoid passing command line arguments when calling pytest, for example in pyproject.toml:

[tool.pytest.ini_options]
image_cache_dir = "tests/plotting/image_cache"

Additionally, to configure the directory that will contain the generated test images:

[tool.pytest.ini_options]
generated_image_dir = "generated_images"

Similarly, configure the directory that will contain any failed test images:

[tool.pytest.ini_options]
failed_image_dir = "failed_images"

Configure directories for when --doc_mode is used:

[tool.pytest.ini_options]
doc_failed_image_dir = "failed_test_images"
doc_generated_image_dir = "generated_test_images"
doc_image_cache_dir = "tests/doc/doc_image_cache"
doc_images_dir = "doc/_build/html/_images"

Note that these directories are relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>.

Configure the image format to be jpg for both unit tests and when using --doc_mode.

[tool.pytest.ini_options]
image_format = "jpg"
doc_image_format = "jpg"

Contributing

Contributions are always welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, pytest-pyvista is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.