Add how-to guides

Author: ConorMacBride

docs/configuration.rst

@@ -36,8 +36,8 @@ To enable image comparison testing, pass ``--mpl`` when running pytest.
 
    pytest --mpl
 
-By default, this option will enable :ref:`baseline image comparison <image-comparison-mode>`.
-:ref:`Baseline hash comparison <hash-comparison-mode>` can be enabled by configuring the :ref:`hash library configuration option <hash-library>`.
+By default, this option will enable :doc:`baseline image comparison <image_mode>`.
+:doc:`Baseline hash comparison <hash_mode>` can be enabled by configuring the :ref:`hash library configuration option <hash-library>`.
 
 Without this option, the tests will still run.
 However, the returned figures will be closed without being compared to a baseline image or hash.
@@ -90,6 +90,7 @@ Directory containing baseline images
 The directory containing the baseline images that will be compared to the test figures.
 The kwarg option (``baseline_dir``) is relative to the test file, while the CLI option (``--mpl-baseline-path``) is relative to where pytest was run.
 Absolute paths can also be used.
+If the directory does not exist, it will be created along with any missing parent directories.
 
 .. code:: bash
 
@@ -196,9 +197,12 @@ File containing baseline hashes
 The file containing the baseline hashes that will be compared to the test figures.
 Both the kwarg option (``hash_library``) and the CLI option (``--mpl-hash-library``) are relative to the test file.
 The file must be a JSON file in the same format as one generated by ``--mpl-generate-hash-library``.
+If its directory does not exist, it will be created along with any missing parent directories.
 
 Configuring this option disables baseline image comparison.
-If you want to enable both hash and baseline image comparison, which we call :ref:`"hybrid mode" <hybrid-mode>`, you must explicitly set the :ref:`baseline directory configuration option <baseline-dir>`.
+If you want to enable both hash and baseline image comparison, which we call :doc:`"hybrid mode" <hybrid_mode>`, you must explicitly set the :ref:`baseline directory configuration option <baseline-dir>`.
+
+.. _controlling-sensitivity:
 
 Controlling the sensitivity of the comparison
 =============================================
@@ -212,6 +216,8 @@ It can be useful to pin versions of Matplotlib and FreeType so as to avoid autom
 The ``pytest-mpl`` configuration options in this section allow you to control the sensitivity of the comparison.
 Adjusting these options *may* allow tests to pass across a range of Matplotlib and FreeType versions.
 
+.. _tolerance:
+
 RMS tolerance
 -------------
 | **kwarg**: ``tolerance=<value>``
@@ -231,7 +237,7 @@ The specified tolerance value can be a float or an integer between 0 and 255.
 .. rubric:: How the RMS difference is calculated
 
 Result images and baseline images are *always* converted to PNG files before comparison.
-Each are read as a 4-dimensional array of RGBA pixels (or just RGB if fully opaque) with values between 0 and 255.
+Each are read as an array of RGBA pixels (or just RGB if fully opaque) with values between 0 and 255.
 If the result image and the baseline image have different aspect ratios, the test will always fail.
 The RMS difference is calculated as the square root of the mean of the squared differences between the result image and the baseline image.
 If the RMS difference is greater than the tolerance, the test will fail.
@@ -352,6 +358,9 @@ Directory to write testing artifacts to
 The directory to write result images and test summary reports to.
 The path is relative to where pytest was run.
 Absolute paths are also supported.
+If the directory does not exist, it will be created along with any missing parent directories.
+
+.. _results-always:
 
 Whether to save result images for passing tests
 -----------------------------------------------
@@ -370,7 +379,7 @@ This option is applied automatically when generating a HTML summary.
 
 .. rubric:: Relevance to "hybrid mode"
 
-When in :ref:`"hybrid mode" <hybrid-mode>`, a baseline image comparison is only performed if the test fails hash comparison.
+When in :doc:`"hybrid mode" <hybrid_mode>`, a baseline image comparison is only performed if the test fails hash comparison.
 However, enabling this option will force a comparison to the baseline image even if the test passes hash comparison.
 This option is useful for always *comparing* the result images against the baseline images, while only *assessing* the tests against the hash library.
 This secondary comparison will **not** affect the success status of the test, but any failures (including diff images) will be included in generated summary reports.
@@ -379,6 +388,8 @@ Some projects store their baseline images in a separate repository, and only kee
 This means that they cannot update the baseline images until after the PR is merged.
 Enabling this option allows them to ensure the hashes are correct before merging the PR, but also see how the PR affects the baseline images, as the diff images will always be shown in the HTML summary.
 
+.. _generate-summary:
+
 Generate test summaries
 -----------------------
 | **kwarg**: ---
@@ -408,4 +419,8 @@ The summaries will be written to the :ref:`results directory <results-path>`.
 When generating a HTML summary, the ``--mpl-results-always`` option is automatically applied.
 Therefore images for passing tests will also be shown.
 
-For more information, see the :doc:`summary reports documentation <summaries>`.
+For examples of how the summary reports look in different operating modes, see:
+
+* :doc:`image_mode`
+* :doc:`hash_mode`
+* :doc:`hybrid_mode`

docs/hash_mode.rst

@@ -0,0 +1,83 @@
+.. title:: Hash comparison mode
+
+#####################
+Hash Comparison Mode
+#####################
+
+This how-to guide will show you how to use the hash comparison mode of ``pytest-mpl``.
+
+In this mode, the hash of the image is compared to the hash of the baseline image.
+Only the hash value of the baseline image, rather than the full image, needs to be stored in the repository.
+This means that the repository size is reduced, and the images can be regenerated if necessary.
+This approach does however make it more difficult to visually inspect any changes to the images.
+
+If your goal is to not commit any images to the code repository, then you should consider using :doc:`hybrid mode <hybrid_mode>` instead.
+In this mode, the hashes can be stored in the code repository, while the baseline images are stored in a separate repository and accessed through a URL when testing.
+
+Generating baseline hashes
+==========================
+
+Once a suite of image comparison tests have been written, baseline hashes should be generated by setting ``--mpl-generate-hash-library``:
+
+.. code-block:: bash
+
+    pytest --mpl-generate-hash-library=your_project/tests/hashes.json
+
+It is important to visually inspect the figures before generating baseline hashes.
+So, as well as generating baseline hashes, this command runs baseline image comparison tests.
+If no baseline images exist in the default directory, this command will fail.
+
+A better option is to generate baseline images along with the baseline hashes to ensure that the images are as expected, even if you do not wish to use them for comparison:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-generate-path=baseline
+
+To assist with inspecting the generated images (and hashes), a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-hash-library=test_hashes.json \
+      --mpl-generate-path=baseline \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_generate`
+
+You should choose a directory within you repository to store the baseline hashes.
+It's usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. ``mpl35_ft261.json``.
+The hash library file should then be committed to the repository.
+
+Running hash comparison tests
+=============================
+
+When running the tests, the ``--mpl`` flag should be used along with a :ref:`configured hash library path <hash-library>` to enable baseline hash comparison testing:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json
+
+Optionally, a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_hashes_only`
+
+The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored.
+If this is not set, the images will be stored in a temporary directory.
+
+Continue reading
+================
+
+``pytest-mpl`` has many configuration options that can be used to customize the behavior of the hash comparison mode.
+Only a few of the most commonly used options are covered in this guide.
+See the :doc:`configuration options documentation <configuration>` for full details.

docs/hybrid_mode.rst

@@ -0,0 +1,90 @@
+.. title:: Hybrid mode
+
+##############################
+Hybrid Mode: Hashes and Images
+##############################
+
+This how-to guide will show you how to use the hybrid mode of ``pytest-mpl``.
+
+For a full description of the hybrid mode, see the :ref:`hybrid mode section of the get started guide <hybrid-usage>`.
+In summary, hybrid mode uses both baseline images and hashes.
+First, the hash of the image is compared to the hash of the baseline image.
+If the hashes match, the test passes.
+If the hashes do not match, the test fails.
+
+The difference with hybrid mode is that a baseline image comparison will also be carried out if the hashes do not match, or always :ref:`if this has been configured <results-always>`.
+The purpose of the additional image comparison (which does not affect the test result) is to allow the user to visually inspect the difference between the baseline image and the image generated by the test.
+
+In order to keep the code repository size small, it is recommended to store the baseline hashes in the code repository, and the baseline images in a separate repository.
+The baseline hashes should be updated where appropriate in PRs to the code repository.
+However, the baseline images are not updated in these PRs.
+Instead, they should be updated once the PR has been merged, preferably by a CI job.
+
+Another benefit of only updating the baseline images once the PR has been merged is that the PR tests will show the difference between the remote baseline images and the images generated by the PR.
+Even though the tests will pass when the baseline hash matches, the images will still be compared and the difference will be shown in the HTML test summary report, which is useful when reviewing the PR.
+
+Generating baseline hashes and images
+=====================================
+
+Once a suite of image comparison tests have been written, baseline hashes and images should be generated by setting ``--mpl-generate-path`` and ``--mpl-generate-hash-library``:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-hash-library=your_project/tests/test_hashes.json \
+      --mpl-generate-path=baseline \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_generate`
+
+Open the HTML summary file and inspect the figures to ensure that the baseline images are correct.
+If they are, the baseline hashes can be committed to the code repository.
+It's usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. ``mpl35_ft261.json``.
+The baseline images should be copied to a separate repository; preferably within a version specific directory, e.g. ``mpl35_ft261/``.
+
+Running hash comparison tests
+=============================
+
+When running the tests, the ``--mpl`` flag should be used along with a configured :ref:`hash library path <hash-library>` and :ref:`baseline image path <baseline-dir>` to enable hybrid mode testing:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html`
+
+The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored.
+If this is not set, the images will be stored in a temporary directory.
+
+Notice that the baseline image path is set to a URL, which is the location of the baseline images in the separate repository.
+When the baseline image comparison is carried out, the baseline images will be downloaded from this URL.
+
+It is recommended to create a CI job that updates the baseline images in the separate repository once the PR has been merged.
+The CI job should run when new commits are pushed to the default branch.
+The baseline images should only be regenerated and updated if the tests pass in the hash comparison mode.
+
+.. rubric:: Aside: basic HTML summary
+
+This is what the basic HTML summary looks like for the same test as above:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-hash-library=your_project/tests/mpl35_ft261.json \
+      --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \
+      --mpl-results-path=results \
+      --mpl-generate-summary=basic-html,json
+
+:summary:`test_basic_html`
+
+Continue reading
+================
+
+``pytest-mpl`` has many configuration options that can be used to customize the behavior of the hybrid mode.
+Only a few of the most commonly used options are covered in this guide.
+See the :doc:`configuration options documentation <configuration>` for full details.

docs/image_mode.rst

@@ -0,0 +1,70 @@
+.. title:: Image comparison mode
+
+#####################
+Image Comparison Mode
+#####################
+
+This how-to guide will show you how to use the image comparison mode of ``pytest-mpl``.
+This is the default mode when ``pytest-mpl`` is invoked with ``--mpl``.
+
+In this mode, ``pytest-mpl`` will compare the images generated by the test with a baseline image.
+If the images are different, :ref:`to a specified RMS tolerance <tolerance>`, the test will fail.
+The test will also fail if the baseline image is not found or if the test does not generate a figure.
+If the figure has a different aspect ratio to the baseline image, the test will also fail.
+
+Generating baseline images
+==========================
+
+Once a suite of image comparison tests have been written, baseline images should be generated by setting ``--mpl-generate-path``:
+
+.. code-block:: bash
+
+    pytest --mpl-generate-path=your_project/tests/baseline
+
+You should choose a directory within you repository to store the baseline images.
+The generated images should be checked to ensure they are as expected.
+The images should then be committed to the repository.
+
+To assist with inspecting the generated images, a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest \
+      --mpl-generate-path=your_project/tests/baseline \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_generate_images_only`
+
+Running image comparison tests
+==============================
+
+When running the tests, the ``--mpl`` flag should be used to enable baseline image comparison testing:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-baseline-path=your_project/tests/baseline
+
+The :ref:`baseline path <baseline-dir>` should be set to the directory containing the baseline images.
+If the baseline images are not found, the test will fail.
+
+Optionally, a HTML summary report can be generated by setting ``--mpl-generate-summary``:
+
+.. code-block:: bash
+
+    pytest --mpl \
+      --mpl-baseline-path=your_project/tests/baseline \
+      --mpl-results-path=results \
+      --mpl-generate-summary=html,json
+
+:summary:`test_html_images_only`
+
+The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored.
+If this is not set, the images will be stored in a temporary directory.
+
+Continue reading
+================
+
+``pytest-mpl`` has many configuration options that can be used to customize the behavior of the image comparison mode.
+Only a few of the most commonly used options are covered in this guide.
+See the :doc:`configuration options documentation <configuration>` for full details.

docs/index.rst

@@ -7,8 +7,10 @@
 
     installing
     usage
+    image_mode
+    hash_mode
+    hybrid_mode
     configuration
-    summaries
 
 ##################################
 pytest-mpl |release| documentation
@@ -57,21 +59,25 @@ Learning resources
         Tutorials
         ^^^
 
-        - :doc:`Basic usage <usage>`
+        - :doc:`Get started <usage>`
 
     .. grid-item-card::
         :padding: 2
 
         How-tos
         ^^^
 
+        - :doc:`Image comparison mode <image_mode>`
+        - :doc:`Hash comparison mode <hash_mode>`
+        - :doc:`Hybrid mode <hybrid_mode>`
 
     .. grid-item-card::
         :padding: 2
 
         Understand how pytest-mpl works
         ^^^
 
+        Explanatory information is included where relevant throughout the documentation.
 
     .. grid-item-card::
         :padding: 2
@@ -80,14 +86,12 @@ Learning resources
         ^^^
 
         - :doc:`Configuration <configuration>`
-        - :doc:`Summary Reports <summaries>`
-
 
 ************
 Contributing
 ************
 
-pytest-mpl is a community project maintained for and by its users.
+``pytest-mpl`` is a community project maintained for and by its users.
 There are many ways you can help!
 
 - Report a bug or request a feature `on GitHub <https://github.com/matplotlib/pytest-mpl/issues>`__