✨ Add on-the-fly Registered Slide Visualization (#875)

This PR allows registration transform to be loaded as an 'overlay', which will allow the slide to be viewed as if that registration transform had been applied to it. This will work with either an affine transformation saved as a .npy, or a non-rigid one represented by a SimpleITK displacement field saved as a .mha (such as what would be output by DeeperHistoReg, for example).

It introduces a new WSIReader subclass called TransformedWSIReader, which when initialized with a slide and transformation will apply the transformation on the fly when regions are requested.

It also adds '.npy' and '.mha' to the overlay types the visualization tool picks up; upon loading one of these types, the current slide will be opened as a TransformedWSIReader with that transform, allowing it to be visualized on the fly as a registered slide.

Tasks:
 - [x] Implement read_bounds (currently only done read_rect)
 - [x] Correctly handle coord_space argument
 - [x] Check affine capabilities of read_bounds
 - [x] Add unit tests

---------

Co-authored-by: Adam Shephard <[email protected]>
Co-authored-by: adamshephard <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <[email protected]>

Author: 4 people

docs/images/dual_win_reg.png

@@ -119,6 +119,55 @@ A filter can be applied to annotations using the filter box. For example, enteri
 
 The main slide view can be made fullscreen by clicking the fullscreen icon in the small toolbar to the immediate right of the main window. This toolbar also provides a button to save the current view as a .png file.
 
+Visualising Image Registration/Transformation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. image:: images/dual_win_reg.png
+    :width: 100%
+    :align: center
+    :alt: dual window example
+
+
+TIAToolbox provides a powerful registration visualization feature that enables intuitive alignment and comparison of histopathology images—such as H&E and IHC-stained slides—without requiring full whole-slide registration. This is particularly useful for quickly inspecting the accuracy of precomputed registration results.
+
+To use this feature, you must supply a precomputed registration matrix (e.g., .mha or .npy file) generated from an affine or deformable registration process. This matrix is used to align the images visually.
+
+Dual Window Mode:
+"""""""""""""""""
+
+This mode allows side-by-side comparison of registered images.
+
+**Steps:**
+
+* Open **Dual Window Mode** and load the images.
+
+* In one window, open the H&E (source) image.
+
+* In the other window, open the IHC (target) image.
+
+* Load the registration file (e.g., an .mha or .npy file) as an overlay on the source image.
+
+Overlay Mode:
+"""""""""""""
+
+This mode overlays the registered image directly on top of the source image for visual inspection.
+
+
+**Steps:**
+
+* Open the H&E (source) image.
+
+* Overlay the IHC (target) image on the source image.
+
+* Load the registration file (e.g., an .mha or .npy file) as an overlay on the source image.
+
+.. note::
+   Always load the **target image first** when using overlays. If not, the system may incorrectly assume both images are the same, leading to inaccurate transformations. Incorrect ordering may result in misaligned overlays or misleading visualizations.
+
+
+The **order** of source and target images must remain consistent with how the registration matrix was computed. This is as most registration algorithms require the dimensions of both the source and target images to perform the registration transformation. The above examples assume that the H&E image is registered to the IHC images, but if instead you have registered the IHC to the H&E image then please change the order of image loading accordingly.
+
+
 .. _data_format:
 
 3. Data Format Conventions and File Structure

docs/visualization.rst

@@ -125,6 +125,11 @@ def annotation_path(data_path: dict[str, Path]) -> dict[str, object]:
         "annotation_dat_svs_1",
         data_path["base_path"] / "overlays",
     )
+    data_path["affine_trans"] = (
+        data_path["base_path"] / "overlays" / (data_path["slide1"].stem + ".npy")
+    )
+    # save eye as test identity transform
+    np.save(data_path["affine_trans"], np.eye(3))
     data_path["config"] = _fetch_remote_sample(
         "config_2",
         data_path["base_path"] / "overlays",
@@ -246,6 +251,31 @@ def test_remove_dual_window(doc: Document, data_path: pytest.TempPathFactory) ->
     assert main.UI["vstate"].slide_path == data_path["slide1"]
 
 
+def test_add_slide_layer(doc: Document, data_path: pytest.TempPathFactory) -> None:
+    """Test adding a non-annotation slide layer."""
+    slide_select = doc.get_model_by_name("slide_select0")
+    slide_select.value = [data_path["slide1"].name]
+
+    layer_drop = doc.get_model_by_name("layer_drop0")
+    slide_layer_path = str(data_path["slide1"])
+
+    click = MenuItemClick(layer_drop, slide_layer_path)
+    layer_drop._trigger_event(click)
+
+    assert len(layer_drop.menu) == 6
+
+
+def test_transform_overlay(doc: Document, data_path: pytest.TempPathFactory) -> None:
+    """Test adding a transform overlay."""
+    layer_drop = doc.get_model_by_name("layer_drop0")
+    affine_layer_path = str(data_path["affine_trans"])  # sample .npy file
+
+    click = MenuItemClick(layer_drop, affine_layer_path)
+    layer_drop._trigger_event(click)
+
+    assert len(layer_drop.menu) == 6
+
+
 def test_add_annotation_layer(doc: Document, data_path: pytest.TempPathFactory) -> None:
     """Test adding annotation layers."""
     # test loading a geojson file.
@@ -263,7 +293,7 @@ def test_add_annotation_layer(doc: Document, data_path: pytest.TempPathFactory)
     # test loading an annotation store
     slide_select.value = [data_path["slide1"].name]
     layer_drop = doc.get_model_by_name("layer_drop0")
-    assert len(layer_drop.menu) == 5
+    assert len(layer_drop.menu) == 6
     n_renderers = len(doc.get_model_by_name("slide_windows").children[0].renderers)
     # trigger an event to select the annotation .db file
     click = MenuItemClick(layer_drop, str(data_path["annotations"]))

tests/test_app_bokeh.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import logging
 import urllib
 from pathlib import Path, PureWindowsPath
 from typing import TYPE_CHECKING, Callable, NoReturn
@@ -736,6 +737,132 @@ def test_prop_range(app: TileServer) -> None:
         assert layer.renderer.score_fn(0.5) == 0.5
 
 
+def test_registration_dual_window(
+    empty_app: TileServer, tmp_path: Path, remote_sample: Callable
+) -> None:
+    """Test registering slides."""
+    data = make_simple_dat()
+    joblib.dump(data, tmp_path / "test.dat")
+    with empty_app.test_client() as client, empty_app.test_client() as client2:
+        setup_app(client)
+        response = client.put(
+            "/tileserver/slide",
+            data={"slide_path": safe_str(remote_sample("svs-1-small"))},
+        )
+        assert response.status_code == 200
+
+        # Open new window with other slide...
+        setup_app(client2)
+        response = client2.put(
+            "/tileserver/slide",
+            data={"slide_path": safe_str(remote_sample("svs-1-small"))},
+        )
+        assert response.status_code == 200
+
+        response = client2.put(
+            "/tileserver/overlay",
+            data={"overlay_path": safe_str(remote_sample("reg_disp_mha_example"))},
+        )
+        assert response.status_code == 200
+
+
+def test_registration_single_window_same_slide(
+    empty_app: TileServer,
+    remote_sample: Callable,
+    caplog: pytest.LogCaptureFixture,
+) -> None:
+    """Test registering slides."""
+    # First is testing with a single sldie and single registration
+    with empty_app.test_client() as client:
+        setup_app(client)
+        response = client.put(
+            "/tileserver/slide",
+            data={"slide_path": safe_str(remote_sample("svs-1-small"))},
+        )
+        assert response.status_code == 200
+
+        # Capture logger output to check for warning
+        with caplog.at_level(logging.WARNING):
+            response = client.put(
+                "/tileserver/overlay",
+                data={"overlay_path": safe_str(remote_sample("reg_disp_mha_example"))},
+            )
+            assert (
+                "No suitable overlay found. Using current slide as target. "
+                "This may display incorrectly if dimensions differ." in caplog.text
+            )
+            assert response.status_code == 200
+
+
+def test_registration_single_window_nonslide_overlay(
+    empty_app: TileServer,
+    remote_sample: Callable,
+    caplog: pytest.LogCaptureFixture,
+) -> None:
+    """Test registering slides with non-slide overlay."""
+    with empty_app.test_client() as client:
+        setup_app(client)
+        response = client.put(
+            "/tileserver/slide",
+            data={"slide_path": safe_str(remote_sample("svs-1-small"))},
+        )
+        assert response.status_code == 200
+        # check behaviour when an overlay is there, but it isn't suitabe (e.g jpg)
+        response = client.put(
+            "/tileserver/overlay",
+            data={"overlay_path": safe_str(remote_sample("wsi2_4k_4k_jpg"))},
+        )
+        assert response.status_code == 200
+
+        with caplog.at_level(logging.WARNING):
+            response = client.put(
+                "/tileserver/overlay",
+                data={"overlay_path": safe_str(remote_sample("reg_disp_mha_example"))},
+            )
+            assert (
+                "No suitable overlay found. Using current slide as target. "
+                "This may display incorrectly if dimensions differ." in caplog.text
+            )
+            assert response.status_code == 200
+
+
+def test_registration_single_window_different_slide(
+    empty_app: TileServer,
+    tmp_path: Path,
+    remote_sample: Callable,
+    caplog: pytest.LogCaptureFixture,
+) -> None:
+    """Test registering slides."""
+    # Repeat but provide extra overlays
+    data = make_simple_dat()
+    joblib.dump(data, tmp_path / "test.dat")
+    with empty_app.test_client() as client:
+        setup_app(client)
+        response = client.put(
+            "/tileserver/slide",
+            data={"slide_path": safe_str(remote_sample("svs-1-small"))},
+        )
+        assert response.status_code == 200
+
+        # Now add extra slide (i.e. IHC slide as target to register with)
+        response = client.put(
+            "/tileserver/overlay",
+            data={"overlay_path": safe_str(remote_sample("svs-1-small"))},
+        )
+
+        # Capture logger output to check for warning
+        with caplog.at_level(logging.WARNING):
+            response = client.put(
+                "/tileserver/overlay",
+                data={"overlay_path": safe_str(remote_sample("reg_disp_mha_example"))},
+            )
+            assert (
+                "Using slide as source and last overlay as target for registration."
+                in caplog.text
+            )
+            assert response.status_code == 200
+
+
 def test_healthcheck(empty_app: TileServer) -> None:
     """Test the /tileserver/healthcheck endpoint."""
     with empty_app.test_client() as client: