[wkcuber] Make tiff and czi reading more robust (#335)

* use different indexing for tiff reading

* formatting

* adapt czi reading and include new z axis names

* pr feedback 2, includes new channel and sample index

* add new config options for power users

* fix cli interface

* add tests

* fix typo in test

* pr feedback

* remove method artifact

* fix test syntax

* add debuging

* add dependency

* remove unnecessary debugging

Co-authored-by: Robert S <[email protected]>

Author: youri-k

wkcuber/Changelog.md

@@ -28,6 +28,7 @@ For upgrade instructions, please check the respective *Breaking Changes* section
 - Adjust downsampling scheme to always try to minimize the scaled difference between the different dimensions of a mag and renamed the sampling mode `auto` to `anisotropic`. [#391](https://github.com/scalableminds/webknossos-libs/pull/391)
 - Make parameter `executor` optional for `View.for_each_chunk` and `View.for_zipped_chunks`. [#404](https://github.com/scalableminds/webknossos-libs/pull/404)
 - Add option to rename foreign layer with add_{symlink,copy}_layer. [#419](https://github.com/scalableminds/webknossos-libs/pull/419)
+- Improved the `TIFF` and `CZI` reader to work with a wider variety of formats. The module `convert_image_stack_to_wkw` is now capable of making the result webKnossos compatible. [#335](https://github.com/scalableminds/webknossos-libs/pull/335)
 
 ### Fixed
 - Reverted that `dataset.add_symlink_layer` and `dataset.add_copy_layer` resolved the layer path if it was a symlink. [#408](https://github.com/scalableminds/webknossos-libs/pull/408)

wkcuber/testdata/various_tiff_formats/test_C.tif

@@ -18,9 +18,11 @@ rm -r testdata/tiff_mag_2_reference
 sh ${BASEDIR}/anisotropic_downsampling.sh
 sh ${BASEDIR}/compression_and_verification.sh
 rm -r testoutput/tiff_compress
+rm -r testoutput/tiff_compress_broken
 
 sh ${BASEDIR}/in_place_compression.sh
 rm -r testoutput/tiff_compress2
+rm -r testoutput/tiff_compress2.bak
 rm -r testoutput/tiff
 
 sh ${BASEDIR}/tile_cubing.sh
@@ -32,5 +34,8 @@ rm -r testoutput/tiff2
 sh ${BASEDIR}/simple_tiff_cubing_no_compression.sh
 rm -r testoutput/tiff3
 
+sh ${BASEDIR}/tiff_formats_cubing.sh
+rm -r testoutput/tiff4
+
 sh ${BASEDIR}/knossos_conversion.sh
 rm -r testoutput/knossos

wkcuber/testdata/various_tiff_formats/test_CS.tif

@@ -16,12 +16,11 @@ echo "Check equality for uncompressed and compressed dataset"
 python -m wkcuber.check_equality testoutput/tiff testoutput/tiff_compress
 
 echo "Create broken copy of dataset"
-rm -rf testoutput/tiff_compress-broken
-cp -R testoutput/tiff_compress testoutput/tiff_compress-broken
-rm -r testoutput/tiff_compress-broken/color/1/z0/y0/x0.wkw
+cp -R testoutput/tiff_compress testoutput/tiff_compress_broken
+rm -r testoutput/tiff_compress_broken/color/1/z0/y0/x0.wkw
 
 echo "Compare original dataset to broken one and expect to determine difference"
-if python -m wkcuber.check_equality testoutput/tiff testoutput/tiff_compress-broken ; then
+if python -m wkcuber.check_equality testoutput/tiff testoutput/tiff_compress_broken ; then
     echo "Equality check did not fail even though the dataset is broken."
     exit 1
 else

wkcuber/testdata/various_tiff_formats/test_I.tif

@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+set -xe
+mkdir -p testoutput/tiff4
+
+# simplest case: Only XY and I (alternative Z axis name) axis
+python -m wkcuber.convert_image_stack_to_wkw \
+  --jobs 2 \
+  --batch_size 8 \
+  --layer_name color \
+  --max_mag 4 \
+  --target_mag 2\
+  --scale 1,1,1 \
+  --name awesome_data \
+  --no_compress \
+  --wkw_file_len 8 \
+  testdata/various_tiff_formats/test_I.tif testoutput/tiff4
+[ -d testoutput/tiff4/color ]
+[ -d testoutput/tiff4/color/2 ]
+[ $(find testoutput/tiff4/color/2 -mindepth 3 -name "*.wkw" | wc -l) -eq 1 ]
+[ -e testoutput/tiff4/datasource-properties.json ]
+rm -r testoutput/tiff4/*
+
+# to make output wk_compatible, these files require multiple layer
+python -m wkcuber.convert_image_stack_to_wkw \
+  --jobs 2 \
+  --batch_size 8 \
+  --layer_name color \
+  --max_mag 1 \
+  --scale 1,1,1 \
+  --name awesome_data \
+  --no_compress \
+  --wkw_file_len 8 \
+  testdata/various_tiff_formats/test_C.tif testoutput/tiff4
+[ -d testoutput/tiff4/color_0 ]
+[ -d testoutput/tiff4/color_1 ]
+[ -d testoutput/tiff4/color_2 ]
+[ -d testoutput/tiff4/color_3 ]
+[ -d testoutput/tiff4/color_4 ]
+[ -d testoutput/tiff4/color_0/1 ]
+[ $(find testoutput/tiff4/color_0/1 -mindepth 3 -name "*.wkw" | wc -l) -eq 1 ]
+[ -e testoutput/tiff4/datasource-properties.json ]
+rm -r testoutput/tiff4/*
+
+python -m wkcuber.convert_image_stack_to_wkw \
+  --jobs 2 \
+  --batch_size 8 \
+  --layer_name color \
+  --max_mag 1 \
+  --scale 1,1,1 \
+  --name awesome_data \
+  --no_compress \
+  --wkw_file_len 8 \
+  testdata/various_tiff_formats/test_S.tif testoutput/tiff4
+[ -d testoutput/tiff4/color_0 ]
+[ -d testoutput/tiff4/color_1 ]
+[ -d testoutput/tiff4/color_2 ]
+[ -d testoutput/tiff4/color_0/1 ]
+[ $(find testoutput/tiff4/color_0/1 -mindepth 3 -name "*.wkw" | wc -l) -eq 1 ]
+[ -e testoutput/tiff4/datasource-properties.json ]
+rm -r testoutput/tiff4/*
+
+python -m wkcuber.convert_image_stack_to_wkw \
+  --jobs 2 \
+  --batch_size 8 \
+  --layer_name color \
+  --max_mag 1 \
+  --scale 1,1,1 \
+  --name awesome_data \
+  --no_compress \
+  --wkw_file_len 8 \
+  testdata/various_tiff_formats/test_CS.tif testoutput/tiff4
+[ -d testoutput/tiff4/color_0 ]
+[ -d testoutput/tiff4/color_1 ]
+[ -d testoutput/tiff4/color_2 ]
+[ -d testoutput/tiff4/color_3 ]
+[ -d testoutput/tiff4/color_4 ]
+[ -d testoutput/tiff4/color_0/1 ]
+[ $(find testoutput/tiff4/color_0/1 -mindepth 3 -name "*.wkw" | wc -l) -eq 1 ]
+[ -e testoutput/tiff4/datasource-properties.json ]
+rm -r testoutput/tiff4/*
+
+# test wk incompatible configs, program should fail because force parameter was not given
+if python -m wkcuber.convert_image_stack_to_wkw \
+    --jobs 2 \
+    --batch_size 8 \
+    --layer_name color \
+    --max_mag 1 \
+    --scale 1,1,1 \
+    --prefer_channels \
+    --name awesome_data \
+    --no_compress \
+    --wkw_file_len 8 \
+    testdata/various_tiff_formats/test_C.tif testoutput/tiff4; then
+  echo "Conversion worked although invalid config should have been detected."
+  exit 1
+else
+  echo "Conversion did not work as expected due to invalid config."
+fi
+
+# power user configuration: should only create single layer
+python -m wkcuber.convert_image_stack_to_wkw \
+  --jobs 2 \
+  --batch_size 8 \
+  --layer_name color \
+  --max_mag 1 \
+  --scale 1,1,1 \
+  --channel_index 3 \
+  --sample_index 2 \
+  --name awesome_data \
+  --no_compress \
+  --wkw_file_len 8 \
+  testdata/various_tiff_formats/test_CS.tif testoutput/tiff4
+[ -d testoutput/tiff4/color ]
+[ -d testoutput/tiff4/color/1 ]
+[ $(find testoutput/tiff4/color/1 -mindepth 3 -name "*.wkw" | wc -l) -eq 1 ]
+[ -e testoutput/tiff4/datasource-properties.json ]

wkcuber/testdata/various_tiff_formats/test_S.tif

@@ -27,7 +27,7 @@ def test_single_channel_conversion() -> None:
     single_channel_data = np.empty((3, 32, 32, 1, 1), multi_channel_data.dtype)
     for i in range(3):
         single_channel_data[i] = image_reader_manager.read_array(
-            TESTDATA_DIR / "rgb_tiff" / "test_rgb.tif", np.uint8, 0, i
+            TESTDATA_DIR / "rgb_tiff" / "test_rgb.tif", np.uint8, 0, 0, i
         )
 
     single_channel_data = single_channel_data.transpose((1, 2, 0, 3, 4))

wkcuber/tests/scripts/all_tests.sh

@@ -26,7 +26,7 @@ def detect_present_mags(target_path: Path) -> Dict[Path, List[Mag]]:
         mag_paths = list([p for p in layer_p.iterdir() if p.is_dir()])
         for mag_p in mag_paths:
             try:
-                mag = Mag(mag_p.stem)
+                mag = Mag(mag_p.name)
             except (AssertionError, ValueError) as _:
                 continue
             layer_path_to_mags[layer_p].append(mag)
@@ -80,12 +80,12 @@ def main(args: Namespace) -> None:
 
     if not args.no_compress:
         for (layer_path, mags) in layer_path_to_mags.items():
-            layer_name = layer_path.stem
+            layer_name = layer_path.name
             for mag in mags:
                 compress_mag_inplace(args.target_path, layer_name, mag, args)
 
     for (layer_path, mags) in layer_path_to_mags.items():
-        layer_name = layer_path.stem
+        layer_name = layer_path.name
         mags.sort()
         downsample_mags(
             path=args.target_path,