WNet + models code refactor (#36)

* Testing instance methods

Co-Authored-By: gityves <[email protected]>

* Many fixes

- Fixed monai reqs
- Added custom functions for label checking
- Fixed return type of voronoi_otsu and utils.resize
- black

* black

* Complete instance method evaluation

* Enfore pre-commit style

* Removing dask-image

* Fixed erroneous dtype conversion

* Update test_plugin_utils.py

* Update tox.ini

* Added new pre-commit hooks

* Run full suite of pre-commit hooks

* Enforce style

* Documentation update, crop contrast fix

* Updated hooks

* Update setup.cfg

* Many fixes

- Fixed monai reqs
- Added custom functions for label checking
- Fixed return type of voronoi_otsu and utils.resize
- black

* Enfore pre-commit style

* Updated project files

* Removing dask-image

* Latest pre-commit hooks

* Instance segmentation refactor + Voronoi-Otsu

- Improved code for instance segmentation
- Added Voronoi-Otsu labeling from pyclesperanto
TODO : credits for labeling

* isort

* Fix inference

* Added labeling tools + UI tweaks

- Added tools from MLCourse to evaluate labels and auto-correct them
- Instance seg benchmark notebook
- Tweaked utils UI to scale according to Viewer size

Co-Authored-By: gityves <[email protected]>

* Many fixes

- Fixed monai reqs
- Added custom functions for label checking
- Fixed return type of voronoi_otsu and utils.resize
- black

* Added pre-commit hooks

* Update .pre-commit-config.yaml

* Update pyproject.toml

* Update pyproject.toml

Ruff config

* Enfore pre-commit style

* Update .gitignore

* Version bump

* Revert "Version bump"

This reverts commit 6e39971.

* Updated project files

* Fixed wrong value in instance sliders

* Removing dask-image

* Update test_plugin_utils.py

* Relabeling tests

* Added new pre-commit hooks

* Latest pre-commit hooks

* Run full suite of pre-commit hooks

* Model class refactor

* Added LR scheduler in training

- Added ReduceLROnPlateau with params in training
- Updated training guide
- Minor UI attribute refactor
- black

* Update assess_instance.ipynb

* Update .gitignore

* Started adding WNet

* Specify no grad in inference

* First functional WNet inference, no CRF

* Create test_models.py

* Run full suite of pre-commit hooks

* Patch for tests action + style

* Add softNCuts basic test

* Added crf

Co-Authored-By: Nevexios <[email protected]>

* More pre-commit checks

* Functional CRF

* Fix erroneous test comment, added toggle for crf

- Warn if crf not installed
- Fix test

* Specify missing test deps

* Trying to fix deps on Git

* Removed master link to pydensecrf

* Use commit hash

* Removed commit hash

* Removed master

* Update tox.ini

* Update pyproject.toml

* Fixes and improvements

- More CRF info
- Added error handling to scheduler rate
- Added ETA to training
- Updated padding warning trigger size

* Fixes and channel labeling prototype

* Fixes

- Fixed multi-channel instance and csv stats
- Fixed rotation of inference outputs
- Raised max crop size

* Update plugin_model_inference.py

* Update plugin_crop.py

* Fixed patch_func sample number mismatch

* Testing relabel tools

* Fixes in inference

* add model template + fix test + wnet loading opti

- test fixes
- changed crf input reqs
- adapted instance seg for several channels

* Update model_WNet.py

* Update model_VNet.py

* Fixed folder creation when saving to folder

* Fix check_ready for results filewidget

* Added remapping in WNet + ruff config

* Run new hooks

* Small docs update

* Testing fix

* Fixed multithread testing (locally)

* Added proper tests for train/infer

* Slight coverage increase

* Update test_plugin_inference.py

* Set window inference to 64 for WNet

* Update instance_segmentation.py

* Moved normalization to the correct place

* Added auto-set dims for cropping

* Update test_plugin_utils.py

* More WNet

- Added experimental .pt loading for jit models
- More CRF tests
- Optimized WNet by loading inference only

* Update crf test/deps for testing

* Update test_and_deploy.yml

* Update test_and_deploy.yml

* Update tox.ini

* Update test_and_deploy.yml

* Trying to fix tox install of pydensecrf

* Added experimental ONNX support for inference

* Updated WNet for ONNX conversion

* Added dropout param

* Minor fixes in training

* Fix weights file extension in inference + coverage

- Remove unused scripts
- More tests
- Fixed weights type in inference

* Run all hooks

* Fix inference testing

* Changed anisotropy calculation

* Finish rebase + bump version

* Instance segmentation refactor + Voronoi-Otsu

- Improved code for instance segmentation
- Added Voronoi-Otsu labeling from pyclesperanto
TODO : credits for labeling

* Disabled small removal in Voronoi-Otsu

* Added new docs for instance seg

* Docs + UI update

- Updated welcome/README
- Changed step for DoubleCounter

* Update requirements.txt

Fix typo

* isort

* Fix tests

* Fixed parental issues and instance seg widget init

- Fixed widgets parents that were incorrectly init
- Improve use of instance seg. method classes and init

* Fix inference

* Added labeling tools + UI tweaks

- Added tools from MLCourse to evaluate labels and auto-correct them
- Instance seg benchmark notebook
- Tweaked utils UI to scale according to Viewer size

Co-Authored-By: gityves <[email protected]>

* Testing instance methods

Co-Authored-By: gityves <[email protected]>

* Many fixes

- Fixed monai reqs
- Added custom functions for label checking
- Fixed return type of voronoi_otsu and utils.resize
- black

* black

* Complete instance method evaluation

* Added pre-commit hooks

* Enfore pre-commit style

* Update .gitignore

* Version bump

* Updated project files

* Fixed missing parent error

* Fixed wrong value in instance sliders

* Removing dask-image

* Fixed erroneous dtype conversion

* Update test_plugin_utils.py

* Temporary test action patch

* Update plugin_convert.py

* Update tox.ini

Added pocl for testing on GH Actions

* Update tox.ini

* Found existing pocl

* Updated utils test to avoid Voronoi-Otsu

VO is missing CL runtime

* Relabeling tests

* Latest pre-commit hooks

* Run full suite of pre-commit hooks

* Enforce style

* Instance segmentation refactor + Voronoi-Otsu

- Improved code for instance segmentation
- Added Voronoi-Otsu labeling from pyclesperanto
TODO : credits for labeling

* Disabled small removal in Voronoi-Otsu

* Added new docs for instance seg

* Docs + UI update

- Updated welcome/README
- Changed step for DoubleCounter

* Update requirements.txt

Fix typo

* isort

* Fix tests

* Fixed parental issues and instance seg widget init

- Fixed widgets parents that were incorrectly init
- Improve use of instance seg. method classes and init

* Fix inference

* Added labeling tools + UI tweaks

- Added tools from MLCourse to evaluate labels and auto-correct them
- Instance seg benchmark notebook
- Tweaked utils UI to scale according to Viewer size

Co-Authored-By: gityves <[email protected]>

* Testing instance methods

Co-Authored-By: gityves <[email protected]>

* Many fixes

- Fixed monai reqs
- Added custom functions for label checking
- Fixed return type of voronoi_otsu and utils.resize
- black

* black

* Complete instance method evaluation

* Added pre-commit hooks

* Update .pre-commit-config.yaml

* Update pyproject.toml

* Update pyproject.toml

Ruff config

* Enfore pre-commit style

* Update .gitignore

* Version bump

* Revert "Version bump"

This reverts commit 6e39971.

* Updated project files

* Fixed missing parent error

* Fixed wrong value in instance sliders

* Removing dask-image

* Fixed erroneous dtype conversion

* Update test_plugin_utils.py

* Update plugin_convert.py

* Update tox.ini

Added pocl for testing on GH Actions

* Update tox.ini

* Found existing pocl

* Updated utils test to avoid Voronoi-Otsu

VO is missing CL runtime

* Relabeling tests

* Added new pre-commit hooks

* Latest pre-commit hooks

* Run full suite of pre-commit hooks

* Model class refactor

* Added LR scheduler in training

- Added ReduceLROnPlateau with params in training
- Updated training guide
- Minor UI attribute refactor
- black

* Update assess_instance.ipynb

* Update .gitignore

* Started adding WNet

* Specify no grad in inference

* First functional WNet inference, no CRF

* Create test_models.py

* Run full suite of pre-commit hooks

* Patch for tests action + style

* Add softNCuts basic test

* Added crf

Co-Authored-By: Nevexios <[email protected]>

* More pre-commit checks

* Functional CRF

* Fix erroneous test comment, added toggle for crf

- Warn if crf not installed
- Fix test

* Specify missing test deps

* Trying to fix deps on Git

* Removed master link to pydensecrf

* Use commit hash

* Removed commit hash

* Removed master

* Update tox.ini

* Update pyproject.toml

* Fixes and improvements

- More CRF info
- Added error handling to scheduler rate
- Added ETA to training
- Updated padding warning trigger size

* Fixes and channel labeling prototype

* Fixes

- Fixed multi-channel instance and csv stats
- Fixed rotation of inference outputs
- Raised max crop size

* Update plugin_model_inference.py

* Fixed patch_func sample number mismatch

* Testing relabel tools

* Fixes in inference

* add model template + fix test + wnet loading opti

- test fixes
- changed crf input reqs
- adapted instance seg for several channels

* Update model_WNet.py

* Update model_VNet.py

* Fixed folder creation when saving to folder

* Fix check_ready for results filewidget

* Added remapping in WNet + ruff config

* Run new hooks

* Small docs update

* Testing fix

* Fixed multithread testing (locally)

* Added proper tests for train/infer

* Slight coverage increase

* Update test_plugin_inference.py

* Set window inference to 64 for WNet

* Moved normalization to the correct place

* Added auto-set dims for cropping

* Update test_plugin_utils.py

* More WNet

- Added experimental .pt loading for jit models
- More CRF tests
- Optimized WNet by loading inference only

* Update crf test/deps for testing

* Update test_and_deploy.yml

* Update test_and_deploy.yml

* Update tox.ini

* Update test_and_deploy.yml

* Trying to fix tox install of pydensecrf

* Added experimental ONNX support for inference

* Updated WNet for ONNX conversion

* Added dropout param

* Minor fixes in training

* Fix weights file extension in inference + coverage

- Remove unused scripts
- More tests
- Fixed weights type in inference

* Run all hooks

* Fix inference testing

* Changed anisotropy calculation

* Fixed aniso correction and CRF interaction

* Remove duplicate tests

* Finish rebase + changed step to auto in spinbox

* Updated based on feedback from CYHSM

Co-Authored-By: Markus Frey <[email protected]>

* Added minimal WNet notebook for training

* Remove dask

* WNet model docs

* Added QoL shape info for layer selecter

* WNet fixes + PR feedback improvements

* Added imagecodecs to open external datasets

---------

Co-authored-by: gityves <[email protected]>
Co-authored-by: Nevexios <[email protected]>
Co-authored-by: Markus Frey <[email protected]>

Author: 4 people

.coveragerc

@@ -0,0 +1,7 @@
+[report]
+exclude_lines =
+    if __name__ == .__main__.:
+
+[run]
+omit =
+    napari_cellseg3d/setup.py

.github/workflows/test_and_deploy.yml

@@ -7,15 +7,11 @@ on:
   push:
     branches:
       - main
-      - npe2
-      - cy/voronoi-otsu
     tags:
       - "v*" # Push events to matching v*, i.e. v1.0, v20.15.10
   pull_request:
     branches:
       - main
-      - npe2
-      - cy/voronoi-otsu
   workflow_dispatch:
 
 jobs:
@@ -55,6 +51,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           python -m pip install setuptools tox tox-gh-actions
+#         pip install git+https://github.com/lucasb-eyer/pydensecrf.git@master#egg=pydensecrf
 
 # this runs the platform-specific tests declared in tox.ini
       - name: Test with tox

.gitignore

@@ -104,9 +104,11 @@ notebooks/csv_cell_plot.html
 notebooks/full_plot.html
 *.csv
 *.png
+notebooks/instance_test.ipynb
 *.prof
 
 #include test data
 !napari_cellseg3d/_tests/res/test.tif
 !napari_cellseg3d/_tests/res/test.png
 !napari_cellseg3d/_tests/res/test_labels.tif
+cov.syspath.txt

docs/index.rst

@@ -39,8 +39,8 @@ Welcome to napari-cellseg3d's documentation!
     res/code/plugin_convert
     res/code/plugin_metrics
     res/code/model_framework
-    res/code/model_workers
-    res/code/model_instance_seg
+    res/code/workers
+    res/code/instance_segmentation
     res/code/plugin_model_inference
     res/code/plugin_model_training
     res/code/utils

docs/res/code/instance_segmentation.rst

@@ -0,0 +1,53 @@
+instance_segmentation.py
+===========================================
+
+Classes
+-------------
+
+InstanceMethod
+**************************************
+.. autoclass:: napari_cellseg3d.code_models.instance_segmentation::InstanceMethod
+    :members: __init__
+
+ConnectedComponents
+**************************************
+.. autoclass:: napari_cellseg3d.code_models.instance_segmentation::ConnectedComponents
+    :members: __init__
+
+Watershed
+**************************************
+.. autoclass:: napari_cellseg3d.code_models.instance_segmentation::Watershed
+    :members: __init__
+
+VoronoiOtsu
+**************************************
+.. autoclass:: napari_cellseg3d.code_models.instance_segmentation::VoronoiOtsu
+    :members: __init__
+
+
+Functions
+-------------
+
+binary_connected
+**************************************
+.. autofunction:: napari_cellseg3d.code_models.instance_segmentation::binary_connected
+
+binary_watershed
+**************************************
+.. autofunction:: napari_cellseg3d.code_models.instance_segmentation::binary_watershed
+
+volume_stats
+**************************************
+.. autofunction:: napari_cellseg3d.code_models.instance_segmentation::volume_stats
+
+clear_small_objects
+**************************************
+.. autofunction:: napari_cellseg3d.code_models.instance_segmentation::clear_small_objects
+
+to_instance
+**************************************
+.. autofunction:: napari_cellseg3d.code_models.instance_segmentation::to_instance
+
+to_semantic
+**************************************
+.. autofunction:: napari_cellseg3d.code_models.instance_segmentation::to_semantic

docs/res/code/model_instance_seg.rst

@@ -28,18 +28,3 @@ ThresholdUtils
 **********************************
 .. autoclass:: napari_cellseg3d.code_plugins.plugin_convert::ThresholdUtils
     :members: __init__
-
-Functions
------------------------------------
-
-save_folder
-*****************************************
-.. autofunction:: napari_cellseg3d.code_plugins.plugin_convert::save_folder
-
-save_layer
-****************************************
-.. autofunction:: napari_cellseg3d.code_plugins.plugin_convert::save_layer
-
-show_result
-****************************************
-.. autofunction:: napari_cellseg3d.code_plugins.plugin_convert::show_result

docs/res/code/plugin_convert.rst

@@ -18,6 +18,5 @@ Methods
 
 Attributes
 *********************
-
 .. autoclass:: napari_cellseg3d.code_plugins.plugin_model_training::Trainer
    :members:  _viewer, worker, loss_dict, canvas, train_loss_plot, dice_metric_plot

docs/res/code/plugin_model_training.rst

@@ -62,7 +62,3 @@ denormalize_y
 load_images
 **************************************
 .. autofunction:: napari_cellseg3d.utils::load_images
-
-format_Warning
-**************************************
-.. autofunction:: napari_cellseg3d.utils::format_Warning

docs/res/code/utils.rst

@@ -1,4 +1,4 @@
-model_workers.py
+workers.py
 ===========================================
 
 
@@ -10,7 +10,7 @@ Class : LogSignal
 
 Attributes
 ************************
-.. autoclass:: napari_cellseg3d.code_models.model_workers::LogSignal
+.. autoclass:: napari_cellseg3d.code_models.workers::LogSignal
    :members: log_signal
    :noindex:
 
@@ -24,7 +24,7 @@ Class : InferenceWorker
 
 Methods
 ************************
-.. autoclass:: napari_cellseg3d.code_models.model_workers::InferenceWorker
+.. autoclass:: napari_cellseg3d.code_models.workers::InferenceWorker
    :members: __init__, log, create_inference_dict, inference
    :noindex:
 
@@ -39,6 +39,6 @@ Class : TrainingWorker
 
 Methods
 ************************
-.. autoclass:: napari_cellseg3d.code_models.model_workers::TrainingWorker
+.. autoclass:: napari_cellseg3d.code_models.workers::TrainingWorker
    :members: __init__, log, train
    :noindex: