improv rotation/train csv + docs

added rotation toggle in metrics + added training csv + docs

Author: C-Achard

docs/res/guides/metrics_module_guide.rst

@@ -11,7 +11,20 @@ The Dice coefficient is defined as :
 
 It is a measure of similarity between two sets- :math:`0` indicating no similarity and :math:`1` complete similarity.
 
-You will need to provide two folders : one for ground truth labels and one for predicition labels.
+You will need to provide the following parameters:
+
+* Two folders : one for ground truth labels and one for prediction labels.
+
+* The threshold below which the score is considered insufficient.
+  Any pair below that score will be shown on the viewer; and be displayed in red in the plot.
+
+* Whether to automatically determine the best orientation for the computation by rotating and flipping;
+  use this if your images do not have the same orientation.
+
+.. note::
+    Due to changes in orientation of images after running inference, the utility can rotate and flip images randomly to find the best Dice coefficient
+    to compensate. If you have small images with a very large number of labels, this can lead to an inexact metric being computed.
+    Images with a low score might be in the wrong orientation as well when displayed for comparison.
 
 .. important::
     This utility assumes that **predictions are padded to a power of two already.** Ground truth labels can be smaller,
@@ -22,10 +35,6 @@ Once you are ready, press the "Compute Dice" button. This will plot the Dice sco
 Pairs with a low score will be displayed on the viewer for checking, ground truth in **blue**, low score prediction in **red**.
 
 
-.. note::
-    Due to changes in orientation of images after running inference, the utility will rotate and flip images to find the best Dice coefficient
-    to compensate. If you have small images with a very large number of labels, this can lead to an inexact metric being computed.
-    Images with a low score might be in the wrong orientation as well when displayed for comparison.
 
 
 Source code

docs/res/guides/review_module_guide.rst

@@ -4,7 +4,7 @@ Reviewer module guide
 =================================
 
 This module allows you to review your labels, from predictions or manual labeling,
-and correct them if needed. It then saves the status of each file in a csv, for easier monitoring.
+and correct them if needed. It then saves the status of each file in a csv, as well as the time taken per slice review, for easier monitoring.
 
 
 
@@ -17,13 +17,13 @@ Launching the review process
     Folders can be stacks of either **.png** or **.tif** files, ideally numbered with the index of the slice at the end.
 
 .. note::
-    Only single .tif files or folder of several .png or .tif are supported.
+    Only single .tif files or folder of several .png or .tif are currently supported.
 
-* Model name :
+* Csv file name :
     You can then provide a model name, which will be used to name the csv file recording the status of each slice.
 
     If a corresponding csv file exists already, it will be used. If not, a new one will be created.
-    If you choose to create a new dataset, a new csv will be created no matter what,
+    If you choose to create a new dataset, a new csv will always be created,
     with a trailing number if several copies of it already exists.
 
 * Start :

docs/res/guides/training_module_guide.rst

@@ -44,6 +44,8 @@ The training module is comprised of several tabs.
 * Whether to use pre-trained weights that are provided; if you choose to do so, the model will be initialized with the specified weights, possibly improving performance (transfer learning).
   You can also load custom weights; simply ensure they are compatible with the model.
 
+* The proportion of the dataset to keep for training versus validation; if you have a large dataset, you can set it to a lower value to have more accurate validation steps.
+
 2) The second tab, **Augmentation**, lets you define dataset and augmentation parameters such as :
 
 * Whether to use images "as is" (**requires all images to be of the same size and cubic**) or extract patches.
@@ -53,6 +55,7 @@ The training module is comprised of several tabs.
     * The size of patches to be extracted (ideally, please use a value **close to a power of two**, such as 120 or 60 to ensure correct size.)
     * The number of samples to extract from each of your images. A larger number will likely mean better performances, but longer training and larger memory usage.
 
+
 * Whether to perform data augmentation or not (elastic deforms, intensity shifts. random flipping,etc). A rule of thumb for augmentation is :
 
     * If you're using the patch extraction method, enable it if you are using more than 10 samples per image with at least 5 images
@@ -63,6 +66,7 @@ The training module is comprised of several tabs.
 
 * The **model** to use for training (see table above)
 * The **loss function** used for training (see table below)
+* The **learning rate** of the optimizer. Setting it to a lower value if you're using pre-trained weights can improve performance.
 * The **batch size** (larger means quicker training and possibly better performance but increased memory usage)
 * The **number of epochs** (a possibility is to start with 60 epochs, and decrease or increase depending on performance.)
 * The **epoch interval** for validation (for example, if set to two, the module will use the validation dataset to evaluate the model with the dice metric every two epochs.)

napari_cellseg3d/plugin_metrics.py

@@ -13,6 +13,8 @@
 from napari_cellseg3d.model_instance_seg import to_semantic
 from napari_cellseg3d.plugin_base import BasePluginFolder
 
+DEFAULT_THRESHOLD = 0.5
+
 
 class MetricsUtils(BasePluginFolder):
     """Plugin to evaluate metrics between two sets of labels, ground truh and prediction"""
@@ -40,8 +42,13 @@ def __init__(self, viewer: "napari.viewer.Viewer", parent):
         self.btn_compute_dice = ui.make_button(
             "Compute Dice", self.compute_dice
         )
+
+        self.rotate_choice = ui.make_checkbox("Find best orientation")
+
         self.btn_reset_plot = ui.make_button("Clear plots", self.remove_plots)
 
+        self.threshold_box = ui.make_n_spinboxes(min=0.1, max = 1, default=DEFAULT_THRESHOLD,step=0.1, double=True)
+
         self.btn_result_path.setVisible(False)
         self.lbl_result_path.setVisible(False)
 
@@ -72,7 +79,7 @@ def build(self):
 
         w, self.layout = ui.make_container_widget()
 
-        metrics_group_w, metrics_group_l = ui.make_group("Metrics")
+        metrics_group_w, metrics_group_l = ui.make_group("Data")
 
         self.lbl_image_files.setText("Ground truth")
 
@@ -96,11 +103,21 @@ def build(self):
             alignment=ui.LEFT_AL,
         )
 
-        metrics_group_l.addWidget(self.btn_compute_dice, alignment=ui.LEFT_AL)
-
         metrics_group_w.setLayout(metrics_group_l)
+        ############################
+        ui.add_blank(self, self.layout)
+        ############################
+        param_group_w, param_group_l = ui.make_group("Parameters")
+
+        param_group_l.addWidget(self.threshold_box)
+        param_group_l.addWidget(self.rotate_choice)
+
+        param_group_w.setLayout(param_group_l)
 
         self.layout.addWidget(metrics_group_w)
+        self.layout.addWidget(param_group_w)
+
+        self.layout.addWidget(self.btn_compute_dice, alignment=ui.LEFT_AL)
 
         self.layout.addWidget(self.make_close_button(), alignment=ui.LEFT_AL)
 
@@ -109,15 +126,15 @@ def build(self):
 
         ui.make_scrollable(self.layout, self)
 
-    def plot_dice(self, dice_coeffs):
+    def plot_dice(self, dice_coeffs, threshold=DEFAULT_THRESHOLD):
         """Plots the dice loss for each pair of labels on viewer"""
         self.btn_reset_plot.setVisible(True)
         colors = []
 
         bckgrd_color = (0, 0, 0, 0)
 
         for coeff in dice_coeffs:  # TODO add threshold manual setting
-            if coeff < 0.5:
+            if coeff < threshold:
                 colors.append(ui.dark_red)  # 72071d # crimson red
             else:
                 colors.append(ui.default_cyan)  # 8dd3c7 # turquoise cyan
@@ -138,7 +155,7 @@ def plot_dice(self, dice_coeffs):
             dice_plot.invert_yaxis()
 
             self.plots.append(self.canvas)
-            dice_plot.axvline(0.5, color=ui.dark_red)
+            dice_plot.axvline(threshold, color=ui.dark_red)
             dice_plot.set_title(
                 f"Session {len(self.plots)}\nMean dice : {np.mean(dice_coeffs):.4f}"
             )
@@ -157,12 +174,17 @@ def remove_plots(self):
         self.btn_reset_plot.setVisible(False)
 
     def compute_dice(self):
-        """Computes the dice metric between pairs of labels. Rotates the prediction label to find matching orientation as well."""
+        """Computes the dice metric between pairs of labels.
+        Rotates the prediction label to find matching orientation as well."""
         # u = 0
         # t = 0
+
+        threshold = self.threshold_box.value()
+        rotate = self.rotate_choice.isChecked()
+
         total_metrics = []
         self.canvas = (
-            None  # kind of unsafe way to stack plots... but it works.
+            None  # kind of terrible way to stack plots... but it works.
         )
         id = 0
         for ground_path, pred_path in zip(
@@ -230,18 +252,20 @@ def compute_dice(self):
             # )
             # u += 1
 
-            # TODO add rotation toggle
-            pred_flip_x = np.rot90(pred[0][0], axes=(0, 1))
-            pred_flip_y = np.rot90(pred[0][0], axes=(1, 2))
-            pred_flip_z = np.rot90(pred[0][0], axes=(0, 2))
             scores = []
-
-            for p in [pred[0][0], pred_flip_x, pred_flip_y, pred_flip_z]:
-                scores.append(utils.dice_coeff(p, ground))
-                scores.append(utils.dice_coeff(np.flip(p), ground))
-                for i in range(3):
-                    scores.append(utils.dice_coeff(np.flip(p, axis=i), ground))
-
+            if rotate:  # TODO : recored best rotation for display
+                pred_flip_x = np.rot90(pred[0][0], axes=(0, 1))
+                pred_flip_y = np.rot90(pred[0][0], axes=(1, 2))
+                pred_flip_z = np.rot90(pred[0][0], axes=(0, 2))
+
+                for p in [pred[0][0], pred_flip_x, pred_flip_y, pred_flip_z]:
+                    scores.append(utils.dice_coeff(p, ground))
+                    scores.append(utils.dice_coeff(np.flip(p), ground))
+                    for i in range(3):
+                        scores.append(utils.dice_coeff(np.flip(p, axis=i), ground))
+            else:
+                i=0
+                scores.append(utils.dice_coeff(pred, ground))
             # if t < 1:
             #     for i in range(3):
             #         self._viewer.add_image(
@@ -254,7 +278,7 @@ def compute_dice(self):
 
             # print(scores)
             score = max(scores)
-            if score < 0.5:
+            if score < threshold:
                 # TODO add filename ?
                 self._viewer.dims.ndisplay = 3
                 self._viewer.add_image(
@@ -265,4 +289,4 @@ def compute_dice(self):
                 )
             total_metrics.append(score)
         print(f"DICE METRIC :{total_metrics}")
-        self.plot_dice(total_metrics)
+        self.plot_dice(total_metrics, threshold)

napari_cellseg3d/plugin_model_training.py

@@ -1,8 +1,6 @@
 import os
 import shutil
 import warnings
-import zipfile
-import pandas as pd
 from pathlib import Path
 
 import matplotlib.pyplot as plt
@@ -81,7 +79,6 @@ def __init__(
 
         TODO training plugin:
 
-        * Choice of validation proportion
 
         * Custom model loading
 
@@ -460,6 +457,15 @@ def build(self):
         ################
         ui.add_blank(self, data_tab_layout)
         ################
+
+        validation_group_w, validation_group_l = ui.make_group("Validation %")
+
+        validation_group_l.addWidget(self.validation_percent_choice)
+        validation_group_w.setLayout(validation_group_l)
+        data_tab_layout.addWidget(validation_group_w)
+        ################
+        ui.add_blank(self, data_tab_layout)
+        ################
         # buttons
 
         data_tab_layout.addWidget(
@@ -516,12 +522,6 @@ def build(self):
         #######################
         ui.add_blank(augment_tab_w, augment_tab_l)
         #######################
-        validation_group_w, validation_group_l = ui.make_group("Validation %")
-
-        validation_group_l.addWidget(self.validation_percent_choice)
-        validation_group_w.setLayout(validation_group_l)
-        augment_tab_l.addWidget(validation_group_w)
-        #######################
         augment_group_w, augment_group_l = ui.make_group("Augmentation")
         augment_group_l.addWidget(
             self.augment_choice, alignment=ui.LEFT_AL