Merge pull request #1064 from computational-cell-analytics/dev

Prepare new release

Author: constantinpape

doc/band.md

@@ -1,24 +1,27 @@
 # Using `micro_sam` on BAND
 
-BAND is a service offered by EMBL Heidelberg that gives access to a virtual desktop for image analysis tasks. It is free to use and `micro_sam` is installed there.
+BAND is a service offered by EMBL Heidelberg under the "The German Network for Bioinformatics Infrastructure" (de.NBI) that gives access to a virtual desktop for image analysis tasks. It is free to use and `micro_sam` is installed there.
 In order to use BAND and start `micro_sam` on it follow these steps:
 
 ## Start BAND
-- Go to https://band.embl.de/ and click **Login**. If you have not used BAND before you will need to register for BAND. Currently you can only sign up via a Google account.
+- Go to https://bandv1.denbi.uni-tuebingen.de/ and click **Login**. If you have not used BAND before you will need to register for BAND. Currently you can only sign up via a Google account. NOTE: It takes a couple of seconds for the "Launch Desktops" window to appear.
 - Launch a BAND desktop with sufficient resources. It's particularly important to select a GPU. The settings from the image below are a good choice.
 - Go to the desktop by clicking **GO TO DESKTOP** in the **Running Desktops** menu. See also the screenshot below.
 
-![image](https://github.com/computational-cell-analytics/micro-sam/assets/4263537/f965fce2-b924-4fc8-871b-f3201e502138)
+![image](https://github.com/user-attachments/assets/2e615769-b176-4ced-9507-99553aafb1c1)
 
 ## Start `micro_sam` in BAND
-- Select **Applications -> Image Analysis -> uSAM** (see screenshot)
-![image](https://github.com/computational-cell-analytics/micro-sam/assets/4263537/5daeafb3-119b-4104-8708-aab2960cb21c)
-- This will open the micro_sam menu, where you can select the tool you want to use (see screenshot). Note: this may take a few minutes.
-![image](https://github.com/computational-cell-analytics/micro-sam/assets/4263537/900ce0b9-4cf8-418c-94f1-e99ac7bc0086)
-- For testing if the tool works, it's best to use the **2d annotator** first.
-  - You can find an example image to use here: `/scratch/cajal-connectomics/hela-2d-image.png`. Select it via **Select image**. (see screenshot)
-![image](https://github.com/computational-cell-analytics/micro-sam/assets/4263537/5fbd1c53-2ba1-47d4-ae50-dfab890ac9d3)
-- Then press **2d annotator** and the tool will start.
+- Select **Applications -> Image Analysis -> microSAM** (see screenshot)
+![image](https://github.com/user-attachments/assets/f1e7af0d-5cc6-442d-95c7-9010f5996e2a)
+
+- This will open the napari GUI, where you can select the images and annotator tools you want to use (see screenshot). NOTE: this may take a few minutes.
+![image](https://github.com/user-attachments/assets/23296589-be8d-4f9f-9a38-92d5eea60541)
+
+- For testing if the tool works, it's best to use the **Annotator 2d** first.
+  - You can find an example image to use by selection `File` -> `Open Sample` -> `Segment Anything for Microscopy` -> `HeLa 2d example data` (see screenshot)
+  ![image](https://github.com/user-attachments/assets/ac259787-6238-4b68-8739-48a62f169913)
+- Then select `Plugins` -> `Segment Anything for Microscopy` -> `Annotator 2d`, and the tool will start.
+![image](https://github.com/user-attachments/assets/3062e723-6fe0-48f7-835f-663be7d78b87)
 
 ## Transfering data to BAND
 

doc/bioimageio/em_organelles_v4.md

@@ -0,0 +1,19 @@
+# Segment Anything for Electron Microscopy
+
+This is a [Segment Anything](https://segment-anything.com/) model that was specialized for segmenting mitochondria and nuclei in electron microscopy with [micro_sam](https://github.com/computational-cell-analytics/micro-sam).
+This model uses a %s vision transformer as image encoder.
+
+Segment Anything is a model for interactive and automatic instance segmentation.
+We improve it for electron microscopy by finetuning on a large and diverse microscopy dataset.
+It should perform well for segmenting mitochondria and nuclei in electron microscopy. It can also work well for other organelles, but was not explicitly trained for this purpose. You may get better results for other organelles (e.g. ER or Golgi) with the default Segment Anything models.
+
+See [the dataset overview](https://github.com/computational-cell-analytics/micro-sam/blob/master/doc/datasets/em_organelles_v%i.md) for further informations on the training data and the [micro_sam documentation](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html) for details on how to use the model for interactive and automatic segmentation.
+
+NOTE: The model's automatic instance segmentation quality has improved as the latest version updates the segmentation decoder architecture by replacing transposed convolutions with upsampling.
+
+
+## Validation
+
+The easiest way to validate the model is to visually check the segmentation quality for your data.
+If you have annotations you can use for validation you can also quantitative validation, see [here for details](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#9-how-can-i-evaluate-a-model-i-have-finetuned).
+Please note that the required quality for segmentation always depends on the analysis task you want to solve.

doc/bioimageio/lm_v4.md

@@ -0,0 +1,19 @@
+# Segment Anything for Light Microscopy
+
+This is a [Segment Anything](https://segment-anything.com/) model that was specialized for light microscopy with [micro_sam](https://github.com/computational-cell-analytics/micro-sam).
+This model uses a %s vision transformer as image encoder.
+
+Segment Anything is a model for interactive and automatic instance segmentation.
+We improve it for light microscopy by finetuning on a large and diverse microscopy dataset.
+It should perform well for cell and nucleus segmentation in fluorescent, label-free and other light microscopy datasets.
+
+See [the dataset overview](https://github.com/computational-cell-analytics/micro-sam/blob/master/doc/datasets/lm_v%i.md) for further informations on the training data and the [micro_sam documentation](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html) for details on how to use the model for interactive and automatic segmentation.
+
+NOTE: The model's automatic instance segmentation quality has improved as the latest version updates the segmentation decoder architecture by replacing transposed convolutions with upsampling.
+
+
+## Validation
+
+The easiest way to validate the model is to visually check the segmentation quality for your data.
+If you have annotations you can use for validation you can also quantitative validation, see [here for details](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#9-how-can-i-evaluate-a-model-i-have-finetuned).
+Please note that the required quality for segmentation always depends on the analysis task you want to solve.

doc/datasets/em_organelles_v4.md

@@ -0,0 +1,7 @@
+# Electron Microscopy Datasets
+
+The `EM Organelle v4` model was trained on three different electron microscopy datasets with segmentation annotations for mitochondria and nuclei:
+
+1. [MitoEM](https://mitoem.grand-challenge.org/): containing segmentation annotations for mitochondria in volume EM of human and rat cortex.
+2. [MitoLab](https://www.ebi.ac.uk/empiar/EMPIAR-11037/): containing segmentation annotations for mitochondria in different EM modalities.
+3. [Platynereis (Nuclei)](https://zenodo.org/records/3675220): contining segmentation annotations for nuclei in a blockface EM volume of *P. dumerilii*.

doc/datasets/lm_v4.md

@@ -0,0 +1,18 @@
+# Light Microscopy Datasets
+
+The `LM Generalist v4` model was trained on 14 different light microscopy datasets with segmentation annotations for cells and nuclei:
+
+1. [LIVECell](https://sartorius-research.github.io/LIVECell/): containing cell segmentation annotations for phase-contrast microscopy.
+2. [DeepBacs](https://github.com/HenriquesLab/DeepBacs): containing segmentation annotations for bacterial cells in different label-free microscopy modalities.
+3. [TissueNet](https://datasets.deepcell.org/): containing cell segmentation annotations in tissues imaged with fluorescence light microscopy.
+4. [PlantSeg (Root)](https://osf.io/2rszy/): containing cell segmentation annotations in plant roots imaged with fluorescence lightsheet microscopy.
+5. [NeurIPS CellSeg](https://neurips22-cellseg.grand-challenge.org/): containg cell segmentation annotations in phase-contrast, brightfield, DIC and fluorescence microscopy.
+6. [CTC (Cell Tracking Challenge)](https://celltrackingchallenge.net/2d-datasets/): containing cell segmentation annotations in different label-free and fluorescence microscopy settings. We make use of the following CTC datasets: `BF-C2DL-HSC`, `BF-C2DL-MuSC`, `DIC-C2DH-HeLa`, `Fluo-C2DL-Huh7`, `Fluo-C2DL-MSC`, `Fluo-N2DH-SIM+`, `PhC-C2DH-U373`, `PhC-C2DL-PSC"`]
+7. [DSB Nucleus Segmentation](https://www.kaggle.com/c/data-science-bowl-2018): containing nucleus segmentation annotations in fluorescence microscopy. We make use of [this subset](https://github.com/stardist/stardist/releases/download/0.1.0/dsb2018.zip) of the data.
+8. [EmbedSeg](https://github.com/juglab/EmbedSeg): containing cell and nuclei annotations in fluorescence microcsopy.
+9. [YeaZ](https://www.epfl.ch/labs/lpbs/data-and-software): containing segmentation annotations for yeast cells in phase contrast and brightfield microscopy.
+10. [CVZ Fluo](https://www.synapse.org/Synapse:syn27624812/): containing cell and nuclei annotations in fluorescence microsocopy.
+11. [DynamicNuclearNet](https://datasets.deepcell.org/): containing nuclei annotations in fluorescence microscopy.
+12. [CellPose](https://www.cellpose.org/): containing cell annotations in fluorescence microscopy.
+13. [OmniPose](https://osf.io/xmury/): containing segmentation annotations for bacterial cells in phase-contrast and fluorescence microscopy, and worms in brightfield microscopy.
+14. [OrgaSegment](https://zenodo.org/records/10278229): containing segmentation annotations for organoids in brightfield microscopy.

finetuning/generalists/training/electron_microscopy/mito_nuc/obtain_mito_nuc_em_datasets.py

@@ -30,10 +30,10 @@ def _check_dataset_available_for_rois(path, patch_shape):
     We do this only for "platynereis - nuclei", "mitoem" datasets - as we expect specific RoIs only from them
     """
     datasets.get_mitoem_dataset(
-        path=os.path.join(path, "mitoem"), patch_shape=patch_shape, download=True, splits="train"
+        path=os.path.join(path, "mitoem"), patch_shape=patch_shape, download=True, splits="train",
     )
     datasets.get_platynereis_nuclei_dataset(
-        path=os.path.join(path, "platynereis"), patch_shape=patch_shape, download=True
+        path=os.path.join(path, "platynereis"), patch_shape=patch_shape, download=True,
     )
     print("All the datasets are available for RoI splitting")
 
@@ -43,42 +43,49 @@ def get_concat_mito_nuc_datasets(input_path, patch_shape):
 
     sampler = MinInstanceSampler()
     standard_label_trafo = PerObjectDistanceTransform(
-        distances=True, boundary_distances=True, directed_distances=False, foreground=True, instances=True, min_size=0
+        distances=True, boundary_distances=True, directed_distances=False, foreground=True, instances=True, min_size=0,
     )
 
-    # mitoem parameters
+    # MitoEM dataset: ROI values.
     mitoem_train_rois = [np.s_[100:110, :, :], np.s_[100:110, :, :]]
     mitoem_val_rois = [np.s_[0:5, :, :], np.s_[0:5, :, :]]
 
-    # platynereis nuclei dataset parameters
+    # Platynereis (Nuclei) dataset: ROI values.
     platy_root = os.path.join(input_path, "platynereis")
     platy_nuclei_template = "nuclei/train_data_nuclei_%02i.h5"
     platy_nuclei_label_key = "volumes/labels/nucleus_instance_labels"
 
     platy_nuclei_train_samples = [1, 2, 3, 4, 5, 6, 7, 8]
-    platy_nuclei_train_rois = compute_platy_rois(platy_root, platy_nuclei_train_samples, ignore_label=-1,
-                                                 file_template=platy_nuclei_template, label_key=platy_nuclei_label_key)
+    platy_nuclei_train_rois = compute_platy_rois(
+        platy_root, platy_nuclei_train_samples, ignore_label=-1,
+        file_template=platy_nuclei_template, label_key=platy_nuclei_label_key,
+    )
+
     platy_nuclei_val_samples = [9, 10]
-    platy_nuclei_val_rois = compute_platy_rois(platy_root, platy_nuclei_val_samples, ignore_label=-1,
-                                               file_template=platy_nuclei_template, label_key=platy_nuclei_label_key)
+    platy_nuclei_val_rois = compute_platy_rois(
+        platy_root, platy_nuclei_val_samples, ignore_label=-1,
+        file_template=platy_nuclei_template, label_key=platy_nuclei_label_key,
+    )
 
     def mitoem_dataset(split, roi_choice):
         return datasets.get_mitoem_dataset(
             path=os.path.join(input_path, "mitoem"), splits=split, download=True, patch_shape=patch_shape,
             rois=roi_choice, label_transform=standard_label_trafo, ndim=2, raw_transform=identity,
-            sampler=MinInstanceSampler(min_num_instances=5)
+            sampler=MinInstanceSampler(min_num_instances=4),
         )
 
+    # Get MitoEM dataset: mitochondria segmentation in vEM images.
     mitoem_train_dataset = mitoem_dataset("train", mitoem_train_rois)
     mitoem_val_dataset = mitoem_dataset("val", mitoem_val_rois)
 
     def platy_nuclei_dataset(roi_choice, sample_ids):
         return datasets.get_platynereis_nuclei_dataset(
             path=platy_root, patch_shape=patch_shape, download=True, sampler=sampler, ndim=2,
             label_transform=ResizeLabelTrafo(patch_shape[1:]), rois=roi_choice,
-            raw_transform=ResizeRawTrafo(patch_shape[1:], do_rescaling=False), sample_ids=sample_ids
+            raw_transform=ResizeRawTrafo(patch_shape[1:], do_rescaling=False, ensure_rgb=False), sample_ids=sample_ids
         )
 
+    # Get Platynereis (Nuclei) dataset: nuclei segmentation in vEM images.
     platy_nuclei_train_dataset = platy_nuclei_dataset(platy_nuclei_train_rois, sample_ids=platy_nuclei_train_samples)
     platy_nuclei_val_dataset = platy_nuclei_dataset(platy_nuclei_val_rois, sample_ids=platy_nuclei_val_samples)
 
@@ -87,10 +94,11 @@ def cem_dataset(split):
         n_samples = 1620 if split == "train" else 600
         return datasets.cem.get_mitolab_dataset(
             path=os.path.join(input_path, "mitolab"), split=split, val_fraction=0.1, sampler=sampler,
-            raw_transform=ResizeRawTrafo(patch_shape[1:], do_rescaling=False), patch_shape=patch_shape[1:],
-            label_transform=ResizeLabelTrafo(patch_shape[1:]), n_samples=n_samples
+            raw_transform=ResizeRawTrafo(patch_shape[1:], do_rescaling=False, ensure_rgb=False),
+            patch_shape=patch_shape[1:], label_transform=ResizeLabelTrafo(patch_shape[1:]), n_samples=n_samples,
         )
 
+    # Get CEM dataset: mitochondria segmentation in vEM images.
     cem_train_dataset = cem_dataset("train")
     cem_val_dataset = cem_dataset("val")
 
@@ -111,14 +119,19 @@ def cem_dataset(split):
 
 def get_generalist_mito_nuc_loaders(input_path, patch_shape):
     """This returns the concatenated electron microscopy datasets implemented in torch_em:
-    https://github.com/constantinpape/torch-em/tree/main/torch_em/data/datasets
+    https://github.com/constantinpape/torch-em/tree/main/torch_em/data/datasets/electron_microscopy
     It will automatically download all the datasets
+
     NOTE: to remove / replace the datasets with another dataset, you need to add the datasets (for train and val splits)
     in `get_concat_lm_dataset`. The labels have to be in a label mask instance segmentation format.
     i.e. the tensors (inputs & masks) should be of same spatial shape, with each object in the mask having it's own ID.
     IMPORTANT: the ID 0 is reserved for background, and the IDs must be consecutive.
     """
+    # Get the datasets.
     generalist_train_dataset, generalist_val_dataset = get_concat_mito_nuc_datasets(input_path, patch_shape)
+
+    # Get the dataloaders.
     train_loader = get_data_loader(generalist_train_dataset, batch_size=2, shuffle=True, num_workers=16)
     val_loader = get_data_loader(generalist_val_dataset, batch_size=1, shuffle=True, num_workers=16)
+
     return train_loader, val_loader