Update doc and extend finetuned model examples

Author: constantinpape

doc/installation.md

@@ -1,41 +1,43 @@
 # Installation
 
 We provide three different ways of installing `micro_sam`:
-- [From conda](#from-conda) is the recommended way if you want to use all functionality.
-- [From source](#from-source) for setting up a development environment to change and potentially contribute to our software.
+- [From mamba](#from-mamba) is the recommended way if you want to use all functionality.
+- [From source](#from-source) for setting up a development environment to use the latest version and be able to change and contribute to our software.
 - [From installer](#from-installer) to install without having to use conda. This mode of installation is still experimental! It only provides the annotation tools and does not enable model finetuning.
 
 Our software requires the following dependencies:
 - [PyTorch](https://pytorch.org/get-started/locally/)
 - [SegmentAnything](https://github.com/facebookresearch/segment-anything#installation)
 - [elf](https://github.com/constantinpape/elf)
+- [torch_em](https://github.com/constantinpape/torch-em)
 - [napari](https://napari.org/stable/) (for the interactive annotation tools)
-- [torch_em](https://github.com/constantinpape/torch-em) (for the training functionality)
 
-## From conda
 
-`micro_sam` is available as a conda package and can be installed via
-```
-$ conda install -c conda-forge micro_sam
-```
+## From mamba
+
+[mamba](https://mamba.readthedocs.io/en/latest/) is a drop-in replacement for conda, but is much faster than it.
+While the steps below may also work with `conda`, we highly recommend using `mamba`.
+You can follow the instrutions [here](https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html) to install `mamba`.
 
-This command will not install the required dependencies for the annotation tools and for training / finetuning.
-To use the annotation functionality you also need to install `napari`:
+`micro_sam` can then be installed in an existing environment via
 ```
-$ conda install -c conda-forge napari pyqt
+$ mamba install -c conda-forge micro_sam
 ```
-And to use the training functionality `torch_em`:
+or you can create a new environment (here called `micro-sam`) via
 ```
-$ conda install -c conda-forge torch_em
+$ mamba create -c conda-forge -n micro-sam micro_sam
 ```
 
-In case the installation via conda takes too long consider using [mamba](https://mamba.readthedocs.io/en/latest/).
-Once you have it installed you can simply replace the `conda` commands with `mamba`.
+You also need to install napari to use the annotation tool:
+```
+$ mamba install -c conda-forge napari pyqt
+```
+(We don't include napari in the default installation dependencies to keep the choice of rendering backend flexible.)
 
 
 ## From source
 
-To install `micro_sam` from source, we recommend to first set up a conda environment with the necessary requirements:
+To install `micro_sam` from source, we recommend to first set up an environment with the necessary requirements:
 - [environment_gpu.yaml](https://github.com/computational-cell-analytics/micro-sam/blob/master/environment_gpu.yaml): sets up an environment with GPU support.
 - [environment_cpu.yaml](https://github.com/computational-cell-analytics/micro-sam/blob/master/environment_cpu.yaml): sets up an environment with CPU support.
 
@@ -52,11 +54,11 @@ $ cd micro_sam
 3. Create the GPU or CPU environment:
 
 ```
-$ conda env create -f <ENV_FILE>.yaml
+$ mamba env create -f <ENV_FILE>.yaml
 ```
 4. Activate the environment:
 ```
-$ conda activate sam
+$ mamba activate sam
 ```
 5. Install `micro_sam`:
 ```
@@ -65,7 +67,6 @@ $ pip install -e .
 
 **Troubleshooting:**
 
-- On some systems `conda` is extremely slow and cannot resolve the environment in the step `conda env create ...`. You can use `mamba` instead, which is a faster re-implementation of `conda`. It can resolve the environment in less than a minute on any system we tried. Check out [this link](https://mamba.readthedocs.io/en/latest/installation.html) for how to install `mamba`. Once you have installed it, run `mamba env create -f <ENV_FILE>.yaml` to create the env.
 - Installation on MAC with a M1 or M2 processor:
     - The pytorch installation from `environment_cpu.yaml` does not work with a MAC that has an M1 or M2 processor. Instead you need to:
         - Create a new environment: `mamba create -c conda-forge python pip -n sam`
@@ -89,7 +90,7 @@ We also provide installers for Linux and Windows:
 
 **The installers are stil experimental and not fully tested.** Mac is not supported yet, but we are working on also providing an installer for it.
 
-If you encounter problems with them then please consider installing `micro_sam` via [conda](#from-conda) instead.
+If you encounter problems with them then please consider installing `micro_sam` via [mamba](#from-mamba) instead.
 
 **Linux Installer:**
 

doc/python_library.md

@@ -26,4 +26,9 @@ TODO: provide link to the paper with results on how much data is needed
 
 The training logic is implemented in `micro_sam.training` and is based on [torch-em](https://github.com/constantinpape/torch-em). Please check out [examples/finetuning](https://github.com/computational-cell-analytics/micro-sam/tree/master/examples/finetuning) to see how you can finetune on your own data with it. The script `finetune_hela.py` contains an example for finetuning on a small custom microscopy dataset and `use_finetuned_model.py` shows how this model can then be used in the interactive annotation tools.
 
+Since release v0.4.0 we also support training an additional decoder for automatic instance segmentation. This yields better results than the automatic mask generation of segment anything and is significantly faster.
+You can enable training of the decoder by setting `train_instance_segmentation = True` [here](https://github.com/computational-cell-analytics/micro-sam/blob/master/examples/finetuning/finetune_hela.py#L165).
+The script `instance_segmentation_with_finetuned_model.py` shows how to use it for automatic instance segmentation.
+We will fully integrate this functionality with the annotation tool in the next release.
+
 More advanced examples, including quantitative and qualitative evaluation, of finetuned models can be found in [finetuning](https://github.com/computational-cell-analytics/micro-sam/tree/master/finetuning), which contains the code for training and evaluating our microscopy models.

examples/finetuning/README.md

@@ -0,0 +1,6 @@
+# Example for model finetuning
+
+This folder contains example scripts that show how to finetune a SAM model on your own data and how the finetuned model can then be used:
+- `finetune_hela.py`: Shows how to finetune the model on new data. Set `train_instance_segmentation` (line 165) to `True` in order to also train a decoder for automatic instance segmentation.
+- `annotator_with_finetuned_model.py`: Use the finetuned model in the 2d annotator.
+- `instance_segmentation_with_finetuned_model`: Use the finetuned model for automatic instance segmentation (only if you have trained with `train_instance_segmentation = True`).

…amples/finetuning/use_finetuned_model.py …tuning/annotator_with_finetuned_model.pyexamples/finetuning/use_finetuned_model.py renamed to examples/finetuning/annotator_with_finetuned_model.py examples/finetuning/use_finetuned_model.py renamed to examples/finetuning/annotator_with_finetuned_model.py

@@ -4,7 +4,7 @@
 from micro_sam.sam_annotator import annotator_2d
 
 
-def run_annotator_with_custom_model():
+def run_annotator_with_finetuned_model():
     """Run the 2d anntator with a custom (finetuned) model.
 
     Here, we use the model that is produced by `finetuned_hela.py` and apply it
@@ -30,4 +30,4 @@ def run_annotator_with_custom_model():
 
 
 if __name__ == "__main__":
-    run_annotator_with_custom_model()
+    run_annotator_with_finetuned_model()

examples/finetuning/instance_segmentation_with_finetuned_model.py

@@ -0,0 +1,46 @@
+import napari
+import imageio.v3 as imageio
+
+from micro_sam.instance_segmentation import (
+    load_instance_segmentation_with_decoder_from_checkpoint, mask_data_to_segmentation
+)
+from micro_sam.util import precompute_image_embeddings
+
+
+def run_instance_segmentation_with_finetuned_model():
+    """Run the instance segmentation with the finetuned model.
+
+    Here, we use the model that is produced by `finetuned_hela.py` and apply it
+    for an image from the validation set.
+    This only works if 'instance_segmentation_with_decoder' was set to true.
+    """
+    # take the last frame, which is part of the val set, so the model was not directly trained on it
+    image = imageio.imread("./data/DIC-C2DH-HeLa.zip.unzip/DIC-C2DH-HeLa/01/t083.tif")
+
+    # set the checkpoint and the path for caching the embeddings
+    checkpoint = "./checkpoints/sam_hela/best.pt"
+    embedding_path = "./embeddings/embeddings-finetuned.zarr"
+
+    model_type = "vit_b"  # We finetune a vit_b in the example script.
+    # Adapt this if you finetune a different model type, e.g. vit_h.
+
+    # Create the segmenter.
+    segmenter = load_instance_segmentation_with_decoder_from_checkpoint(checkpoint, model_type=model_type)
+    image_embeddings = precompute_image_embeddings(segmenter._predictor, image, embedding_path)
+
+    # Compute the segmentation for the current image:
+    # First initialize the segmenter.
+    segmenter.initialize(image, image_embeddings)
+    # Then compute the actual segmentation. You can set different hyperparameters here,
+    # see the function description of 'generate' for details
+    masks = segmenter.generate(output_mode="binary_mask")
+    segmentation = mask_data_to_segmentation(masks, with_background=True)
+
+    viewer = napari.Viewer()
+    viewer.add_image(image)
+    viewer.add_labels(segmentation)
+    napari.run()
+
+
+if __name__ == "__main__":
+    run_instance_segmentation_with_finetuned_model()