Merge branch 'master' into dev

constantinpape · constantinpape · commit 7d19c419f802 · 2025-05-28T19:15:30.000+02:00
diff --git a/README.md b/README.md
@@ -58,6 +58,6 @@ We have also built follow-up work that is based on `micro_sam`:
 - https://github.com/computational-cell-analytics/medico-sam - improves SAM for medical imaging.
 - https://github.com/computational-cell-analytics/peft-sam - studies parameter efficient fine-tuning for SAM.
 
-## Release Oveverview
+## Release Overview
 
 You can find an overview of changes introduced in previous releases [here](https://github.com/computational-cell-analytics/micro-sam/blob/master/RELEASE_OVERVIEW.md).
diff --git a/doc/faq.md b/doc/faq.md
@@ -6,6 +6,24 @@ If you encounter a problem or question not addressed here feel free to [open an
 ## Installation questions
 
 
+### 0. I am not familiar with `conda` / `mamba` / `micromamba` and would like to get started with a python package manager.
+If you are getting started from scratch, we recommend using `micromamba` to setup your own package manager and creating a new environment with the required dependencies. You can install `micromamba` by following the instructions [here](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html) under the `Automatic install` section. If you are using Linux, macOS, or Git Bash on Windows, the simplest way to of installing `micromamba` is to simply execute the installation script below in your preferred shell:
+
+```bash
+"${SHELL}" <(curl -L micro.mamba.pm/install.sh)
+```
+
+If you are using Windows PowerShell, you should be able to install `micromamba` by running the following command:
+
+```bash
+Invoke-Expression ((Invoke-WebRequest -Uri https://micro.mamba.pm/install.ps1 -UseBasicParsing).Content)
+```
+
+Once you have `micromamba` installed, you can create a new environment with the required dependencies by following the [installation](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#installation) instructions for `micro_sam` (remember to replace `conda` by `micromamba` in the installation commands. For example: to check the list of available environments, `conda env list` becomes `micromamba env list`).
+
+If you have any troubles in the aforementioned steps, remember to first check the [official documentation](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html) for troubleshooting details.
+
+
 ### 1. How to install `micro_sam`?
 The [installation](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#installation) for `micro_sam` is supported in three ways: [from conda](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#from-conda) (recommended), [from source](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#from-source) and [from installers](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#from-installer). Check out our [tutorial video](https://youtu.be/gcv0fa84mCc) to get started with `micro_sam`, briefly walking you through the installation process and how to start the tool.
 
diff --git a/doc/installation.md b/doc/installation.md
@@ -11,7 +11,7 @@ We do **not support** installing `micro_sam` with pip.
 
 ## From conda
 
-`conda` is a python package manager. If you don't have it installed yet you can follow the instructions [here](https://conda-forge.org/download/) to set it up on your system.
+`conda` is a python package manager. If you don't have it installed yet you can follow the instructions [here](https://conda-forge.org/download/) or look into our [installation FAQs](https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#faq) to set it up on your system.
 Please make sure that you are using an up-to-date version of conda to install `micro_sam`.
 You can also use [mamba](https://mamba.readthedocs.io/en/latest/), which is a drop-in replacement for conda, to install it. In this case, just replace the `conda` commands below with `mamba`.
 
diff --git a/doc/start_page.md b/doc/start_page.md
@@ -42,6 +42,8 @@ import micro_sam
 
 It is explained in more detail [here](#using-the-python-library).
 
+We also have a quick start notebook [here](https://github.com/computational-cell-analytics/micro-sam/blob/master/notebooks/quick_start.ipynb).
+
 We provide different finetuned models for microscopy that can be used within our tools or any other tool that supports Segment Anything. See [finetuned models](#finetuned-models) for details on the available models.
 You can also train models on your own data, see [here for details](#training-your-own-model).
 
diff --git a/environment.yaml b/environment.yaml
@@ -27,5 +27,6 @@ dependencies:
     - tqdm
     - timm
     - xarray <2025.3.0
+    - zarr <3.0
     - pip:
         - git+https://github.com/ChaoningZhang/MobileSAM.git
diff --git a/examples/.gitignore b/examples/.gitignore
diff --git a/examples/README.md b/examples/README.md
@@ -10,6 +10,10 @@ And python scripts for automatic segmentation and tracking:
 - `automatic_segmentation.py`: Run automatic segmentation on 2d images.
 - `automatic_tracking.py`: Run automatic tracking on 2d timeseries images.
 
+And examples for using the `micro_sam` automatic segmentation feature:
+- `quick_start.py`: run the automatic segmentation feature of `micro_sam` on an example 2d image.
+- `automatic_segmentation.py`: run the automatic segmentation feature (with an extensive description) of `micro_sam` on some example data.
+
 We provide Jupyter Notebooks for using automatic segmentation and finetuning on some example data in the [notebooks](../notebooks/) folder.
 
 The folder `finetuning` contains example scripts that show how a Segment Anything model can be fine-tuned
diff --git a/examples/quick_start.py b/examples/quick_start.py
@@ -0,0 +1,27 @@
+"""This script is a quick start to the automatic instance segmentation feature
+supported by the Segment Anything for Microscopy models.
+
+NOTE: You can install `micro_sam` in a conda environment like: `conda install -c conda-forge micro_sam`
+"""
+
+import napari
+
+from skimage.data import cells3d
+
+from micro_sam.automatic_segmentation import get_predictor_and_segmenter, automatic_instance_segmentation
+
+
+# Load an example image from the 'scikit-image' library.
+image = cells3d()[30, 0]
+
+# Load the Segment Anything for Microscopy model.
+predictor, segmenter = get_predictor_and_segmenter(model_type="vit_b_lm")
+
+# Run automatic instance segmentation (AIS) on our image.
+instances = automatic_instance_segmentation(predictor=predictor, segmenter=segmenter, input_path=image)
+
+# Visualize the image and corresponding instance segmentation result.
+v = napari.Viewer()
+v.add_image(image, name="Image")
+v.add_labels(instances, name="Prediction")
+napari.run()
diff --git a/finetuning/README.md b/finetuning/README.md
@@ -1,6 +1,15 @@
 # Segment Anything Finetuning
 
-Code for finetuning Segment Anything data on microscopy data and evaluating the finetuned models.
+**Important!!!!!! Read this before using this code or creating an issue** 
+
+This folder contains finetuning and evaluation code for applying Segment Anything model to microscopy data using the `micro_sam` library.
+This code was used for our experiments in the publication, but it may become outdated due to changes in function signatures etc., and often does not use the functionality that we recommend
+to users. We also don't actively maintain the code here.
+Please refer to the [example notebooks](https://github.com/computational-cell-analytics/micro-sam/tree/master/notebooks) and [example scripts](https://github.com/computational-cell-analytics/micro-sam/tree/master/examples) for well maintained and documented `micro_sam` examples. 
+
+
+**Important!!!!!! Read this before using this code or creating an issue**
+
 
 ## Example: LIVECell
 
diff --git a/finetuning/evaluation/cellpose_baseline.py b/finetuning/evaluation/cellpose_baseline.py
@@ -9,7 +9,7 @@
 from micro_sam.evaluation.evaluation import run_evaluation
 
 # from util import get_paths   # for hlrn
-from util import get_pred_paths 
+from util import get_pred_paths
 
 
 # EXPERIMENT_ROOT = "/scratch/projects/nim00007/sam/experiments/benchmarking/cellpose"  # for hlrn
diff --git a/finetuning/evaluation/evaluate_amg.py b/finetuning/evaluation/evaluate_amg.py
@@ -9,7 +9,7 @@
 )
 
 
-def run_amg_inference(dataset_name, model_type, checkpoint, experiment_folder, peft_kwargs):
+def run_amg_inference(dataset_name, model_type, checkpoint, experiment_folder):
     val_image_paths, val_gt_paths = get_paths(dataset_name, split="val")
     test_image_paths, _ = get_paths(dataset_name, split="test")
     prediction_folder = run_amg(
@@ -19,7 +19,6 @@ def run_amg_inference(dataset_name, model_type, checkpoint, experiment_folder, p
         val_image_paths=val_image_paths,
         val_gt_paths=val_gt_paths,
         test_image_paths=test_image_paths,
-        peft_kwargs=peft_kwargs,
     )
     return prediction_folder
 
@@ -35,9 +34,8 @@ def eval_amg(dataset_name, prediction_folder, experiment_folder):
 
 def main():
     args = get_default_arguments()
-    peft_kwargs = {"rank": args.peft_rank, "module": args.peft_module}
     prediction_folder = run_amg_inference(
-        args.dataset, args.model, args.checkpoint, args.experiment_folder, peft_kwargs
+        args.dataset, args.model, args.checkpoint, args.experiment_folder,
     )
     eval_amg(args.dataset, prediction_folder, args.experiment_folder)
 
diff --git a/finetuning/evaluation/evaluate_instance_segmentation.py b/finetuning/evaluation/evaluate_instance_segmentation.py
@@ -10,7 +10,7 @@
 
 
 def run_instance_segmentation_with_decoder_inference(
-    dataset_name, model_type, checkpoint, experiment_folder, peft_kwargs,
+    dataset_name, model_type, checkpoint, experiment_folder,
 ):
     val_image_paths, val_gt_paths = get_paths(dataset_name, split="val")
     test_image_paths, _ = get_paths(dataset_name, split="test")
@@ -21,7 +21,6 @@ def run_instance_segmentation_with_decoder_inference(
         val_image_paths=val_image_paths,
         val_gt_paths=val_gt_paths,
         test_image_paths=test_image_paths,
-        peft_kwargs=peft_kwargs,
     )
     return prediction_folder
 
@@ -37,9 +36,8 @@ def eval_instance_segmentation_with_decoder(dataset_name, prediction_folder, exp
 
 def main():
     args = get_default_arguments()
-    peft_kwargs = {"rank": args.peft_rank, "module": args.peft_module}
     prediction_folder = run_instance_segmentation_with_decoder_inference(
-        args.dataset, args.model, args.checkpoint, args.experiment_folder, peft_kwargs,
+        args.dataset, args.model, args.checkpoint, args.experiment_folder,
     )
     eval_instance_segmentation_with_decoder(args.dataset, prediction_folder, args.experiment_folder)
 
diff --git a/finetuning/evaluation/iterative_prompting.py b/finetuning/evaluation/iterative_prompting.py
@@ -45,10 +45,7 @@ def main():
     start_with_box_prompt = args.box  # overwrite to start first iters' prompt with box instead of single point
 
     # get the predictor to perform inference
-    peft_kwargs = {"rank": args.peft_rank, "module": args.peft_module}
-    predictor = get_sam_model(
-        model_type=args.model, checkpoint_path=args.checkpoint, peft_kwargs=peft_kwargs,
-    )
+    predictor = get_sam_model(model_type=args.model, checkpoint_path=args.checkpoint)
 
     prediction_root = _run_iterative_prompting(
         args.dataset, args.experiment_folder, predictor, start_with_box_prompt, args.use_masks
diff --git a/finetuning/evaluation/precompute_embeddings.py b/finetuning/evaluation/precompute_embeddings.py
@@ -12,10 +12,7 @@
 def main():
     args = get_default_arguments()
 
-    peft_kwargs = {"rank": args.peft_rank, "module": args.peft_module}
-    predictor = get_sam_model(
-        model_type=args.model, checkpoint_path=args.checkpoint, peft_kwargs=peft_kwargs,
-    )
+    predictor = get_sam_model(model_type=args.model, checkpoint_path=args.checkpoint)
     embedding_dir = os.path.join(args.experiment_folder, "embeddings")
     os.makedirs(embedding_dir, exist_ok=True)
 
diff --git a/finetuning/generalists/training/electron_microscopy/boundaries/train_boundaries_em_generalist.py b/finetuning/generalists/training/electron_microscopy/boundaries/train_boundaries_em_generalist.py
@@ -54,7 +54,7 @@ def finetune_boundaries_em_generalist(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=20, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=20)
     train_loader, val_loader = get_generalist_boundaries_loaders(input_path=args.input_path, patch_shape=patch_shape)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/generalists/training/electron_microscopy/mito_nuc/train_mito_nuc_em_generalist.py b/finetuning/generalists/training/electron_microscopy/mito_nuc/train_mito_nuc_em_generalist.py
@@ -54,7 +54,7 @@ def finetune_mito_nuc_em_generalist(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=15, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=15)
     train_loader, val_loader = get_generalist_mito_nuc_loaders(input_path=args.input_path, patch_shape=patch_shape)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/generalists/training/histopathology/train_histopathology_generalist.py b/finetuning/generalists/training/histopathology/train_histopathology_generalist.py
@@ -27,7 +27,7 @@ def finetune_hp_generalist(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10)
     train_loader, val_loader = get_generalist_hp_loaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/generalists/training/light_microscopy/train_lm_generalist.py b/finetuning/generalists/training/light_microscopy/train_lm_generalist.py
@@ -23,7 +23,7 @@ def finetune_lm_generalist(args):
 
     # all the stuff we need for training
     train_loader, val_loader = get_generalist_lm_loaders(input_path=args.input_path, patch_shape=patch_shape)
-    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 5, "verbose": True}
+    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 5}
 
     # Run training.
     sam_training.train_sam(
diff --git a/finetuning/livecell_finetuning.py b/finetuning/livecell_finetuning.py
@@ -54,7 +54,7 @@ def finetune_livecell(args):
 
     # all the stuff we need for training
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
-    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 10, "verbose": True}
+    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 10}
 
     # freeze encoder: 35.02 GB
     # late lora 50% classic: 43.142 GB
diff --git a/finetuning/specialists/resource-efficient/covid_if_finetuning.py b/finetuning/specialists/resource-efficient/covid_if_finetuning.py
@@ -86,7 +86,7 @@ def finetune_covid_if(args):
     train_loader, val_loader = get_dataloaders(
         patch_shape=patch_shape, data_path=args.input_path, n_images=args.n_images
     )
-    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 3, "verbose": True}
+    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 3}
 
     # Run training
     sam_training.train_sam(
diff --git a/finetuning/specialists/training/electron_microscopy/boundaries/cremi_finetuning.py b/finetuning/specialists/training/electron_microscopy/boundaries/cremi_finetuning.py
@@ -104,7 +104,7 @@ def finetune_cremi(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10)
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/specialists/training/electron_microscopy/organelles/asem_finetuning.py b/finetuning/specialists/training/electron_microscopy/organelles/asem_finetuning.py
@@ -111,7 +111,7 @@ def finetune_asem(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10)
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/specialists/training/histopathology/pannuke_finetuning.py b/finetuning/specialists/training/histopathology/pannuke_finetuning.py
@@ -24,7 +24,7 @@ def finetune_pannuke(args):
 
     # all the stuff we need for training
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
-    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 10, "verbose": True}
+    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 10}
 
     # Run training.
     sam_training.train_sam(
diff --git a/finetuning/specialists/training/light_microscopy/deepbacs_finetuning.py b/finetuning/specialists/training/light_microscopy/deepbacs_finetuning.py
@@ -94,9 +94,7 @@ def finetune_deepbacs(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
-        optimizer, mode="min", factor=0.9, patience=50, verbose=True
-    )
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=50)
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/specialists/training/light_microscopy/dynamicnuclearnet_finetuning.py b/finetuning/specialists/training/light_microscopy/dynamicnuclearnet_finetuning.py
@@ -67,7 +67,7 @@ def finetune_dynamicnuclearnet(args):
 
     # all the stuff we need for training
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
-    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 10, "verbose": True}
+    scheduler_kwargs = {"mode": "min", "factor": 0.9, "patience": 10}
 
     # Run training.
     sam_training.train_sam(
diff --git a/finetuning/specialists/training/light_microscopy/neurips_cellseg_finetuning.py b/finetuning/specialists/training/light_microscopy/neurips_cellseg_finetuning.py
@@ -94,7 +94,7 @@ def finetune_neurips_cellseg_root(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=50, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=50)
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/specialists/training/light_microscopy/plantseg_root_finetuning.py b/finetuning/specialists/training/light_microscopy/plantseg_root_finetuning.py
@@ -85,7 +85,7 @@ def finetune_plantseg_root(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10)
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/finetuning/specialists/training/light_microscopy/tissuenet_finetuning.py b/finetuning/specialists/training/light_microscopy/tissuenet_finetuning.py
@@ -86,7 +86,7 @@ def finetune_tissuenet(args):
 
     # all the stuff we need for training
     optimizer = torch.optim.Adam(joint_model_params, lr=1e-5)
-    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10, verbose=True)
+    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10)
     train_loader, val_loader = get_dataloaders(patch_shape=patch_shape, data_path=args.input_path)
 
     # this class creates all the training data for a batch (inputs, prompts and labels)
diff --git a/micro_sam/bioimageio/bioengine_export.py b/micro_sam/bioimageio/bioengine_export.py
@@ -89,7 +89,7 @@ def export_image_encoder(
     weight_output_folder = os.path.join(output_folder, "1")
     os.makedirs(weight_output_folder, exist_ok=True)
 
-    predictor = get_sam_model(model_type=model_type, checkpoint_path=checkpoint_path)
+    predictor = get_sam_model(model_type=model_type, checkpoint_path=checkpoint_path, device="cpu")
     encoder = predictor.model.image_encoder
 
     encoder.eval()
@@ -143,7 +143,8 @@ def export_onnx_model(
     weight_output_folder = os.path.join(output_folder, "1")
     os.makedirs(weight_output_folder, exist_ok=True)
 
-    _, sam = get_sam_model(model_type=model_type, checkpoint_path=checkpoint_path, return_sam=True)
+    _, sam = get_sam_model(model_type=model_type, checkpoint_path=checkpoint_path, return_sam=True, device="cpu")
+
     weight_path = os.path.join(weight_output_folder, "model.onnx")
 
     onnx_model = SamOnnxModel(
diff --git a/micro_sam/evaluation/livecell.py b/micro_sam/evaluation/livecell.py
diff --git a/micro_sam/training/training.py b/micro_sam/training/training.py
diff --git a/notebooks/README.md b/notebooks/README.md
diff --git a/notebooks/automatic_segmentation.ipynb b/notebooks/automatic_segmentation.ipynb
diff --git a/notebooks/inference_and_evaluation.ipynb b/notebooks/inference_and_evaluation.ipynb
diff --git a/notebooks/quick_start.ipynb b/notebooks/quick_start.ipynb
diff --git a/notebooks/sam_finetuning.ipynb b/notebooks/sam_finetuning.ipynb
diff --git a/scripts/for_benchmarking_ais/train_semanticsam.py b/scripts/for_benchmarking_ais/train_semanticsam.py
