failure and errors running finetune_hela.py

[hzqfox]

Dear micro-sam developers,

I'm Ziqiang Huang, image analyst at the Imaging Centre at EMBL Heidelberg. I'm trying to use micro-sam to segment some single channel flurorescent images of some special cell type, that has morphology like microglia and quite irregular in shape, size, and morphology.

I created a set of training data, 54 images in total, all 650 by 650 pixel 2D (originally 16-bit image), and converted to 8-bit. I created their corresponding masks with micro-sam 2D annotator.
I'm making use of the finetune_hela.py script in finetuning example to train my own model. However despite I tried myself to fix a few places, it still doesn't work. I found some place in the code it requires only 8-bit image? And some place it needs all the input data to be the same size?
I changed in finetune_hela.py:
line 34-35: the path of image_dir and segmentation_dir to my input data folder and mask folder;
line 54, 56: roi = np.s_[:30, :, :] and roi= np.s_[30:, :, :]
line 83: patch_shape = (1, 650, 650) # is this necessar?
line 84: n_objects_per_batch = 1 # I have various number of objects in each of the input image/batch, but majority has 1, I tried both keep all of them, and remove all but 1, and neither worked
line 124: model_type = "vit_b_lm"

I got the environment setup correctly with the instruction and can use the cuda device successfully.

When running the script I got initially the training OK, but reporting during training looks wrong with "current metric" always be 'nan', and 'best metric' always being 'inf'

More detailed output from the console I pasted it here, for your reference:

(sam) C:\Users\...\micro-sam\examples\finetuning>python -m finetune_hela

Verifying labels in 'train' dataloader:  36%|██████████████████████████████████████▉                                                                     | 18/50 [00:24<00:43,  1.36s/it]
Verifying labels in 'val' dataloader:  72%|███████████████████████████████████████████████████████████████████████████████▏                              | 36/50 [00:25<00:09,  1.44it/s]
Start fitting for 1800 iterations /  100 epochs
with 18 iterations per epoch
Training with mixed precision
Epoch 11: average [s/it]: 39.558808, current metric: nan, best metric: inf:  12%|███████▋                                                        | 216/1800 [4:52:37<16:46:31, 38.13s/it]Stopping training because there has been no improvement for 10 epochs
Finished training after 11 epochs / 216 iterations.
The best epoch is number 0.
Epoch 11: average [s/it]: 39.558808, current metric: nan, best metric: inf:  12%|███████▋                                                        | 216/1800 [5:02:29<36:58:18, 84.03s/it]
Training took 18249.626536607742 seconds (= 05:304:10 hours)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\...\micro-sam\examples\finetuning\finetune_hela.py", line 141, in <module>
    main()
  File "C:\Users\...\micro-sam\examples\finetuning\finetune_hela.py", line 137, in main
    export_model(checkpoint_name, model_type)
  File "C:\Users\...\micro-sam\examples\finetuning\finetune_hela.py", line 113, in export_model
    export_custom_sam_model(
  File "C:\Users\...\micro-sam\micro_sam\util.py", line 508, in export_custom_sam_model
    _, state = get_sam_model(
               ^^^^^^^^^^^^^^
  File "C:\Users\...\micro-sam\micro_sam\util.py", line 376, in get_sam_model
    raise ValueError(f"Checkpoint at {checkpoint_path} could not be found.")
ValueError: Checkpoint at checkpoints\sam_silvia3\best.pt could not be found.

I attachd a image and mask pair for your reference:

[anwai98]

Hi @hzqfox,

Thanks for your interest in micro-sam. And thanks for sharing all the relevant details about your task at hand!

I have a few questions:

• Are you training micro-sam with the additional instance segmentation decoder or without? (this is important for us to understand the reason for NaN values in the loss computed)
• In the image-mask pair provided above, I see there are two similar objects placed on top of each other, however the one in bottom is the only one annotated. Is there a reason for this / is the bottom object the only one you'd like to segment?

I'll try to answer other smaller questions meanwhile:

However despite I tried myself to fix a few places, it still doesn't work.

Hmm, this should not be the case. Could you share the error trace where things break exactly? (or is loss going to NaN values the core issue)

I found some place in the code it requires only 8-bit image?

Yes, that's right!

And some place it needs all the input data to be the same size?

You don't need to prepare it beforehand. The dataloaders should be able to pad / crop over smaller / larger images (respectively). Let me know if you come across any issues on this!

When running the script I got initially the training OK, but reporting during training looks wrong with "current metric" always be 'nan', and 'best metric' always being 'inf'

Do you observe a pattern of any port? (eg. the loss goes NaN after the 3rd epoch / something like that)

Also, if you'd like, please feel free to send over a few images so that I can test a training setup and see if I can reproduce what's going on!

[hzqfox]

Hi Anwai,

Thank you for getting back to me so quickly.
I'm not entirely sure that I get all the points correct but I will try to answer the raised questions:
1, I am trying to train micro-sam on top of the vit_b_lm, basically because the image modality and size. However aside from the modification on line 124, I didn't adjust other parts that might related to the instance segmentation decoder.
2, I have multiple objects labelled in most cases, with the annotator 2d workflow. However the initial attempt with the multiple objects in mask also failed. Though I don't think the failure or Nan values are from this. As a consequence I tried again with only 1 object per mask, partially because I see in line 84: there's a variable n_objects_per_batch, and I assume it expecting a fixed number of object per mask? So I set it to 1, and modified the masks so each of them contains only 1 object. Unfortunately this also doesn't work.
3, I did try in the very beginnig to provide different sizes of input image and mask pairs. But the code run into error which I don't remember precisely where was it. I figured back then I need to make all the input images the same size, at least for the demo finetune_hela script.
4, the loss is always NaN, while the best metric is always Inf. And I never get a better metric then the 1st epoch

I attached the data and mask pair for your reference, if you can figure out a training setup that works with them, that will be great!
dataset.zip

Best Regards,
Ziqiang

[anwai98]

Hi @hzqfox,

Thanks for the detailed response!

1, I am trying to train micro-sam on top of the vit_b_lm, basically because the image modality and size. However aside from the modification on line 124, I didn't adjust other parts that might related to the instance segmentation decoder.

I see, okay. To be precise, what's the value you set for train_instance_segmentation_decoder, passed to sam_training.train_sam function? (this determines if you are training with the additional decoder or without) (although my assumption is you train with the additional decoder, since it's the default, but would be good to confirm)

I see in line 84: there's a variable n_objects_per_batch, and I assume it expecting a fixed number of object per mask?

Well it's not really a training framework limitation per-say. You can train with as much objects as you want. What n_objects_per_batch essentially does is to limit the number of objects we train on per batch, primarily due to hardware limitations!

But the code run into error which I don't remember precisely where was it.

You are right. With the default setup from the example scripts, it is the expectation. However, setting is_seg_dataset to False would have sorted you right away ;)

4, the loss is always NaN, while the best metric is always Inf. And I never get a better metric then the 1st epoch

I see.

I attached the data and mask pair for your reference, if you can figure out a training setup that works with them, that will be great!

This is awesome, thanks! I tried training on your data, and works off the shelf without any issues - no NaNs in loss or metric values encountered until 10 epochs! (here's the PR for it: #1086)

FYI, I am on the latest dev for micro-sam and latest main for torch-em and needed no code changes to train a model!

I'm not entirely sure that I get all the points correct but I will try to answer the raised questions

Please let me know if something's still unclear. I'd be happy to elaborate!

[hzqfox]

I got the following error when set is_seg_dataset to False:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "...\micro-sam\examples\finetuning\finetune_hela.py", line 144, in <module>
    main()
  File "...\micro-sam\examples\finetuning\finetune_hela.py", line 139, in main
    run_training(checkpoint_name, model_type, train_instance_segmentation)
  File "...\micro-sam\examples\finetuning\finetune_hela.py", line 92, in run_training
    train_loader = get_dataloader("train", patch_shape, batch_size, train_instance_segmentation)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\micro-sam\examples\finetuning\finetune_hela.py", line 71, in get_dataloader
    loader = torch_em.default_segmentation_loader(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\segmentation.py", line 266, in default_segmentation_loader
    ds = default_segmentation_dataset(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\segmentation.py", line 391, in default_segmentation_dataset
    ds = _load_image_collection_dataset(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\segmentation.py", line 157, in _load_image_collection_dataset
    raw_paths, label_paths = _get_paths(raw_paths, raw_key, label_paths, label_key, roi)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\segmentation.py", line 144, in _get_paths
    rpath, lpath = rpath[roi], lpath[roi]
                   ~~~~~^^^^^
TypeError: list indices must be integers or slices, not tuple

In addition, I cloned a fresh dev branch and directly run the finetune_hela.py with original implementation with downloaded hela sample data. However, I still get the nan and inf values when training the model, even with the downloaded hela sample data... I think my torch_em is also up to date. Do you have any idea where this problem might originate from?

[anwai98]

I got the following error when set is_seg_dataset to False:

Did you try it in my provided script or your own scripts?

However, I still get the nan and inf values when training the model, even with the downloaded hela sample data...

Hmm that is extremely weird and unexpected. Something is definitely off!

I suggest the following:

• Send us your current pytorch version: conda list | grep "pytorch"
• Check if you are on latest torch-em: python -c "import torch_em; print(torch_em.__version__)"
• Check if you are on latest micro-sam: python -c "import micro_sam; print(micro_sam.__version__)"

If our packages are outdated, please install micro-sam and torch-em from source, stick to the branches I suggested above and try again!

Another safe try might be to reinstall everything in a new environment and see if that does the trick?!

Also, would make sense to give Google Colab (https://colab.research.google.com/) or Kaggle Notebooks (https://www.kaggle.com/code) a shot to see if the training works there for your data or is something going on with your environment / setup!

[hzqfox]

I will try to reinstall everything in a new environment and try the sample again.

Meanwhile I am sending over the related info here:

• pytorch version:
  libopenvino-pytorch-frontend 2025.0.0 he0c23c2_3 conda-forge
  pytorch 2.6.0 cuda126_mkl_py312_hd6ead81_304 conda-forge
  pytorch-lightning 2.5.1.post0 pypi_0 pypi

• torch-em version
  0.7.9

• micro-sam version
  1.4.0

I tried the script you provided, and got

• some userwarning:

...\.conda\envs\sam\Lib\site-packages\torch_em\util\util.py:92: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\bld\libtorch_1744233537601\work\torch\csrc\utils\tensor_numpy.cpp:209.)
  tensor = torch.from_numpy(tensor)

• and a different error though also in the torch_em package:

Verifying labels in 'train' dataloader:  20%|█████████████████████████████████████████▏                                                                                                                                                                    | 10/50 [02:52<11:29, 17.23s/it]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "...\Documents\GitHub\micro-sam\examples\finetuning\finetune_silvia.py", line 67, in <module>
    main()
  File "...\Documents\GitHub\micro-sam\examples\finetuning\finetune_silvia.py", line 56, in main
    train_embl_alm_data(checkpoint_name)
  File "...\Documents\GitHub\micro-sam\examples\finetuning\finetune_silvia.py", line 41, in train_embl_alm_data
    sam_training.train_sam(
  File "...\Documents\GitHub\micro-sam\micro_sam\training\training.py", line 242, in train_sam
    _check_loader(train_loader, with_segmentation_decoder, "train", verify_n_labels_in_loader)
  File "...\Documents\GitHub\micro-sam\micro_sam\training\training.py", line 82, in _check_loader
    for x, y in tqdm(
                ^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\tqdm\std.py", line 1181, in __iter__
    for obj in iterable:
               ^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch\utils\data\dataloader.py", line 708, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch\utils\data\dataloader.py", line 1480, in _next_data
    return self._process_data(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch\utils\data\dataloader.py", line 1505, in _process_data
    data.reraise()
  File "...\.conda\envs\sam\Lib\site-packages\torch\_utils.py", line 733, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 10.
Original Traceback (most recent call last):
  File "...\.conda\envs\sam\Lib\site-packages\torch\utils\data\_utils\worker.py", line 349, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
           ^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch\utils\data\_utils\fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
            ~~~~~~~~~~~~^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\data\concat_dataset.py", line 39, in __getitem__
    return ds[idx_in_ds]
           ~~^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\data\segmentation_dataset.py", line 215, in __getitem__
    raw, labels = self._get_sample(index)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.conda\envs\sam\Lib\site-packages\torch_em\data\segmentation_dataset.py", line 187, in _get_sample
    raise RuntimeError(f"Could not sample a valid batch in {self.max_sampling_attempts} attempts")
RuntimeError: Could not sample a valid batch in 500 attempts

[anwai98]

pytorch 2.6.0 cuda126_mkl_py312_hd6ead81_304 conda-forge

Okay, this looks good!

torch-em version: 0.7.9

This is outdated. We've made quite some changes in the past few months. You should update this!

micro-sam version: 1.4.0

Also quite outdated. We have new models, changed some stuff here and there in micro-sam which might be affecting things in some way.

And regarding other errors, it still surprises me why I didn't get the errors. But again, it's hard to debug if we are not on the same package versions!

I will try to reinstall everything in a new environment and try the sample again.

This is the safest bet. I suggest to install the latest micro-sam version in a fresh environment, and see how things go from there! (and please first start with the example finetuning script and then move ahead with finetuning on your own data)

Let me know how it goes!

[hzqfox]

Hi @anwai98

Unfortunately after several attemps on re-install micro-sam (both latest master and dev branch) and with clean env, and also with a script (attached) that utilizing the functions in micro_sam.training, I still get the current metric: nan and best metric: inf ... errors
It is even persist on the freshly cloned repo with provided example finetune_hela.py

At this point, I myself cannot figure out what is the source of this problem any more...
By the way, is it possible to share with me the model you trained (#1086) from the labelled data I provided? It would be quite interesting and helpful to test that model on some of our other images.

The full error message I paste here below:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Start fitting for 600 iterations / 12 epochs
with 50 iterations per epoch
Training with mixed precision
Epoch 11: average [s/it]: 47.745513, current metric: nan, best metric: inf: 100%|███████████████████████████████████████████████████████████████████████████▊| 599/600 [8:14:32<00:47, 47.62s/it]Stopping training because there has been no improvement for 10 epochs
Finished training after 11 epochs / 600 iterations.
The best epoch is number 0.
Epoch 11: average [s/it]: 47.745513, current metric: nan, best metric: inf: 100%|███████████████████████████████████████████████████████████████████████████▊| 599/600 [8:17:02<00:49, 49.79s/it]
Training took 29823.98039507866 seconds (= 08:497:04 hours)
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "...\microsam\train_custom_microsam_model.py", line 84, in
export_instance_segmentation_model(
File "....conda\envs\microsam\Lib\site-packages\micro_sam\training\training.py", line 389, in export_instance_segmentation_model
trained_state = torch.load(trained_model_path, weights_only=False, map_location="cpu")["model_state"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....conda\envs\microsam\Lib\site-packages\torch\serialization.py", line 1319, in load
with _open_file_like(f, "rb") as opened_file:
^^^^^^^^^^^^^^^^^^^^^^^^
File "....conda\envs\microsam\Lib\site-packages\torch\serialization.py", line 659, in _open_file_like
return _open_file(name_or_buffer, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....conda\envs\microsam\Lib\site-packages\torch\serialization.py", line 640, in init
super().init(open(name, mode))
^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/vit_b_lm_smallset/best.pt'
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

train_custom_microsam_model.py

[anwai98]

Hi @hzqfox,

Hmm that sounds concerning. I am sorry to hear that!

Are you using a GPU from the cluster or a local system? Would be great to hear the system configuration from you and search around what might be happening!

At this point, I myself cannot figure out what is the source of this problem any more...

I can recommend trying it on Kaggle Notebooks. They have enough resources to try training a model for you, which would qualify an answer whether it's your system creating the issue or something else fundamentally in the data!

By the way, is it possible to share with me the model you trained (#1086) from the labelled data I provided? It would be quite interesting and helpful to test that model on some of our other images.

Ofc, I would be happy to. I just need to retrain the data because I am confident I ended up deleting it haha. I will train a model for you and send you the link to the trained model later today!

[hzqfox]

I will try with kaggle notebooks, or a different system.
Is it expected to do some preprocessing on the masks before training, or they data type need to be specified? I generated the masks from annotation2D UI, save to tif 32-bit, with background as 0, and each label occupy a integer. In my training script, the mask dypte is uint32, and with exactly the same values, though quite a lot of the mask only contains 1 object so only 0 and 1 values.
Thank you very much for the effort to help generating the trained model.

[anwai98]

Hi @hzqfox,

I sent you a model in our email conversation (sorry for the delay again)!

Is it expected to do some preprocessing on the masks before training, or they data type need to be specified?

For me, all was good to go to train a micro-sam model without any issues. Let me know how the model performs on some new images!