Artifacts when applying fine-tuned model to large image

[maxeyje]

After fine-tuning the model with an image size of (4096, 6144), I am attempting to run it on a very large sample of size (80896,242688). In the automatic instance segmentation function, the tile_size is set to [3596, 5644] with a halo of [500, 500]. Unfortunately, this produces a large number of errors/artifacts, especially in regions that are all 0 (even though examples of edge images were included in the fine-tuning set). Do you have any recommendations to improve the application? Thank you for your help!

[anwai98]

Hi @maxeyje,

Happy to hear back from you. And thanks once again for your interest in micro-sam.

Could you help us with a few more details?

1. What are these large images about? Are these some microscopy images? And what is the target object you are trying to segment / train(ed) the model on?
2. What is the patch shape you trained the model on? (I see you mentioned an image size value, not sure if that's what you trained with. If yes, I might have an answer for you)
3. And just out of curiosity, how large are each objects in your images?

On a higher level, micro-sam resizes all inputs internally to (1024, 1024). So assuming for a second that the image size of (4096, 6144) is the patch shape you trained on, micro-sam resized the images to (1024, 1024). Now for inference, you have to keep in mind that maybe the training didn't go as expected because each patches were downsized further and the model is picking up random stuff (as you observe)

Once I have an answer from you for the questions above, we can try to figure out the best way-to-go with this / retrain a proper model!

PS. Would be nice if you could share an image, if possible (so that I can understand the task better)

[maxeyje]

Thank you for the quick response! The data is 2D EM and we are segmenting the nucleus. The fine-tuning was completed on images that were (4096, 6144) without any tiling (so I am assuming they were internally downsampled to (1024, 1024) based on your response). Interestingly, the fine-tuned model performs almost perfectly on test images (4096, 6144). It is only when I used the tiling function on the larger image that the artifacts appeared. The training tiles were cropped regions from the larger image so the objects in the two data sets should be the same size. The nuclei are on the order of 1286x2361 pixels with significant variance.

[anwai98]

Hi @maxeyje,

Thanks for the detailed context. I understand the reason for choosing the tiling heuristics now.

The training tiles were cropped regions from the larger image so the objects in the two data sets should be the same size.

That is indeed true for the trained model. It is hard to say though what artifacts are getting picked up.

Is the resolution for the trained images and the test images close enough?

Also, would it be okay for you if you could send me the trained model and one or two images? (I understand the images are too massive, a relatively smaller crop with a few nuclei in there should also do the trick). I just wanna see what are the artifacts being picked and if it's easier to get rid of them by changing some parameters! (if it's okay, you can share them to me via email: anwai.archit@gmail.com)

[maxeyje]

Thanks for your response. The resolution for the training and test sets are identical. The only difference is the overall size of the images. I will see if I can get permission to share images. I appreciate your help!

[anwai98]

The only difference is the overall size of the images.

How different are we talking? (could you give me an estimate spatial shapes of training and test images)

I will see if I can get permission to share images. I appreciate your help!

No worries if it's too much stress. It'll be slightly tricky to communicate taking guesses about the data and targets, but if you could find one of your publications / another publication in a similar direction, that should do the trick too!

Another thing you could try (assuming you are giving automatic segmentation a shot) is to observe what are the artifacts. Are they small dots / regions / surface areas which are quite far from being a nucleus? If these regions are small in general, you can increase the min_size argument (should be a part of the GUI / CLI / python scripts) to filter out these small regions. If it's picking up another structure, then the question comes how did the training go (i.e. how many training samples did the loader generate, what was the total number of epochs the model was trained for).

Hope this helps! (I'll leave this issue open for now in case you have any concrete questions)

[maxeyje]

The original test set that the model performs well on are the same size as the training set. The problems only arise with the (80896, 242688) image, which the smaller images are cropped from.

For the errors, there are some in each category. I will increase the min_size argument to help with the small errors! The ones I am most concerned about are actually on a part of the image that is all 0s (the data collection procedure produces data that is not square, introducing regions of black). The resulting objects look like nuclei in shape and size but are on part of the image with no data.

[anwai98]

The resulting objects look like nuclei in shape and size but are on part of the image with no data.

That's indeed concerning. It's hard to say why it pops up though (could be a reason for a severe downsizing of images / some strange border artifacts picked up by the model / normalization artifact). The easiest fix here would be to match the segmentations with the image data and check for whether the segmentation is in a valid intensity region, otherwise throw them out! (the simplest heuristic I can think of to filter these out)

[maxeyje]

Thanks! I appreciate all of your help.

[anwai98]

Let us know how this goes ahead @maxeyje. We'll be happy to offer our support!