Inconsistency in segmentation

[SE-Wd]

Hi,

Recently I have segmented the image in the "examples" folder (/examples/torrey_pines_beach/torrey_pines_beach_image.jpg). I attempted to segment two images of different dimensions separately: one being the original-sized image (assuming the variable name after cv2 reading is img), and the other being an image with its height reduced by several pixels (e.g., img[:958, :, :], the origianal height is 960). During the segmentation, I set remove_edge_grains=True. In the end, I discovered that the grain segmentation results for the two images were not entirely identical, with some grains showing inconsistent segmentation, which led to differences in the calculated average area of all grains. What might be the possible reasons for this discrepancy?

Thank you!

[dirtbirb]

Disclaimer: I'm not the dev, but I've worked with this a lot!

I assume the difference isn't just due to a grain or two touching the edge when the height is reduced and disappearing, correct? How significant are these differences? Does the UNET output look much different, or just the final grains?

My understanding is that neural networks are always a bit of a "black box", so it can be hard to explain exactly why they do anything. In general, my impression is that some small error/variance should be expected with even minor changes to the input. If the UNET model suggests a grain by placing prompts in a slightly different place, for example, then the SAM will start with a slightly different example of what it's supposed to look for, which will result in slightly different edges for the final grain. If everything is working well, these differences should be insignificant compared to other sources of error.

It's possible that the post-processing done by this module could be involved as well, so it's not entirely magic. It's also possible that using a newer SAM version or tweaking some input parameters would improve stability. If you're seeing significant differences like entire grains disappearing (that aren't touching the edge) or reporting wildly different areas, it might be worth diving into this further.