How to remove mask/masks from bigger mask?

[innat]

From the instance segmentation model, I've received the predicted mask. However, because of a post situation, I need to remove the small mask/masks from the bigger mask. Please see the figure below

In the above figure, the predicted blue mask is the bigger mask and there are 2 small masks detected inside the blue mask. Now, I need to remove these smaller masks and not the outer masks (i.e. yellow, and green masks).

(So, far I've separated the smaller masks and the bigger mask and used iou to compare them to get the maximum score and thus identify the inner masks. But it's not sound a proper way. Looking for some efficient method.)

[zhanghang1989]

There may not be existing method to do that. You may write a customize function by comparing the boxes