setting size_divisibility changes the shape of output sem_seg mask

[srushti-rashmishirish]

Hello kind friends at Detectron2,

I am using the deeplab v3+ backbone with detectron 2.
For my particular use case, I have found that if I input an image of arbitrary size (for e.g. 5,7) the sem_seg mask shape is always a nearest higher multiple of 4. (in this case 8,8).
This behavior doesn't change even after I set the size_divisibility as 4.
After setting the size_divisibility as 4, the input images are getting padded appropriately, but seems like the padding is scrapped off while writing a sem_seg mask.

Any pointers on if this is expected or if I am missing any other flag that I need to set?

Thanks a lot for you help in advance