Ignoring order of keypoints

[Macok]

I'm trying to detect game boards in images, and pinpoint their corners' locations (there are 4 corners in every image).
During training, my loss_keypoint value stays high.
I think the problem is that detectron takes into account an order in which keypoints are given in the training set.
Meanwhile, my game boards are perfectly symmetrical squares, so it's impossible to order their corners.
Is it possible to tell detectron to ignore order of keypoints during training?