Add clamping_mode for KeyPoints

[AntoineSimoulin]

Context

This PR proposes to add a clamping_mode parameter for KeyPoints. This parameter is similar to what have been added for BoundingBoxes (#9128). We are adding this parameter here to avoid the issue observed in #9223, where clamping modifies the position of KeyPoints and create a misalignment with the actual locations in the transformed image. This PR will need to be supplemented with another PR to implement a class similar to SanitizeBoundingBoxes transform to remove non valid KeyPoints, e.g. outside of the image canvas.

Implementation details

In the current proposition, we align the possible values for the clamping_mode parameters with what is already existing for BoundingBoxes, that is "soft", "hard", and None. We would like KeyPoints clamping mode to be consistent with existing data structures. We propose the following behaviors for each value:

• "soft": this is the default value if not specified otherwise. This will not apply any clamping operation;
• None: This will not apply any clamping operation;
• "hard": This will apply the current default transformation with each coordinates being bounded between 0 and canvas_size - 1.

We choose this behavior as we believe that we want the default behavior to avoid creating misalignment with the transformed image. In this case, KeyPoints are not clamped unless explicitly setting clamping_mode="hard".

Illustration of the changes

We illustrate below how those changes impact the code.

# we manually set bounding boxes and key points for the demo; in a real pipeline, these would come from the data or a CV model’s output.
orig_pts = tv_tensors.KeyPoints(
    [
        [
            [445, 700],  # nose
            [320, 660],
            [370, 660],
            [420, 660],  # left eye
            [300, 620],
            [420, 620],  # left eyebrow
            [475, 665],
            [515, 665],
            [555, 655],  # right eye
            [460, 625],
            [560, 600],  # right eyebrow
            [370, 780],
            [450, 760],
            [540, 780],
            [450, 820],  # mouth
        ],
    ],
    canvas_size=(orig_img.size[1], orig_img.size[0]),
    clamping_mode="hard"
)
cropper = v2.RandomCrop(size=(128, 128))
crops = [cropper((orig_img, orig_pts)) for _ in range(4)]
plot([(orig_img, orig_pts)] + crops)

With clamping_mode="hard"

With clamping_mode="soft" or None

Testing

Please run the following unit tests

pytest test/test_transforms_v2.py -vvv -k "keypoints"
...
718 passed, 9057 deselected in 2.91s

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9234

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 37d73d4 with merge base 58eb039 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Callidior]

Looks good to me overall. Just a few minor remarks.

[Callidior]

clamping_mode_tuto in the docs currently only covers bounding boxes and would need to be updated as well.

[Callidior]

With CLAMPING_MODE_TYPE now being used for both bounding boxes and keypoints alike, it would make sense to move its definition out of _bounding_boxes, e.g., directly to tv_tensors.__init__.

[Callidior]

I think it could be useful to allow setting the clamping modes of bounding boxes and keypoints to different values by passing a dictionary.

For example:

• SetClampingMode("soft") sets the clamping mode of both bounding boxes and keypoints to "soft".
• SetClampingMode({tv_tensors.BoundingBoxes: "hard", tv_tensors.KeyPoints: "soft"}) sets the clamping mode of bounding boxes to "hard" and that of keypoints to "soft".
• SetClampingMode({tv_tensors.BoundingBoxes: "hard"}) sets the clamping mode of bounding boxes to "hard" and leaves that of keypoints unchanged.

[Callidior]

Looks like this line is redundant and can be removed.