Detection training fails when tiling enabled with small tiling size

[piotrgrubicki]

Version: 2.12.0-release-20250812052841

1. Create a detection project, annotate images (in my case it is a video but it is most probably not relevant).
2. Start training (in my case DFine-X.
3. Open Advanced settings.
4. Turn on tiling, change the size to 300 (<- a value smaller than most annotations).
5. Start

Training fails. The log complains (full enclosed):

missing keys in source state_dict: decoder.decoder.box_distance_weight

train.log

[piotrgrubicki]

The same problem for segmentation. Logs:

train2.log

[eugene123tw]

There seems to be an issue with manual tiling, possibly caused by incorrect tile configuration. For now, I recommend lowering the exposure and using automatic tiling while we work on a fix.

Attach DFine-X with automatic tiling:

[eugene123tw]

The root cause is an incorrect tile configuration. Out of 9 annotated images, 7 contain large objects (see example). The manually set tile size of 300×300 is smaller than the annotation boxes, so when these images are split into tiles, most annotations don’t fit within a single tile. This results in too few valid annotations for training, validation, and testing, causing the failure.

This issue is coming from manual misconfiguration. Automatic tiling is designed to avoid exactly this problem. I recommend closing this ticket and either:

• Use automatic tiling, or
• Manually set a tile size (e.g., 600×600) that is larger than the largest object in the dataset.

[leoll2]

The manually set tile size of 300×300 is smaller than the annotation boxes, so when these images are split into tiles, most annotations don’t fit within a single tile. This results in too few valid annotations for training, validation, and testing, causing the failure.

@eugene123tw could OTX assert this condition and raise a more informative error (e.g. "Subset X does not have enough annotations. This may happen if the tile size is too small, as most of the annotations can't fit in a tile and get discarded.")? Right now the error is missing keys in source state_dict: decoder.decoder.box_distance_weight which is hard to decode.

On Geti side, we can also improve the description of the "Tile size" configurable parameter, to recommend choosing a value larger than most annotations.

[leoll2]

On Geti side, we can also improve the description of the "Tile size" configurable parameter, to recommend choosing a value larger than most annotations.

I created a PR for this: #1108

[eugene123tw]

@leoll2 Thanks for your feedback, I will create a PR in OTX: open-edge-platform/training_extensions#4519

[eugene123tw]

open-edge-platform/training_extensions#4520