[QUESTION] Custom albumentations for Detection / Instance Segmentation

[james-imi]

🤔 What’s your question?

I am training a DINOV3 LTDETR model for medical xray domain. I have 10K+ unlabeled images and 3K labeled images.

What I want to do is to

• distill/pretrain the DINO backbone to the 10K unlabeled dataset
• finetune the model with a custom augmentation

Since the dataset are radiographs, during fine-tuning the model isnt compatible with some augmentations like Solarize and this specific problem will fail if I add a Flip augmentation (which is the default). Moreover, when deploying, I am doing letterbox padding like what YOLO does.

Is there a way I could use a custom Albumentations for training?

[stegmuel]

Hello,
Thanks for reaching out.
During fine-tuning you can disable random_flip in the following way:

    lightly_train.train_object_detection(
        out="out/my_experiment",
        model="model_name",
        ...,
        transform_args={
            "random_flip": None,
        },
        ...,
    )

Solarization is not used during object detection fine-tuning. Nonetheless, you can also disable it during pre-training:

    lightly_train.pretrain(
        out="out/my_experiment", 
        ...,
        method="distillation",
        transform_args={
            "solarize": None,
            "random_flip": None,
        },
       ...,
    )

On the other hand we do not support user-defined augmentations yet.

[james-imi]

@stegmuel

when i try to do this

    model="dinov3/vits16-eomt-inst-coco",       # Pass the YOLO model.
    steps=100,  # Small number of epochs for demonstration
    batch_size=32,
    num_workers=8,
    seed=42,
    overwrite=True,
    transform_args={
        "image_size": (640, 640),
        # "normalize": {
        #     "mean": [0.485, 0.456, 0.406],
        #     "std": [0.229, 0.224, 0.225],
        # },
        # "val"
    },

i get this error

 File "/home/.../envs/sam/lib/python3.10/site-packages/albumentations/augmentations/crops/transforms.py", line 130, in get_params_dependent_on_data
    raise CropSizeError(
albumentations.augmentations.crops.transforms.CropSizeError: Crop size (height, width) exceeds image dimensions (height, width): (640, 640) vs (320, 320)

i tried adding the image_size explicitly but it doesnt work.

if i change it to

model="dinov3/vitb16-eomt-inst-coco",       # Pass the YOLO model.

it gives this error

   pytorch/transforms.py", line 49, in apply_to_mask
    return torch.from_numpy(mask)
ValueError: At least one stride in the given numpy array is negative, and tensors with negative strides are not currently supported. (You can probably work around this by making a copy of your array  with array.copy().) 

with '2.4.0+cu121' torch

[james-imi]

@stegmuel how about for training? like object detection?

[james-imi]

this one also yields a weird error even thought this is from the documentation

    transform_args={
        "image_size": (640, 640),
        "random_resize": None,
        "normalize": {
            "mean": [0.485, 0.456, 0.406],
            "std": [0.229, 0.224, 0.225],
        },
    },

ValidationError: 4 validation errors for DINOv3EoMTInstanceSegmentationTrainTransformArgs
normalize.NormalizeArgs.mean
  Input should be a valid tuple [type=tuple_type, input_value=[0.485, 0.456, 0.406], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/tuple_type
normalize.NormalizeArgs.std
  Input should be a valid tuple [type=tuple_type, input_value=[0.229, 0.224, 0.225], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/tuple_type
normalize.literal['auto']
  Input should be 'auto' [type=literal_error, input_value={'mean': [0.485, 0.456, 0...: [0.229, 0.224, 0.225]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/literal_error
random_resize
  Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden

During handling of the above exception, another exception occurred:

ConfigValidationError: Found 4 errors in the config!
  Invalid type for key 'normalize.NormalizeArgs.mean': Input should be a valid tuple but got [0.485, 0.456, 0.406] with type 'list'
  Invalid type for key 'normalize.NormalizeArgs.std': Input should be a valid tuple but got [0.229, 0.224, 0.225] with type 'list'
  Invalid type for key 'normalize.literal['auto']': Input should be 'auto' but got {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]} with type 'dict'
  Unknown key: 'random_resize

EDIT: the normalize error is solved by upgrading pedantic. random_resize error still is wrong

ValidationError: 1 validation error for DINOv3EoMTInstanceSegmentationTrainTransformArgs
random_resize
  Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden

[stegmuel]

The random_resize is because it is not part of the augmentations used in instance segmentation. This page is for distillation.

@stegmuel how about for training? like object detection?

This confuses me. My original answer is for training an object detection model.

model="dinov3/vits16-eomt-inst-coco", # Pass the YOLO model.
steps=100, # Small number of epochs for demonstration
batch_size=32,
num_workers=8,
seed=42,
overwrite=True,
transform_args={
"image_size": (640, 640),
# "normalize": {
# "mean": [0.485, 0.456, 0.406],
# "std": [0.229, 0.224, 0.225],
# },
# "val"
},

Can you share the full block code? I tried it on my side and did not encounter the issue you reported. Can you also provide the full error trace?

[james-imi]

@stegmuel seems to be a specific Pytorch version issue. I tried the training inside the docker and it works.

Now, the problem seems to be that it has an error if your training set has images with NO annotations. Something like cannot stack tensors. I tried removing them and not the training proceeds.

On a side note, I think the documentation has been quite lacking. I've tried debugging it straight from the source code and edited it. For example, documentation doesnt explicitly say how to change how to log things, the steps are still regarded as steps and not epochs, etc.

But I was able to start the training nonetheless.