RF-DETR 1.1.0 vs 1.2.1 : same train parameters doesn't work after updating

[MateoPetitet]

Search before asking

• I have searched the RF-DETR issues and found no similar bug report.

Bug

Hello !

I'm about to start a new training session for my custom model. However, I encounter a problem when I use the latest version (1.2.1) vs when I used the older 1.1.0 version. Here are my parameters, please note that I'm exploring on cpu and system memory for now as I want to be sure everything works before renting a server :

model.train(dataset_dir=data_dir,
            device='cpu',
            num_workers=1,
            epochs=num_epochs, 
            batch_size=1, 
            grad_accum_steps=16,
            lr=learn, 
            output_dir=output_path, 
            resolution=1008, 
            weight_decay=1e-8, 
            checkpoint_interval=cp_interval, 
            resume=cp_resume,
            early_stopping=True,
            early_stopping_patience=15,
            num_queries=300, 
            num_select=300, 
            dec_layers=6
            )

I don't have any problem when training wit these parameters in the old version (I managed to start my epoch, even if it was very long on cpu) but the latest version returns the following error : RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.

I don't know where it comes from, and the full errors output doesn't really help to know how to fix. Any idea what's happening ?

Thank you for reading !

Environment

RF-DETR : 1.1.0/1.2.1
OS : ZorinOS 17.3 core (equivalent Ubuntu LTE 22.
Python : 3.10
PyTorch : 2.6.0 version
CPU : core i5 11400H

Minimal Reproducible Example

#run this with rf-detr 1.1.0 the 1.2.1 to see the difference

device = torch.device('cpu')
data_dir = os.path.join(args.path)
output_path = os.path.join(args.output)
model = RFDETRLarge(pretrained=True)
num_epochs = args.epochs
learn = args.learning_rate
cp_interval = args.checkpoint_interval
model.train(dataset_dir=data_dir,
                device='cpu',
                num_workers=1,
                epochs=num_epochs, 
                batch_size=1, 
                grad_accum_steps=16,
                lr=learn, 
                output_dir=output_path, 
                resolution=1008, 
                weight_decay=1e-8, 
                checkpoint_interval=cp_interval, 
                resume=cp_resume,
                early_stopping=True,
                early_stopping_patience=15,
                num_queries=300, 
                num_select=300, 
                dec_layers=6 
                )

Additional

No response

Are you willing to submit a PR?

• Yes, I'd like to help by submitting a PR!

[isaacrob-roboflow]

looks like the same issue as #297 I am wondering if that issue wasn't using the GPU properly or there was an issue with the AMD drivers

we never tested CPU-only training, I buy that there could be a new bug

it's not letting me view the full stacktrace without installing an extension for their website which I'm not going to do. can you paste it in a comment?

[isaacrob-roboflow]

have you tried training in colab for a free gpu?

[MateoPetitet]

Sorry, I didn't know it needed an extension ! I use it without one, it's weird. Anyway, here it is :

 File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/detr.py", line 81, in train
    self.train_from_config(config, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/detr.py", line 187, in train_from_config
    self.model.train(
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/main.py", line 341, in train
    train_stats = train_one_epoch(
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/engine.py", line 129, in train_one_epoch
    outputs = model(new_samples, new_targets)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/lwdetr.py", line 146, in forward
    features, poss = self.backbone(samples)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/__init__.py", line 27, in forward
    x = self[0](tensor_list)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/backbone.py", line 124, in forward
    feats = self.encoder(tensor_list.tensors)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2.py", line 188, in forward
    x = self.encoder(x)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py", line 1073, in forward
    embedding_output = self.embeddings(pixel_values)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py", line 291, in forward
    embeddings = self.patch_embeddings(pixel_values.to(dtype=target_dtype))
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py", line 212, in forward
    embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
    return F.conv2d(
RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.

I didn't include the first line as it justs show that the error is in the train function.

looks like the same issue as #297

Right ! But since it was on GPU I marked it as different

have you tried training in colab for a free gpu?

I'm gonna try it on the server I'm gonna rent, today I hope ; since I know it works on the old version I'm not worried that I won't be able to fine-tune, I'll just use 1.1.0 but I'll try to replicate in the new version + cuda gpu !

[MateoPetitet]

I'll also gonna try again with the last version of pytorch, I'm just waiting for it to be downloaded (my internet is very slow)

[MateoPetitet]

Update : i tried again on cpu with the latest version of pytorch / torchvision and it returned basically the same thing :

  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/detr.py", line 81, in train
    self.train_from_config(config, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/detr.py", line 187, in train_from_config
    self.model.train(
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/main.py", line 341, in train
    train_stats = train_one_epoch(
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/engine.py", line 129, in train_one_epoch
    outputs = model(new_samples, new_targets)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/lwdetr.py", line 146, in forward
    features, poss = self.backbone(samples)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/__init__.py", line 27, in forward
    x = self[0](tensor_list)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/backbone.py", line 124, in forward
    feats = self.encoder(tensor_list.tensors)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2.py", line 188, in forward
    x = self.encoder(x)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py", line 1073, in forward
    embedding_output = self.embeddings(pixel_values)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py", line 291, in forward
    embeddings = self.patch_embeddings(pixel_values.to(dtype=target_dtype))
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py", line 212, in forward
    embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/user/miniforge3/envs/rfdetr/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
    return F.conv2d(
RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.

[isaacrob-roboflow]

Ok. I think I know what the problem is. Annoying little PyTorch issue :)

Don't have much bandwidth right now (expect some exciting things in the near future 😏) and since we don't recommend training on CPU and you're not blocked I think this is somewhat low priority. Will fix as I have time :)

[MateoPetitet]

Considering the time it takes to pass just a single epoch on CPU I understand why it's low priority 😆 I'll update when I manage to try on cuda, hopefully it will work !