Something went wrong in training train_mvod_lstm1.py

[Vickywz]

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 256, 1, 1]] is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I don't know how to modify it, I would like to ask you，thank you！

[onurbagoren]

Hello, did you ever get a resolution on this? Also running into the same issue

[nis243]

Hello.
In RTX3080Ti + torch 1.10.0+cu113, the same phenomenon occurred.
It worked somehow by downgrading the version.

It's not realistic, but if you want it to work on the CPU, you'll probably need to modify the '.cuda()' part of 'network/mvod_*.py'.

RTX3080Ti + Python 3.6.13 + torch 1.4.0 + use CPU

RTX2080Ti + Python 3.6.13 + torch 1.4.0+cu100
Why is val commented on train_mvod_lstm1.py? I don't understand the meaning. ...

[rui-wang-ryan]

I see the same error. it is probably due to

The BatchNorm layers' internal running_mean and running_var buffers are modified in-place during the forward pass of each frame. When retain_graph=True is used, autograd expects these buffers (which are part of the graph needed for earlier frames' gradients) to remain unchanged. maybe some other in place operation, besides, Batchnorm2D.