RuntimeError: Allocation on device

[BenDes21]

Hi there, getting this with 12gb VRAM, is normal ?

The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/__torch__/saicinpainting/training/trainers/default.py", line 13, in forward
masked_img = torch.mul(img, torch.rsub(mask, 1, 1))
input = torch.cat([masked_img, mask], 1)
_1 = torch.mul(mask, (_0).forward(input, ))
~~~~~~~~~~~ <--- HERE
_2 = torch.mul(torch.rsub(mask, 1, 1), img)
return torch.add(_1, _2, alpha=1)
File "code/__torch__/saicinpainting/training/modules/ffc.py", line 9, in forward
def forward(self: __torch__.saicinpainting.training.modules.ffc.FFCResNetGenerator,
input: Tensor) -> Tensor:
return (self.model).forward(input, )
~~~~~~~~~~~~~~~~~~~ <--- HERE
class FFC_BN_ACT(Module):
__parameters__ = []
File "code/__torch__/torch/nn/modules/container/___torch_mangle_818.py", line 78, in forward
_32 = getattr(self, "1")
_33 = (getattr(self, "0")).forward(input, )
_34 = (_31).forward((_32).forward(_33, ), )
~~~~~~~~~~~~ <--- HERE
_35 = (_29).forward((_30).forward(_34, ), )
_36, _37, = _35
File "code/__torch__/saicinpainting/training/modules/ffc.py", line 25, in forward
_1 = self.bn_g
_2 = self.act_l
_3 = (self.bn_l).forward((self.ffc).forward(argument_1, ), )
~~~~~~~~~~~~~~~~~~ <--- HERE
_4 = (_2).forward(_3, )
_5 = (_1).forward()
File "code/__torch__/torch/nn/modules/batchnorm.py", line 16, in forward
_1 = self.running_mean
_2 = self.bias
input = torch.batch_norm(argument_1, self.weight, _2, _1, _0, False, 0.10000000000000001, 1.0000000000000001e-05, True)
~~~~~~~~~~~~~~~~ <--- HERE
return input

Traceback of TorchScript, original code (most recent call last):
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/functional.py(2150): batch_norm
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py(140): forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/root/home/code/lama/saicinpainting/training/modules/ffc.py(253): forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/container.py(119): forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/root/home/code/lama/saicinpainting/training/modules/ffc.py(367): forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/root/home/code/lama/saicinpainting/training/trainers/default.py(83): forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/jit/_trace.py(940): trace_module
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/jit/_trace.py(742): trace
bin/to_jit.py(46): main
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/core/utils.py(160): run_job
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/hydra.py(105): run
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/utils.py(371): <lambda>
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/utils.py(211): run_and_report
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/utils.py(368): _run_hydra
/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/main.py(53): decorated_main
bin/to_jit.py(59): <module>
RuntimeError: Allocation on device