RuntimeError: Trying to resize storage that is not resizable at . . \aten\src\TH\THStorageFunctions.cpp:87

[kynnemall]

I've been trying to train DECODE but keep running into this error which I initially thought was due to the Windows-Pytorch multiprocessing issue highlighted in the documentation. I tried a few different ways to disable multiprocessing using the params.yaml and each time it seemed to help because the model trained for about 10-20 more epochs but then the error appeared again.

Has anyone else had this issue and knows how to get around it?

[Haydnspass]

Hey @kynnemall
could you post the output of conda list when ran inside the decode_env? It should show the list of packages with versions.

[kynnemall]

Conda packages listed in attached text file
decode_env_pkgs.txt

[Haydnspass]

Hm that looks okay.

[Haydnspass]

Could you set the following in your yaml parameter file and provide me a 'param_run.yaml' of the training that failed?

# ...
Hardware:
  num_worker_train: 0
  torch_threads: 1
# ...

[kynnemall]

I've started training with your suggested changes and will see how they get on . "param_run.yaml" attached in text format
param_run.txt

[Haydnspass]

So I can already see the issue in this param_run: The number of workers is set to 1; if you run into multiprocessing issues (on Windows) you should set it to 0. Worker=1 will still do multiprocessing, but with one core, which is not what you wanted.
If you set it to 0 most probably you do not need to set the torch_threads to 1 and can just omit this line.

[Haydnspass]

Hey @kynnemall
I just wanted to follow up. Has it worked? Is this solved?

[kynnemall]

Hi, yes that solved my problem. I was waiting to see if training completed successfully before replying and was so happy the model trained to completion that I forgot to reply.

Thanks again for your help