Consultation on CUDA Out-of-Memory Issue When Running the Code with TCGA - BRCA Dataset #18
Description
Dear author,
I'm using your code and have encountered some issues. Here is some information about my setup and the output I got:
Environment Information
- I'm using a single NVIDIA GeForce RTX 3090 GPU with 24GB of memory.
- The dataset I'm using is TCGA - BRCA.
Output
The size of train dataset: 804
The size of val dataset: 95
The size of test dataset: 98
use encoder_decoder: default
Training: 33%|██████████████████████████████████████████▏ | 267/804 [00:31<01:03, 8.4
Traceback (most recent call last):
File "main.py", line 198, in <module>
mp.spawn(main,
File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/mnt/data0/BLZ/Wsi-Caption-master/main.py", line 181, in main
trainer.train(local_rank)
File "/mnt/data0/BLZ/Wsi-Caption-master/modules/trainer.py", line 58, in train
result = self._train_epoch(epoch)
File "/mnt/data0/BLZ/Wsi-Caption-master/modules/trainer.py", line 238, in _train_epoch
loss.backward()
File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/jw102/.conda/envs/wsicaption/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.55 GiB (GPU 0; 23.69 GiB total capacity; 19.42 GiB already allocated; 923.69 MiB free; 21.33 GiB reserved in total brch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
It seems that the code is running out of CUDA memory. I've checked the error message, but I'm not sure how to fix it properly. Could you please give me some advice on how to modify the code to avoid this CUDA out - of - memory issue?
Thank you very much!