What is consuming RAM after on_train_epoch_start but before training_step

[malfonsoarquimea]

Hi! I am currently facing an issue where a lot of ram is consumed while training. By running the debugger while monitoring ram consumption I managed to find that the ram consumption process happens after on_train_epoch_start callback and before training_step, and I don't have any on_train_batch_start callback defined. I think the issue is related to how I defined my dataset and dataloaders, but I can not find what part of the code is producing the memory consumption since I don't know which piece of the code is called in between on_train_epoch_start and training_stepso I can not debug it.
Do you have some insight on where to look to find where is memory being consumed?
Thanks very much in advance

[rohitgr7]

it must be the batch that is extracted from your dataloader that is being passed to your training_step.

[malfonsoarquimea]

Thanks very much @rohitgr7, thanks to your help, i was able to track the memory consumption to line 153 of loops/epoch/training_epoch_loop.py.

https://github.com/PyTorchLightning/pytorch-lightning/blob/f300b60f472b56f1aabd428a268184b53d3f9a55/pytorch_lightning/loops/epoch/training_epoch_loop.py#L153

The code runs flawlessly to that point and there, while executing this line, the RAM consumption starts to increase until all the available ram is consumed. What is exactly done there to produce such a huge memory consumption?

[rohitgr7]

is it happening on the latest release? I remember some issues were fixed recently in the data fetcher.

[malfonsoarquimea]

@rohitgr7 yes, I am using pytorch lightning 1.6.3

[rohitgr7]

ah! then it's an issue in PL. Is it possible for you to share a reproducible script?

[malfonsoarquimea]

@rohitgr7 Sorry for the delay, didn't have time to follow this issue. I will try to create a reproducible script so you can debug it!
regards!