Any idea to delete dataset_Train after "fit"? #13009
Unanswered
allanchan339
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Due to IO performance, I usually load all image-label pairs in RAM to accelerate the model training.
It is done by creating a large NumPy array in the dataset.init(). This method consumes almost all the RAM. Therefore, I need to manually delete the dataset_Train after training to free some space for dataset_Test.
The GC and
del
methods work well in PyTorch. However, PyTorch lightning recommends using DataModule to wrap the dataset_Train, dataset_Test and etc. I don't know how to implement the same deletion after "fit".Any idea to implement the same implementation when using the datamodule to wrap?
Beta Was this translation helpful? Give feedback.
All reactions