Efficient way of loading data from an h5 dataset? #13861
Unanswered
malfonsoarquimea
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi! I am writing the code of a datamodule whose get_item() method will simply perform some operations to chose an index and then picking the item at this index from the h5 dataset.
The thing is that, as far as I know, it will be much slower to pick a batch one by one from the h5 file than picking the whole batch at once. Is there some more clever approach to do this so that the datamodule itself is more efficient (accounting for the dataset and the dataloaders)
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions