Multi-processing with IterableDataset Warning #6940
Unanswered
ozen
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
Replies: 1 comment
-
It was fixed in #6972, the message should be clearer now. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hello. The warning message given when using an IterableDataset that has
__len__
defined got me confused.https://github.com/PyTorchLightning/pytorch-lightning/blob/049006a59cf62d3539aaf19037aca1caac98abfd/pytorch_lightning/utilities/data.py#L40-L45
Is the part
e.g. batch size > 1
should bee.g. num_workers > 1
or am I missing something?Is this warning about what IterableDataset doc explains about configuring the workers? If I write my IterableDataset similarly to the example provided in the doc and also define a
__len__
function, will there be any problems due to using PL?Thanks.
Beta Was this translation helpful? Give feedback.
All reactions