How to display progress on IterableDataset?

[rongcuid]

When I use an IterableDataset, the Trainer.fit() gives something like:

Epoch 0: : 21it [02:41,  7.69s/it, loss=0.663, v_num=18]  

Which does not display a progress. Even if I specify a validation interval, there is no progress shown. Is there a way I can specify an "epoch length", or something like that for IterableDataset?

[awaelchli]

Hi
IterableDatasets don't have a lengh defined. They can either be infinite in size or raise StopIteration at an arbitrary point. Hence, there is no notion of epochs and we cannot anticipate when it finishes iterations.

If you really need to have epochs, I suggest you try to rewrite your dataset logic into a regular Dataset.

[awaelchli]

Epoch 0: 28%|██▊ | 8878/32000 [00:07<00:19, 1169.46it/s, loss=-7.08e+03, v_num=21]

[rongcuid]

That sounds nice. However, if I subclass it, do I need to do special things to prevent the dataset from returning the same items when I use num_workers > 0 in DataLoader? I have not looked into WebDataset's code yet so I don't know how it handles this.

[awaelchli]

In webdataset, apparently you can set the length:
https://github.com/tmbdev/webdataset/blob/3431eeada3cba81c19966604116c4b5d6d6994ae/webdataset/dataset.py#L272
It's not documented.

[rongcuid]

Oh, that is great! Thank you for your help.

[rongcuid]

Just so you know, if I define a __len__ that returns a different value from actual, DataLoader will flood the screen with UserWarnings.