Is anyone using webdataset with pl?

[cwerner]

Hi all 👋

I was wondering if any PyTorch lightning practitioners are using webdataset (https://github.com/tmbdev/webdataset) for data loading? I initially planned on implementing hdf5 dataloaders to avoid having >500k files in my filesystem but figured webdataset tars would actually be a better fit...
I tried to use it for my semantic segmentation project but I’m stuck in the middle (decode failing and I haven’t previously used iterabledataset).
Unfortunately, the documentation is pretty scarce and I can’t find many examples. I think It’d help me a lot if I’d see an actual implementation example with current PyTorch lightning.

cheers,
Christian

[cwerner]

Ok, seems like webdataset decode has issues decoding my tif shards. When I switch to pngs my pipeline works... still interested in example code if people use webdataset with pl to learn some tricks 😉

[rmchurch]

Feeling through this myself right now. A little verbose, but saw this example repo of using webdataset with PL https://github.com/webdataset/webdataset-lightning