Skip to content

Commit d5f0c1a

Browse files
authored
Merge branch 'master' into Docs/drop-eco-ci
2 parents 66a760f + 3726e54 commit d5f0c1a

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

docs/source-pytorch/data/alternatives.rst

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,12 @@ The webdataset library contains a small wrapper (``WebLoader``) that adds a flui
9999
import lightning as L
100100
import webdataset as wds
101101
102-
dataset = wds.WebDataset(urls)
102+
dataset = wds.WebDataset(
103+
urls,
104+
# needed for multi-gpu or multi-node training
105+
workersplitter=wds.shardlists.split_by_worker,
106+
nodesplitter=wds.shardlists.split_by_node,
107+
)
103108
train_dataloader = wds.WebLoader(dataset)
104109
105110
model = ...

0 commit comments

Comments
 (0)