File tree Expand file tree Collapse file tree 1 file changed +7
-2
lines changed Expand file tree Collapse file tree 1 file changed +7
-2
lines changed Original file line number Diff line number Diff line change @@ -99,14 +99,19 @@ The webdataset library contains a small wrapper (``WebLoader``) that adds a flui
99
99
import lightning as L
100
100
import webdataset as wds
101
101
102
- dataset = wds.WebDataset(urls)
102
+ dataset = wds.WebDataset(
103
+ urls,
104
+ # needed for multi-gpu or multi-node training
105
+ workersplitter = wds.shardlists.split_by_worker,
106
+ nodesplitter = wds.shardlists.split_by_node,
107
+ )
103
108
train_dataloader = wds.WebLoader(dataset)
104
109
105
110
model = ...
106
111
trainer = L.Trainer()
107
112
trainer.fit(model, train_dataloader)
108
113
109
- You can find a complete example `here <https://github.com/webdataset/webdataset-lightning >`__.
114
+ You can find a complete example `here <https://github.com/webdataset/webdataset-lightning >`__.
110
115
111
116
NVIDIA DALI
112
117
^^^^^^^^^^^
You can’t perform that action at this time.
0 commit comments