How to convert the batched graph sampled by NeighborLoader() to sub-graph list efficiently

[xiachenrui]

Hi all,

I want to convert the batched graph sampled by NeighborLoader(batch_size = 1000, ...) to list of sub-graphs (then store them in OnDiskDataset). But the sampled graph is Data (not Batch) object and I am not sure how to convert it to list of sub-graphs.
And NeighborLoader(batch_size = 1) is too slow (30 sub-graphs per second, but I have 10 Million sub-graphs).

Any suggestion will be useful !

[rusty1s]

Is NeighborLoader(batch_size=1) even slow when using it with many workers? You could do larger batches, but then you need to convert your batch to single subgraphs back which again takes time to compute. Alternatively, it might be also a good idea to store data in batches rather than individually.