How to create a custom loader and sampler in pyg?

[pratham16cse]

I want to create a custom loader and sampler (similar to the ones exist such as ClusterLoader, HGTLoader, HGTSampler). Although modifying the source code of these loaders and samplers is an option, it is not trivial. I was wondering if there is an easier way to achieve this? Any help is appreciated.

Thanks,
Prathamesh

[rusty1s]

The most straightforward way to do this is to utilize NodeLoader or LinkLoader and pass in a custom BaseSampler that implements Sampler.sample_from_nodes or Sampler.sample_from_edges. Is this possible for your use-case?

[pratham16cse]

Thanks @rusty1s for the response.

Yes, I was also thinking of inheriting the NodeLoader class and implementing the sample_from_nodes function. However, I could not find any example implementation of such custom sampler which implements Sampler.sample_from_nodes or Sampler.sample_from_edges. If there exists any that I might have missed, please provide the link here. Thanks for your help.

[rusty1s]

• NeighborSampler
• HGTSampler (this one might be easier to understand)

Note that both will propagate the sample_from_nodes call to a dedicated C++ implementation.

[rish-16]

Hello! I had a similar follow-up and wanted to see if it's possible to build it with PyG. I have a dataset of different graphs of different sizes (# of nodes). IS there a way to batch together graphs with the same number of nodes? Do I need to modify BaseSampler or is there an easier way? Thank you!

[rusty1s]

Would DynamicBatchSampler resolve your issue?

[pratham16cse]

If I understand correctly, you have a dataset of multiple graphs and you want all the graphs with the same number of nodes in a single batch.

Here is a potential way to achieve this: You can write a sampler class that keeps track of all unique lengths of the graphs in the dataset. While creating a batch, you can randomly select a size, say s, and return all the graphs of size s from the dataset. You can use the subgraph function from pyg to get the induced subgraph as a batch. Make sure that the indices are not messed up while doing this. You can implement this through pytorch dataloader or using just a pure python class as well.

[rish-16]

That works too! Thanks :)

I maintain a CSV with info about each sample (# nodes, etc). I can sample from that in the Sampler.